CN107958230B - Facial expression recognition method and device - Google Patents
Facial expression recognition method and device Download PDFInfo
- Publication number
- CN107958230B CN107958230B CN201711402426.XA CN201711402426A CN107958230B CN 107958230 B CN107958230 B CN 107958230B CN 201711402426 A CN201711402426 A CN 201711402426A CN 107958230 B CN107958230 B CN 107958230B
- Authority
- CN
- China
- Prior art keywords
- face
- image
- facial
- expression
- facial expression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention is suitable for the technical field of facial expression recognition, and provides a facial expression recognition method and a device, wherein the method comprises the following steps: acquiring an image to be processed; extracting a face image from the image to be processed; performing expression classification on the face image based on deep learning to obtain a face expression label of the face image, wherein the face expression label indicates the expression of the face; performing face verification on the face image to obtain a face verification result, wherein the face verification result indicates information of a user to which the face belongs; and displaying the facial expression label and the facial verification result. The method and the device can solve the problems that in the prior art, the expression identification accuracy is low, the expressions are lack of correlation, and different expressions of the same user cannot be obtained.
Description
Technical Field
The invention belongs to the technical field of facial expression recognition, and particularly relates to a facial expression recognition method and device.
Background
Facial expression recognition refers to analyzing and detecting the expression state of a human face from a given image, so as to determine the psychological emotion of a recognized object, such as nature, joy, anger, surprise and the like. Facial expression recognition is an important field, and is beneficial to the development of numerous fields such as character mood analysis, character personality analysis, depression detection and the like. Therefore, the human body recognition method based on the facial expression has very important value in solving the problem of human body recognition. However, the existing facial expression recognition has poor robustness on expression features, and is easily interfered by noise such as identity information, so that the accuracy of expression recognition is low. Meanwhile, the current expression recognition algorithm can only recognize target expressions generally, and the expressions lack correlation and cannot acquire different expressions of the same user.
Therefore, a new technical solution is needed to solve the above technical problems.
Disclosure of Invention
In view of the above, the invention provides a facial expression recognition method and device, so as to solve the problems that in the prior art, the accuracy of expression recognition is low, expressions are lack of correlation, and different expressions of the same user cannot be known.
The first aspect of the present invention provides a facial expression recognition method, including:
acquiring an image to be processed;
extracting a face image from the image to be processed;
performing expression classification on the face image based on deep learning to obtain a face expression label of the face image, wherein the face expression label indicates the expression of the face;
performing face verification on the face image to obtain a face verification result, wherein the face verification result indicates information of a user to which the face belongs;
and displaying the facial expression label and the facial verification result.
A second aspect of the present invention provides a facial expression recognition apparatus, including:
the image acquisition module is used for acquiring an image to be processed;
the face extraction module is used for extracting a face image from the image to be processed;
the expression classification module is used for carrying out expression classification on the face image based on deep learning to obtain a face expression label of the face image, wherein the face expression label indicates the expression of the face;
the face verification module is used for performing face verification on the face image to obtain a face verification result, wherein the face verification result indicates information of a user to which the face belongs;
and the display module is used for displaying the facial expression label and the facial verification result.
A third aspect of the present invention provides a facial expression recognition apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the facial expression recognition method according to the first aspect when executing the computer program.
A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the facial expression recognition method as described in the first aspect above.
Compared with the prior art, the invention has the following beneficial effects: the scheme of the invention obtains the image to be processed, extracts the face image from the image to be processed, carries out expression classification on the face image based on deep learning, obtains the face expression label of the face image, and obtains the face verification result by carrying out face verification on the face image, thereby obtaining the user to which the face belongs. The scheme of the invention improves the accuracy of expression recognition by recognizing the expressions in the face images based on deep learning, and can correlate the expressions in the face images by performing face verification on the face images to judge whether the expressions belong to different expressions of the same user.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flow chart illustrating an implementation of a facial expression recognition method according to an embodiment of the present invention;
FIG. 2a is an exemplary diagram of different expressions of the same user; FIG. 2b is an exemplary diagram of different user expressions;
fig. 3 is a schematic flow chart illustrating an implementation of a facial expression recognition method according to a second embodiment of the present invention;
fig. 4 is a schematic view of an implementation flow of a facial expression recognition method according to a third embodiment of the present invention;
fig. 5 is a schematic diagram of a facial expression recognition apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic diagram of a facial expression recognition apparatus according to a fifth embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
It should be understood that, the sequence numbers of the steps in this embodiment do not mean the execution sequence, and the execution sequence of each process should be determined by the function and the inherent logic of the process, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
In order to explain the technical means of the present invention, the following description will be given by way of specific examples.
Referring to fig. 1, which is a schematic view of an implementation flow of a facial expression recognition method according to an embodiment of the present invention, as shown in the figure, the facial expression recognition method may include the following steps:
step S101, acquiring an image to be processed.
In the embodiment of the present invention, the image to be processed may be an image directly input to the facial expression recognition device, or an image obtained by inputting a video to the facial expression recognition device or an image obtained by directly connecting the facial expression recognition device to the camera device. In general, each frame is extracted from a video and processed by using a video or camera. It should be noted that the number of the images to be processed may be one or more, and is not limited herein. For example, the image to be processed is two frames of images of which the video is separated by one second, that is, the number of the image to be processed is two.
And S102, extracting a face image from the image to be processed.
In the embodiment of the invention, the position information of the face in the image to be processed can be determined through a Dlib machine learning library, and then the face extraction is carried out from the image to be processed, wherein the Dlib is a machine learning library written by C + + and comprises a plurality of machine learning common algorithms. If the image to be processed contains a plurality of faces, after the faces in the image to be processed are extracted, a plurality of face images with different sizes may be obtained, then expression classification and face verification are respectively carried out on the obtained face images so as to identify the expression of each face, and information of a user to which each face belongs is obtained, so that whether the faces belong to the same user can be judged.
And step S103, performing expression classification on the face image based on deep learning, and acquiring a face expression label of the face image.
Wherein the facial expression label indicates an expression of the face.
In the embodiment of the present invention, the facial image may be subjected to expression classification through a Convolutional Neural Network (CNN), that is, an expression of a face in the facial image is recognized.
And step S104, carrying out face verification on the face image to obtain a face verification result.
Wherein the face verification result indicates information of a user to which the face belongs.
In the embodiment of the present invention, the face verification may be performed on the face image through a face verification model (for example, a deep face verification model), specifically, the face verification model may verify which user's face in the face image belongs to in a facial expression database. The facial expression database may store information of a plurality of users and facial expressions of each of the plurality of users, where the information of the user may refer to identification information representing identities of the users, and may be capable of distinguishing different users, for example, setting a serial number for each user in the facial expression database.
Alternatively, after step S102 is performed, steps S103 and S104 may be performed simultaneously.
And step S105, displaying the facial expression label and the facial verification result.
Specifically, the facial image, the facial expression label and the facial verification result may be displayed, and the facial expression label and the facial verification result may be displayed at an appointed position of the facial image (for example, above, below, left side or right side of the facial image), so that a user can know which facial image the facial expression label and the facial verification result correspond to, and the facial expression database is updated, so as to increase the facial expression label of the user to which the face belongs in the facial expression data.
In the embodiment of the present invention, displaying the facial expression label can facilitate a user to check the expression of the face in the facial image, displaying the facial verification result can facilitate to check which user the face in the facial image belongs to, and simultaneously displaying the facial expression label and the facial verification result can facilitate the user to check which user the expression of the face in the facial image belongs to. When the number of the face images extracted in step S102 is plural, the facial expression labels and the face verification results of the face images may be displayed at the designated positions of each face image. Fig. 2a shows an example of different expressions of the same user, and p1 in fig. 2a is the serial number of the user. An example diagram of different user expressions is shown in fig. 2 b. Different sequence numbers may represent different users, so that the user can know whether the facial expression is recognized to belong to the same user by looking at the sequence numbers in the facial image, such as p2 and p3 in fig. 2 b.
The embodiment of the invention identifies the expressions in the face images based on the deep learning, improves the accuracy of expression identification, and can correlate the expressions in the face images by carrying out face verification on the face images to judge whether the expressions belong to different expressions of the same user.
Referring to fig. 3, which is a schematic view of an implementation flow of the facial expression recognition method according to the second embodiment of the present invention, as shown in the figure, the facial expression recognition method may include the following steps:
step S301, an image to be processed is acquired.
The step is the same as step S101, and reference may be made to the related description of step S101, which is not repeated herein.
Step S302, extracting a face image from the image to be processed.
The step is the same as step S102, and reference may be made to the related description of step S102, which is not repeated herein.
Step S303, adjusting the size of the face image to a first preset size.
In an embodiment of the present invention, the size of the face image may be scaled to a first preset size, specifically, the size of the face image may be scaled to a size of M × M (e.g., 48 × 48), where M is an integer greater than zero.
Step S304, segmenting an image with a second preset size from the N preset positions in the adjusted face image.
Wherein N is an integer greater than zero.
In the embodiment of the present invention, after the size of the face image is scaled, the scaled face image is segmented, and an image with a second preset size can be segmented from N preset positions in the face image, that is, N images with the second preset size are segmented from the face image. For example, an image with a size of 42 × 42 is segmented from the top left corner, the bottom left corner, the top right corner, the bottom right corner, and the center of the face image, i.e., five images with a size of 42 × 42 are segmented from the face image.
Step S305, inputting the N divided images into a Convolutional Neural Network (CNN) expression classification model for prediction, and acquiring the facial expression labels of the facial images.
Wherein the facial expression label indicates an expression of the face.
In the implementation of the present invention, the images at the N preset positions obtained in step S304 may be input into a trained CNN expression classification model for prediction, so that the prediction probability of each expression of each image at the N preset positions may be obtained, and the expression with the largest average value of the prediction probabilities in each expression is taken as the expression of the facial image.
The training method of the CNN expression classification model may specifically be: obtaining an expression classification data set, preprocessing all images in the expression classification data set (screening face images from all the images, adjusting the size of the screened face images to a first preset size), obtaining face images with the first preset size, randomly dividing each face image in the face images to divide K (wherein K is an integer larger than zero, such as eight) images with the second preset size, randomly turning over the K images divided from each face image for training, which is helpful for improving the spatial adaptability of the model, obtaining expression classification results, then calculating a Loss function Loss, and performing model optimization by using a random gradient descent optimizer, wherein the learning rate can be 0.001, the total iteration number can be 50000 times, and the test can be performed every 1000 times, the test data set is also preprocessed to obtain a face image with a first preset size, then images with a second preset size are respectively segmented at preset positions in the face image for testing, finally, the classification probability of the preset positions is averaged to obtain an expression classification result, the accuracy rate is calculated, and a model with high accuracy rate is reserved.
Optionally, the inputting the N segmented images into a convolutional neural network CNN expression classification model for prediction, and the obtaining the facial expression label of the facial image includes:
inputting the N segmented images into a CNN expression classification model for prediction, and obtaining the prediction probability of each image of various human face expressions in the N images;
calculating the average value of the prediction probability of each facial expression in the N images according to the prediction probability of each image in the N images of the facial expressions;
and taking the facial expression with the maximum average value of the prediction probabilities in the plurality of facial expressions as a facial expression label of the facial image.
Wherein the plurality of facial expressions include, but are not limited to, nature, happy, surprise, hurting, fear, anger, and the like.
Illustratively, an image with the size of 42 × 42 is respectively segmented from the upper left corner, the lower left corner, the upper right corner, the lower right corner and the center position of the face image, and can be respectively defined as a first image, a second image, a third image, a fourth image and a fifth image, the five images are input into a CNN expression classification model for prediction, and the prediction probabilities of nature, happiness, surprise, heart injury, fear, anger and the like in the first image are respectively 0.6, 0.1 and 0; the prediction probabilities of the six expressions in the second image are 0.5, 0.2, 0.1, 0 and 0.1 respectively; the prediction probabilities of the six expressions in the third image are 0.6, 0.1 and 0 respectively; the prediction probabilities of the six expressions in the fourth image are 0.5, 0.2, 0.1 and 0 respectively; the prediction probabilities of the six expressions in the fifth image are 0.7, 0, 0.1, 0 and 0.1 respectively; the average values of the prediction probabilities of the six expressions in the five images can be calculated to be 0.58 (natural), 0.12 (happy), 0.1 (surprised), 0.1 (hurry), 0.06 (afraid) and 0.04 (angry) according to the prediction probabilities of the six expressions in each image, so that the human face expression in the human face image can be judged to be natural.
And step S306, carrying out face verification on the face image to obtain a face verification result.
Wherein the face verification result indicates information of a user to which the face belongs.
The step is the same as step S104, and reference may be made to the related description of step S104, which is not repeated herein.
And step S307, displaying the facial expression label and the facial verification result.
The step is the same as step S105, and reference may be made to the related description of step S105, which is not repeated herein.
The embodiment of the invention adds expression classification to the face image through the CNN on the basis of the first embodiment, thereby identifying the expression in the face image and improving the accuracy of expression identification.
Referring to fig. 4, which is a schematic view of an implementation flow of a facial expression recognition method provided by the third embodiment of the present invention, as shown in the figure, the facial expression recognition method may include the following steps:
step S401, acquiring an image to be processed.
The step is the same as step S101, and reference may be made to the related description of step S101, which is not repeated herein.
Step S402, extracting a face image from the image to be processed.
The step is the same as step S102, and reference may be made to the related description of step S102, which is not repeated herein.
Step S403, performing expression classification on the face image based on deep learning, and acquiring a face expression label of the face image.
Wherein the facial expression label indicates an expression of the face.
The step is the same as step S103, and reference may be made to the related description of step S103, which is not described herein again.
Step S404, adjusting the size of the face image to a third preset size.
In an embodiment of the present invention, the size of the face image may be scaled to a third preset size, specifically, the size of the face image may be scaled to a size of L1 × L2 (for example, 39 × 31), where L1 and L2 are integers greater than zero.
Step S405, the adjusted face image is divided into a plurality of images.
In the embodiment of the present invention, after the size of the face image is scaled, the face image after scaling adjustment may be randomly segmented, and the face image after scaling adjustment may be segmented into a plurality of images. The sizes of the plurality of images may be the same or different, and are not limited herein, and the number of the plurality of images is not limited.
Step S406, inputting the plurality of images into a face verification model, and acquiring the classification probability of each user of the face in a facial expression database.
The facial expression database may be a database in which information of a large number of users and facial expression labels of each user are stored.
Step S407, if the maximum value in the classification probabilities is greater than a preset threshold, determining that the user to which the face belongs is the user corresponding to the maximum value in the classification probabilities.
In the embodiment of the present invention, the plurality of divided images are respectively input into a face verification model (for example, a deep id face verification model), so as to obtain a classification probability of each user in a facial expression database of a face in the face image, that is, a classification probability that a user of the face in the face image belongs to each user in the facial expression database. For example, the facial expression database stores facial expression labels of 1000 users, the 1000 users are respectively numbered with serial numbers, such as p1, p2, p3, …, and p1000, the classification probability of each user of the face in the facial expression database in fig. 1 is 0.8, 0, 0.2, 0, …, and 0, and the preset probability is 0.6, it can be determined that the facial expression label in fig. 1 belongs to the user p1, that is, the facial image in fig. 1 belongs to the facial image of the user p1, and then the facial expression label of the face and the serial number p1 of the user can be displayed above the facial image in fig. 1.
Optionally, the embodiment of the present invention further includes:
if the maximum value in the classification probabilities is smaller than or equal to a preset threshold value, determining that the user to which the face belongs does not exist in the facial expression database;
and adding the facial expression label and the information of the user to which the face belongs to the facial expression database.
Illustratively, the facial expression database contains 1000 users, but the facial image extracted from the image to be processed does not belong to the facial image of any of the 1000 users, at this time, the serial number of the user corresponding to the facial image may be set as p1001, and the correspondence between the facial expression label of the facial image and the information of the user is added to the facial expression database.
In the embodiment of the invention, the facial expression database can be updated by adding the facial expression labels and the information of the user to which the face belongs to the facial expression database, and meanwhile, in order to improve the accuracy of the facial verification model and facilitate the subsequent facial verification of the facial image of the user to which the face belongs through the facial verification model, the last Soft-max layer of the facial verification model can be updated and retrained.
And step S408, displaying the facial expression label and the information of the user to which the face belongs.
The step is the same as step S105, and reference may be made to the related description of step S105, which is not repeated herein.
The embodiment of the invention identifies the expressions in the face images based on the deep learning, improves the accuracy of expression identification, and can correlate the expressions in the face images by carrying out face verification on the face images to judge whether the expressions belong to different expressions of the same user.
Fig. 5 is a schematic diagram of a facial expression recognition apparatus according to a fourth embodiment of the present invention, and for convenience of description, only the parts related to the embodiment of the present invention are shown.
The facial expression recognition device includes:
an image obtaining module 51, configured to obtain an image to be processed;
a face extraction module 52, configured to extract a face image from the image to be processed;
an expression classification module 53, configured to perform expression classification on the face image based on deep learning to obtain a facial expression label of the face image, where the facial expression label indicates an expression of the face;
a face verification module 54, configured to perform face verification on the face image to obtain a face verification result, where the face verification result indicates information of a user to which the face belongs;
and the display module 55 is configured to display the facial expression label and the facial verification result.
Optionally, the expression classification module 53 includes:
the first adjusting unit is used for adjusting the size of the face image to a first preset size;
the first segmentation unit is used for segmenting an image with a second preset size from N preset positions in the adjusted face image, wherein N is an integer larger than zero;
the prediction unit is used for inputting the N divided images into a Convolutional Neural Network (CNN) expression classification model for prediction to obtain a human face expression label of the human face image;
the prediction unit includes:
the predicting subunit is used for inputting the N divided images into a CNN expression classification model for prediction to obtain the prediction probability of each image of the N images of various human face expressions;
a calculating subunit, configured to calculate, according to the prediction probabilities of the multiple facial expressions in each of the N images, a mean value of the prediction probabilities of each of the multiple facial expressions in the N images;
and the determining subunit is used for taking the facial expression with the maximum average value of the prediction probabilities in the plurality of facial expressions as the facial expression label of the facial image.
Optionally, the face verification module 54 includes:
the second adjusting unit is used for adjusting the size of the face image to a third preset size;
the second segmentation unit is used for segmenting the adjusted face image into a plurality of images;
the image input unit is used for inputting the plurality of images into a face verification model and acquiring the classification probability of each user of the face in a facial expression database;
the first determining unit is used for determining that the user to which the face belongs is the user corresponding to the maximum value in the classification probabilities if the maximum value in the classification probabilities is larger than a preset threshold value;
a second determining unit, configured to determine that a user to which the face belongs does not exist in the facial expression database if a maximum value in the classification probabilities is smaller than or equal to a preset threshold;
and the adding unit is used for adding the facial expression label and the information of the user to which the face belongs to the facial expression database.
The facial expression recognition device provided by the embodiment of the present invention can be applied to the first, second, and third embodiments of the foregoing methods, and for details, reference is made to the description of the first, second, and third embodiments of the foregoing methods, and details are not described herein again.
Fig. 6 is a schematic diagram of a facial expression recognition apparatus according to a fifth embodiment of the present invention. As shown in fig. 6, the facial expression recognition apparatus 6 of this embodiment includes: a processor 60, a memory 61 and a computer program 62 stored in said memory 61 and executable on said processor 60. The processor 60, when executing the computer program 62, implements the steps in the above-described embodiments of the facial expression recognition method, such as the steps S101 to S105 shown in fig. 1. Alternatively, the processor 60, when executing the computer program 62, implements the functions of the modules/units in the above-mentioned device embodiments, such as the functions of the modules 51 to 55 shown in fig. 5.
Illustratively, the computer program 62 may be partitioned into one or more modules/units that are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 62 in the facial expression recognition apparatus 6. For example, the computer program 62 may be divided into an image acquisition module, a face extraction module, an expression classification module, a face verification module, and a display module, and each module has the following specific functions:
the image acquisition module is used for acquiring an image to be processed;
the face extraction module is used for extracting a face image from the image to be processed;
the expression classification module is used for carrying out expression classification on the face image based on deep learning to obtain a face expression label of the face image, wherein the face expression label indicates the expression of the face;
the face verification module is used for performing face verification on the face image to obtain a face verification result, wherein the face verification result indicates information of a user to which the face belongs;
and the display module is used for displaying the facial expression label and the facial verification result.
Optionally, the expression classification module includes:
the first adjusting unit is used for adjusting the size of the face image to a first preset size;
the first segmentation unit is used for segmenting an image with a second preset size from N preset positions in the adjusted face image, wherein N is an integer larger than zero;
the prediction unit is used for inputting the N divided images into a Convolutional Neural Network (CNN) expression classification model for prediction to obtain a human face expression label of the human face image;
the prediction unit includes:
the predicting subunit is used for inputting the N divided images into a CNN expression classification model for prediction to obtain the prediction probability of each image of the N images of various human face expressions;
a calculating subunit, configured to calculate, according to the prediction probabilities of the multiple facial expressions in each of the N images, a mean value of the prediction probabilities of each of the multiple facial expressions in the N images;
and the determining subunit is used for taking the facial expression with the maximum average value of the prediction probabilities in the plurality of facial expressions as the facial expression label of the facial image.
Optionally, the face verification module includes:
the second adjusting unit is used for adjusting the size of the face image to a third preset size;
the second segmentation unit is used for segmenting the adjusted face image into a plurality of images;
the image input unit is used for inputting the plurality of images into a face verification model and acquiring the classification probability of each user of the face in a facial expression database;
the first determining unit is used for determining that the user to which the face belongs is the user corresponding to the maximum value in the classification probabilities if the maximum value in the classification probabilities is larger than a preset threshold value;
a second determining unit, configured to determine that a user to which the face belongs does not exist in the facial expression database if a maximum value in the classification probabilities is smaller than or equal to a preset threshold;
and the adding unit is used for adding the facial expression label and the information of the user to which the face belongs to the facial expression database.
The facial expression recognition device 6 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The facial expression recognition device may include, but is not limited to, a processor 60 and a memory 61. It will be understood by those skilled in the art that fig. 6 is only an example of the facial expression recognition apparatus 6, and does not constitute a limitation to the facial expression recognition apparatus 6, and may include more or less components than those shown, or combine some components, or different components, for example, the facial recognition apparatus may further include an input-output device, a network access device, a bus, etc.
The Processor 60 may be a Central Processing Unit (CPU), and may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 61 may be an internal storage unit of the facial expression recognition apparatus 6, such as a hard disk or a memory of the facial expression recognition apparatus 6. The memory 61 may also be an external storage device of the facial expression recognition apparatus 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are equipped on the facial expression recognition apparatus 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the facial expression recognition apparatus 6. The memory 61 is used to store the computer program and other programs and data required by the facial expression recognition apparatus. The memory 61 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (7)
1. A facial expression recognition method is characterized by comprising the following steps:
acquiring an image to be processed;
extracting a face image from the image to be processed;
performing expression classification on the face image based on deep learning to obtain a face expression label of the face image, wherein the face expression label indicates the expression of the face;
performing face verification on the face image to obtain a face verification result, wherein the face verification result indicates information of a user to which the face belongs;
displaying the facial expression label and the facial verification result;
the expression classification is carried out on the face image based on the deep learning, and the obtaining of the face expression label of the face image comprises the following steps:
adjusting the size of the face image to a first preset size;
respectively segmenting an image with a second preset size from N preset positions in the adjusted face image, wherein N is an integer larger than zero;
inputting the N divided images into a Convolutional Neural Network (CNN) expression classification model for prediction to obtain a facial expression label of the facial image;
inputting the N segmented images into a Convolutional Neural Network (CNN) expression classification model for prediction, and acquiring a facial expression label of the facial image, wherein the facial expression label comprises:
inputting the N segmented images into a CNN expression classification model for prediction, and obtaining the prediction probability of each image of various human face expressions in the N images;
calculating the average value of the prediction probability of each facial expression in the N images according to the prediction probability of each image in the N images of the facial expressions;
and taking the facial expression with the maximum average value of the prediction probabilities in the plurality of facial expressions as a facial expression label of the facial image.
2. The method of claim 1, wherein the performing face verification on the face image and obtaining a face verification result comprises:
adjusting the size of the face image to a third preset size;
segmenting the adjusted face image into a plurality of images;
inputting the images into a face verification model, and acquiring the classification probability of each user of the face in a facial expression database;
and if the maximum value in the classification probabilities is larger than a preset threshold value, determining that the user to which the face belongs is the user corresponding to the maximum value in the classification probabilities.
3. The facial expression recognition method of claim 2, further comprising:
if the maximum value in the classification probabilities is smaller than or equal to a preset threshold value, determining that the user to which the face belongs does not exist in the facial expression database;
and adding the facial expression label and the information of the user to which the face belongs to the facial expression database.
4. A facial expression recognition apparatus, characterized in that the facial expression recognition apparatus comprises:
the image acquisition module is used for acquiring an image to be processed;
the face extraction module is used for extracting a face image from the image to be processed;
the expression classification module is used for carrying out expression classification on the face image based on deep learning to obtain a face expression label of the face image, wherein the face expression label indicates the expression of the face;
the face verification module is used for performing face verification on the face image to obtain a face verification result, wherein the face verification result indicates information of a user to which the face belongs;
the display module is used for displaying the facial expression label and the facial verification result;
the expression classification module comprises:
the first adjusting unit is used for adjusting the size of the face image to a first preset size;
the first segmentation unit is used for segmenting an image with a second preset size from N preset positions in the adjusted face image, wherein N is an integer larger than zero;
the prediction unit is used for inputting the N divided images into a Convolutional Neural Network (CNN) expression classification model for prediction to obtain a human face expression label of the human face image;
the prediction unit includes:
the predicting subunit is used for inputting the N divided images into a CNN expression classification model for prediction to obtain the prediction probability of each image of the N images of various human face expressions;
a calculating subunit, configured to calculate, according to the prediction probabilities of the multiple facial expressions in each of the N images, a mean value of the prediction probabilities of each of the multiple facial expressions in the N images;
and the determining subunit is used for taking the facial expression with the maximum average value of the prediction probabilities in the plurality of facial expressions as the facial expression label of the facial image.
5. The apparatus of claim 4, wherein the face verification module comprises:
the second adjusting unit is used for adjusting the size of the face image to a third preset size;
the second segmentation unit is used for segmenting the adjusted face image into a plurality of images;
the image input unit is used for inputting the plurality of images into a face verification model and acquiring the classification probability of each user of the face in a facial expression database;
the first determining unit is used for determining that the user to which the face belongs is the user corresponding to the maximum value in the classification probabilities if the maximum value in the classification probabilities is larger than a preset threshold value;
a second determining unit, configured to determine that a user to which the face belongs does not exist in the facial expression database if a maximum value in the classification probabilities is smaller than or equal to a preset threshold;
and the adding unit is used for adding the facial expression label and the information of the user to which the face belongs to the facial expression database.
6. A facial expression recognition apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the facial expression recognition method according to any one of claims 1 to 3 when executing the computer program.
7. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for facial expression recognition according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711402426.XA CN107958230B (en) | 2017-12-22 | 2017-12-22 | Facial expression recognition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711402426.XA CN107958230B (en) | 2017-12-22 | 2017-12-22 | Facial expression recognition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107958230A CN107958230A (en) | 2018-04-24 |
| CN107958230B true CN107958230B (en) | 2020-06-23 |
Family
ID=61956765
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711402426.XA Active CN107958230B (en) | 2017-12-22 | 2017-12-22 | Facial expression recognition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107958230B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108737729A (en) * | 2018-05-04 | 2018-11-02 | Oppo广东移动通信有限公司 | Automatic photographing method and device |
| CN108921042B (en) * | 2018-06-06 | 2019-08-23 | 四川大学 | A kind of face sequence expression recognition method based on deep learning |
| CN108921061B (en) * | 2018-06-20 | 2022-08-26 | 腾讯科技（深圳）有限公司 | Expression recognition method, device and equipment |
| CN109559193A (en) * | 2018-10-26 | 2019-04-02 | 深圳壹账通智能科技有限公司 | Product method for pushing, device, computer equipment and the medium of intelligent recognition |
| CN109711357A (en) * | 2018-12-28 | 2019-05-03 | 北京旷视科技有限公司 | A kind of face identification method and device |
| CN109934149B (en) * | 2019-03-06 | 2022-08-09 | 百度在线网络技术（北京）有限公司 | Method and apparatus for outputting information |
| CN110210399A (en) * | 2019-05-31 | 2019-09-06 | 广东世纪晟科技有限公司 | Face recognition method based on uncertainty quantization probability convolution neural network |
| CN110738598A (en) * | 2019-08-28 | 2020-01-31 | 咪咕文化科技有限公司 | Image adaptation method, electronic device and storage medium |
| CN110907909B (en) * | 2019-10-30 | 2023-09-12 | 南京市德赛西威汽车电子有限公司 | Radar target identification method based on probability statistics |
| CN111274447A (en) * | 2020-01-13 | 2020-06-12 | 深圳壹账通智能科技有限公司 | Target expression generation method, device, medium and electronic equipment based on video |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101620669A (en) * | 2008-07-01 | 2010-01-06 | 邹采荣 | Method for synchronously recognizing identities and expressions of human faces |
| CN103516518A (en) * | 2013-07-03 | 2014-01-15 | 北京百纳威尔科技有限公司 | Safety verification method and device |
| CN106257489A (en) * | 2016-07-12 | 2016-12-28 | 乐视控股（北京）有限公司 | Expression recognition method and system |
| CN106295566A (en) * | 2016-08-10 | 2017-01-04 | 北京小米移动软件有限公司 | Facial expression recognizing method and device |
| CN106339673A (en) * | 2016-08-19 | 2017-01-18 | 中山大学 | ATM identity authentication method based on face recognition |
| CN106778444A (en) * | 2015-11-23 | 2017-05-31 | 广州华久信息科技有限公司 | A kind of expression recognition method based on multi views convolutional neural networks |
| CN107480622A (en) * | 2017-08-07 | 2017-12-15 | 深圳市科迈爱康科技有限公司 | Micro- expression recognition method, device and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150235073A1 (en) * | 2014-01-28 | 2015-08-20 | The Trustees Of The Stevens Institute Of Technology | Flexible part-based representation for real-world face recognition apparatus and methods |
| US9552510B2 (en) * | 2015-03-18 | 2017-01-24 | Adobe Systems Incorporated | Facial expression capture for character animation |
-
2017
- 2017-12-22 CN CN201711402426.XA patent/CN107958230B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101620669A (en) * | 2008-07-01 | 2010-01-06 | 邹采荣 | Method for synchronously recognizing identities and expressions of human faces |
| CN103516518A (en) * | 2013-07-03 | 2014-01-15 | 北京百纳威尔科技有限公司 | Safety verification method and device |
| CN106778444A (en) * | 2015-11-23 | 2017-05-31 | 广州华久信息科技有限公司 | A kind of expression recognition method based on multi views convolutional neural networks |
| CN106257489A (en) * | 2016-07-12 | 2016-12-28 | 乐视控股（北京）有限公司 | Expression recognition method and system |
| CN106295566A (en) * | 2016-08-10 | 2017-01-04 | 北京小米移动软件有限公司 | Facial expression recognizing method and device |
| CN106339673A (en) * | 2016-08-19 | 2017-01-18 | 中山大学 | ATM identity authentication method based on face recognition |
| CN107480622A (en) * | 2017-08-07 | 2017-12-15 | 深圳市科迈爱康科技有限公司 | Micro- expression recognition method, device and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| Facial Expression Recognition with CNN Ensemble;Kuang Liu 等;《2016 International Conference on Cyberworlds》;20160930;163-166 * |
| Identity-Aware Convolutional Neural Network for Facial Expression Recognition;Zibo Meng 等;《2017 12th IEEE International Conference on Automatic Face & Gesture recognition》;20170603;1-8 * |
| 基于偏最小二乘回归的人脸身份和表情同步识别方法;周晓彦 等;《中国图象图形学报》;20090515;第14卷(第5期);801-808 * |
| 基于卷积神经网络的人脸表情识别;徐鹏 等;《人工智能》;20150625;第34卷(第12期);45-47 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107958230A (en) | 2018-04-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107958230B (en) | Facial expression recognition method and device | |
| CN107944020B (en) | Face image searching method and device, computer device and storage medium | |
| CN107633204B (en) | Face occlusion detection method, apparatus and storage medium | |
| US20190392587A1 (en) | System for predicting articulated object feature location | |
| CN110020592A (en) | Object detection model training method, device, computer equipment and storage medium | |
| CN107633207A (en) | AU characteristic recognition methods, device and storage medium | |
| CN107679447A (en) | Facial characteristics point detecting method, device and storage medium | |
| WO2019119396A1 (en) | Facial expression recognition method and device | |
| EP4047509A1 (en) | Facial parsing method and related devices | |
| CN111353385B (en) | Pedestrian re-identification method and device based on mask alignment and attention mechanism | |
| US20230334893A1 (en) | Method for optimizing human body posture recognition model, device and computer-readable storage medium | |
| CN111401339A (en) | Method and device for identifying age of person in face image and electronic equipment | |
| CN112052746A (en) | Target detection method and device, electronic equipment and readable storage medium | |
| CN105844204B (en) | Human behavior recognition method and device | |
| CN110175500B (en) | Finger vein comparison method, device, computer equipment and storage medium | |
| CN108875549A (en) | Image-recognizing method, device, system and computer storage medium | |
| Lahiani et al. | Hand pose estimation system based on Viola-Jones algorithm for android devices | |
| CN113627576B (en) | Code scanning information detection method, device, equipment and storage medium | |
| CN110717407A (en) | Human face recognition method, device and storage medium based on lip language password | |
| CN114639152A (en) | Multi-modal voice interaction method, device, equipment and medium based on face recognition | |
| CN110008922A (en) | Image processing method, unit, medium for terminal device | |
| CN113378852A (en) | Key point detection method and device, electronic equipment and storage medium | |
| CN112699907A (en) | Data fusion method, device and equipment | |
| CN111476775B (en) | DR symptom identification device and method | |
| CN115249377A (en) | Method and device for identifying micro-expression |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |