CN112836680A - Visual sense-based facial expression recognition method - Google Patents
Visual sense-based facial expression recognition method Download PDFInfo
- Publication number
- CN112836680A CN112836680A CN202110234838.7A CN202110234838A CN112836680A CN 112836680 A CN112836680 A CN 112836680A CN 202110234838 A CN202110234838 A CN 202110234838A CN 112836680 A CN112836680 A CN 112836680A
- Authority
- CN
- China
- Prior art keywords
- sample set
- expression recognition
- expression
- classifier
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000008921 facial expression Effects 0.000 title claims abstract description 39
- 230000000007 visual effect Effects 0.000 title claims abstract description 12
- 230000014509 gene expression Effects 0.000 claims abstract description 102
- 238000012549 training Methods 0.000 claims abstract description 41
- 238000012360 testing method Methods 0.000 claims abstract description 22
- 238000007781 pre-processing Methods 0.000 claims abstract description 17
- 238000013528 artificial neural network Methods 0.000 claims description 34
- 238000004364 calculation method Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 11
- 230000001815 facial effect Effects 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 210000004709 eyebrow Anatomy 0.000 claims description 6
- 210000002569 neuron Anatomy 0.000 claims description 6
- 239000011521 glass Substances 0.000 claims description 2
- 230000010365 information processing Effects 0.000 abstract description 2
- 230000008859 change Effects 0.000 description 13
- 239000013598 vector Substances 0.000 description 13
- 238000005516 engineering process Methods 0.000 description 8
- 239000011159 matrix material Substances 0.000 description 6
- 230000036544 posture Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 210000001508 eye Anatomy 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 206010011469 Crying Diseases 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 210000000214 mouth Anatomy 0.000 description 1
- 210000003739 neck Anatomy 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 210000001331 nose Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biomedical Technology (AREA)
- Evolutionary Biology (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention provides a visual-based facial expression recognition method. The present invention relates to the field of information processing. The invention comprises the following steps: acquiring a face image sample set, and preprocessing the face image sample set; extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set; introducing the expression feature sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier; and putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier. The invention can identify the expression degree of the facial expression.
Description
Technical Field
The invention relates to the field of information processing, in particular to a visual-based facial expression recognition method.
Background
In recent years, with the rise of artificial intelligence technology and the great promotion of computer technology, people put forward higher and higher requirements on human-computer interaction tasks. Meanwhile, in the communication between people, not only the language symbols, but also the facial expressions and other body languages are the components for transmitting information.
At present, because human expressions have different degrees, for example, smiling expressions can be classified into laughing, smiling and the like, crying expressions can be classified into choking, crying and the like, and the existing expression recognition systems cannot recognize different degrees of the same expression.
Disclosure of Invention
The invention aims to provide a visual-based facial expression recognition method, which can recognize the expression degree of facial expressions.
The embodiment of the invention is realized by the following steps:
the embodiment of the application provides a visual-based facial expression recognition method, which comprises the following steps of;
acquiring a face image sample set, and preprocessing the face image sample set;
extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set;
introducing the expression feature sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier;
and putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In some embodiments of the present invention, the obtaining a face image sample set, and the preprocessing the face image sample set includes converting color images of the face image sample set into grayscale images:
wherein, RGB is the color component of each color image pixel point, and g is the converted gray value.
In some embodiments of the present invention, the expression feature samples obtained by extracting the features of the preprocessed face image sample set comprise face geometric features.
In some embodiments of the present invention, the geometric features of the human face include glasses, eyebrows, nose, and mouth.
In some embodiments of the present invention, the change of the expressive feature comprises a change of a texture shape of a geometric feature of the human face.
In some embodiments of the present invention, the facial geometric feature extraction method of the expression feature samples of the expression feature sample set is based on the spatiotemporal features of the difference image.
In some embodiments of the present invention, the expression recognition classifier is a BP neural network classifier.
In some embodiments of the present invention, the BP neural network classifier takes the expression feature samples as input, performs linear combination in the BP neural network, and outputs each neuron through a nonlinear activation function, where each neuron obtains a calculation result.
In some embodiments of the present invention, the operation process of the BP neural network classifier includes two stages:
forward propagation: firstly, inputting an expression feature sample set into an input layer, performing weighting calculation through a hidden layer, and finally outputting the expression feature sample set by an output layer, wherein the former layer is equivalent to the input layer of the next layer in the processing of each layer;
and (3) back propagation: when the information is transmitted to an output layer, comparing the result with a given label, judging whether a convergence condition is reached, and if so, ending the training process; if not, the layer-by-layer backward propagation is carried out, and the weights are sequentially adjusted until the convergence condition is met.
In some embodiments of the invention, the activation function:
where x is the actual output.
Compared with the prior art, the embodiment of the invention has at least the following advantages or beneficial effects:
a vision-based facial expression recognition method comprises the following steps;
acquiring a face image sample set, and preprocessing the face image sample set;
extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set;
introducing the expression feature sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier;
and putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In the above embodiment, a visual-based facial expression recognition method includes four steps in total, first acquiring a face image sample set, and preprocessing the face image sample set, wherein, the method for preprocessing the face image sample set is to convert color images in RGB space into gray level images, since the color of each pixel in a color image is determined by R, G, B three components, and each component has 256 values, therefore, the color change range of 1600 or more tens of thousands of such a pixel point can be obtained, the storage box is calculated to be burdened by too large data volume, and the change range of one pixel point of the gray image is only 256 cases, therefore, by converting the face image sample into the gray level image, the subsequent image processing calculation amount is greatly reduced, meanwhile, the gray-scale image can still reflect the distribution and the characteristics of the overall and local chromaticity and brightness levels of the whole image which is consistent from the color image. Extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set, wherein the expression characteristic sample is used as an obvious and effective strategy in order to identify the type and the change condition of the face expression; introducing the expression characteristic sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier, wherein the expression recognition classifier can adopt a BP neural network classifier, the BP neural network classifier is a learning model based on an artificial neural network structure, the network weight can be continuously modified by setting a plurality of layers of hidden layers and in a reverse back propagation way, therefore, the purpose of learning is achieved, 5000 human face pictures are adopted as a training sample level in the application, 5000 human face pictures are selected as a test sample set, in order to ensure the training effect, after each training, the training result is compared with the last training result, if the error is increased, the weight value is adjusted towards the negative direction, if the error is decreased, the weight value is adjusted towards the positive direction, therefore, the identification accuracy is continuously improved, and the training process is completed when the preset training times or the preset convergence condition is reached. And finally, putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In the embodiment, different degrees of the same expression are distinguished more accurately according to facial expression recognition, facial expression recognition is achieved through four steps, image preprocessing adopts a method of converting RGB space into gray level images, feature extraction adopts texture analysis technology of gray level co-occurrence matrixes, expression space-time extraction based on difference images is achieved, a BP neural network method is adopted as a classifier, the BP neural network has three layers, feature vectors are used as input, expressions after fine classification are used as coding output, a learning model is established through continuous learning training, a convolutional neural network is constructed by combining a deep learning technology, and accuracy of expression recognition is further improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a flowchart of a visual sense-based facial expression recognition method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the individual features of the embodiments can be combined with one another without conflict.
Examples
Please refer to fig. 1. The method for recognizing the facial expression based on the vision comprises the following steps of;
acquiring a face image sample set, and preprocessing the face image sample set;
extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set;
introducing the expression feature sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier;
and putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In the above embodiment, a visual-based facial expression recognition method includes four steps in total, first acquiring a face image sample set and preprocessing the face image sample set, wherein,
because other body parts such as shoulders, necks and the like can be acquired while the three-dimensional face is scanned, the face needs to be separated to remove non-face redundant information, the face data is acquired, because factors such as the posture, the distance and the like can have the difference of the size and the inclination angle, in order to reduce the difference of the two aspects, the face needs to be normalized and aligned, in order to determine the normalized coordinate value of each point cloud data of the face, the nose cusp point is selected as a characteristic reference point in the embodiment, and therefore the original three-dimensional point data is preprocessed through four steps of nose cusp positioning, face normalization, posture correction and depth map acquisition; the nose tip is used as a local highest point and has a top cap shape to extract nose tip points, for each point P, the adjacent points can be regarded as being distributed on a sphere with the point as the center of circle, and effective energy (P) is definedi-P) and NpInner product of (d); di＝(Pi-P)*Np＝(Pi-P)cosɑ；NpRepresenting two normalized normals, P being the point to be selected, its three-dimensional coordinates and normal vector NpIs known as PiFor the data points at the peak top, the angle a between the normal line and the adjacent point is greater than 90 °, so the effective energy d isiNegative values, this constraint is not sufficient to detect the nose tip, since many points of the cheek, chin, etc. also satisfy this condition, but this step greatly reduces the search range; then calculate the energy diMean values μ and Be of statistical properties of2Variance of (a):
all points fall into two categories: nasal and non-nasal cusps, (x)i,yi),i＝1,2… …, l are training samples, where xiIs a two-dimensional vector (mu, Be)2)，yiE { -1,1} represents a class, and the support vector SVM is combined to distinguish two classes by finding the minimum value of an energy function:
wherein the kernel function K
For test vector x, the function class is determined by,
b*is a parameter obtained by optimizing an energy function, and L is the number of the obtained support vectors; when the nose tip point is detected, the nose tip point is positioned at the origin of coordinates, so that all three-dimensional face models in a database are aligned, and meanwhile, the size range of the face is determined by taking the nose tip point as the center according to the theory of three five eyes, the face is normalized to be the standard size, and the size of the normalized face is 130 to 100. although the collected object is required to be kept as still as possible when the three-dimensional face data is collected, the face has many degrees of freedom in three-dimensional space, the input postures at different angles can increase the recognition difficulty, the posture correction is a step of setting all input three-dimensional faces under the same coordinate, and the posture correction coordinate is constructed based on the steps: suppose face point cloud V is a set of N vertices, V ═ Vi∈R3|1≤I≤N}；
Calculating the mass center of all point clouds V of the human face:
constructing a coefficient matrix of point cloud vertex distribution:
the face curved surface has unequal length, the coefficient C is subjected to characteristic decomposition, and 3 characteristic values lambda with different sizes can be obtained1≥λ2≥λ3And correspondingEigenvector Be1，б2，б3(ii) a Mixing O withvAs origin, is1As the Y axis, is2As the X-axis, is3As the Z-axis, a pose coordinate system PCS is defined, in which all faces have the same frontal pose, and the original point cloud is transformed to a new coordinate space:
the preprocessed three-dimensional point cloud can be mapped to a two-dimensional plane through orthogonal projection, the depth information of the point cloud is obtained, and corresponding color information is converted into a gray map. The gray scale image method is to convert the color image in the RGB space into a gray scale image, because the color of each pixel in the color image is determined by R, G, B components, and each component has 256 value conditions, so that one pixel point can have 1600 more than ten thousand color change ranges, the data volume is too large, which causes a certain burden to the calculation of a storage box, and one pixel point change range of the gray scale image has only 256 conditions, so that the human face image sample is converted into the gray scale image, the subsequent image processing calculation amount is greatly reduced, and simultaneously, the gray scale image can still reflect the distribution and the characteristics of the integral and local chromaticity and brightness levels of the whole image consistent with the color image. Extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set, wherein the expression characteristic sample is used as an obvious and effective strategy in order to identify the type and the change condition of the face expression; introducing the expression characteristic sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier, wherein the expression recognition classifier can adopt a BP neural network classifier, the BP neural network classifier is a learning model based on an artificial neural network structure, the network weight can be continuously modified by setting a plurality of layers of hidden layers and in a reverse back propagation way, therefore, the purpose of learning is achieved, 5000 human face pictures are adopted as a training sample level in the application, 5000 human face pictures are selected as a test sample set, in order to ensure the training effect, after each training, the training result is compared with the last training result, if the error is increased, the weight value is adjusted towards the negative direction, if the error is decreased, the weight value is adjusted towards the positive direction, therefore, the identification accuracy is continuously improved, and the training process is completed when the preset training times or the preset convergence condition is reached. And finally, putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In the embodiment, different degrees of the same expression are distinguished more accurately according to facial expression recognition, facial expression recognition is achieved through four steps, image preprocessing adopts a method of converting RGB space into gray level images, feature extraction adopts texture analysis technology of gray level co-occurrence matrixes, expression space-time extraction based on difference images is achieved, a BP neural network method is adopted as a classifier, the BP neural network has three layers, feature vectors are used as input, expressions after fine classification are used as coding output, a learning model is established through continuous learning training, a convolutional neural network is constructed by combining a deep learning technology, and accuracy of expression recognition is further improved.
In some embodiments of the present invention, a face image sample set is obtained, and the pre-processing of the face image sample set comprises converting color images of the face image sample set into grayscale images:
wherein, RGB is the color component of each color image pixel point, and g is the converted gray value.
In this embodiment, since the color of each pixel in the color image is determined by R, G, B components, and each component has 256 value cases, such a pixel point can have 1600 or more tens of thousands of color change ranges, the data size is too large, which causes a certain burden on the calculation of the storage box, and the change range of a pixel point of the gray image is only 256 cases, so that the human face image sample is converted into the gray image, thereby greatly reducing the subsequent image processing calculation amount, and simultaneously, the gray image can still reflect the distribution and the characteristics of the overall and local chromaticity and brightness levels of the whole image consistent with the color image.
In some embodiments of the present invention, the facial feature samples obtained by extracting the features of the preprocessed facial image sample set are composed of facial geometric features.
In this embodiment, the geometric features of the human face refer to the position changes of various organs, eyes, eyebrows, nose, mouth, and the like on the human face, and further include the position changes of the canthus, eyebrow tip, and mouth corner.
In some embodiments of the invention, the geometric features of the human face include eyes, eyebrows, nose, and mouth.
In this embodiment, the eyes, eyebrows, nose, and mouth can significantly and effectively express facial expressions.
In some embodiments of the invention, the changes in expressive features include changes in the shape of the texture of the geometric features of the face.
In this embodiment, the texture analysis uses a spatial gray level co-occurrence matrix method, and obtains a matrix by calculating the number of times that two gray levels are adjacent in a certain direction in an image, where the direction generally includes horizontal, 45 °, 90 °, and 135 °.
In some embodiments of the present invention, the facial geometric feature extraction method of the expression feature samples of the expression feature sample set passes through the spatiotemporal features of the differential image.
In this embodiment, the data set of the facial expression video may extract a feature vector from a facial image of each frame in the obtained video, where the feature vector is used for geometric features of expression recognition, and refers to changes of several major organs on the face, and may directly perform a difference operation on each frame image and a neutral expression frame in a video sequence, and use the facial expression texture features extracted by the gray level co-occurrence matrix method, where the difference operation refers to performing matrix subtraction on gray level co-occurrence matrices of two frames of images, and then expanding a new gray level co-occurrence matrix into a vector in a row to obtain a facial geometric feature vector.
In some embodiments of the invention, the expression recognition classifier is a BP neural network classifier.
In this embodiment, the BP neural network is a learning model based on an artificial neural network structure, and the learning training purpose can be achieved by setting multiple hidden layers and continuously modifying the network weight in a reverse back propagation manner.
In some embodiments of the present invention, the BP neural network classifier performs linear combination in the BP neural network using the expression feature samples as input, and outputs each neuron through a nonlinear activation function, where each neuron obtains a calculation result.
In some embodiments of the present invention, the working process of the BP neural network classifier comprises two phases:
forward propagation: firstly, inputting an expression feature sample set into an input layer, performing weighting calculation through a hidden layer, and finally outputting the expression feature sample set by an output layer, wherein the former layer is equivalent to the input layer of the next layer in the processing of each layer;
and (3) back propagation: when the information is transmitted to an output layer, comparing the result with a given label, judging whether a convergence condition is reached, and if so, ending the training process; if not, the layer-by-layer backward propagation is carried out, and the weights are sequentially adjusted until the convergence condition is met.
In this embodiment, the BP neural network is a learning model based on an artificial neural network structure, and the learning training purpose can be achieved by setting multiple hidden layers and continuously modifying the network weight in a reverse back propagation manner.
In some embodiments of the invention, the activation function:
where x is the actual output.
In this embodiment, in order to adjust the weight in the back propagation process, an error function is determined before the learning training, and the square sum of the difference between the actual output x and the expected output y is used to represent:
after a large amount of learning and training, the BP neural network reduces errors according to continuously adjusted network weights, and therefore expected effects are achieved, in order to guarantee accuracy, the BP neural network adopts a three-layer neural network structure, wherein an input layer is arranged, the input layer comprises 10 nodes and corresponds to a group of facial expression feature input vectors, a hidden layer comprises 10 nodes, and 9 nodes in total in the output layer represent output results of 9 expressions.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application.
In summary, the method for recognizing facial expressions based on vision provided by the embodiment of the application includes the following steps;
acquiring a face image sample set, and preprocessing the face image sample set;
extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set;
introducing the expression feature sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier;
and putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In the above embodiment, a visual-based facial expression recognition method includes four steps in total, first acquiring a face image sample set, and preprocessing the face image sample set, wherein, the method for preprocessing the face image sample set is to convert color images in RGB space into gray level images, since the color of each pixel in a color image is determined by R, G, B three components, and each component has 256 values, therefore, the color change range of 1600 or more tens of thousands of such a pixel point can be obtained, the storage box is calculated to be burdened by too large data volume, and the change range of one pixel point of the gray image is only 256 cases, therefore, by converting the face image sample into the gray level image, the subsequent image processing calculation amount is greatly reduced, meanwhile, the gray-scale image can still reflect the distribution and the characteristics of the overall and local chromaticity and brightness levels of the whole image which is consistent from the color image. Extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set, wherein the expression characteristic sample is used as an obvious and effective strategy in order to identify the type and the change condition of the face expression; introducing the expression characteristic sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier, wherein the expression recognition classifier can adopt a BP neural network classifier, the BP neural network classifier is a learning model based on an artificial neural network structure, the network weight can be continuously modified by setting a plurality of layers of hidden layers and in a reverse back propagation way, therefore, the purpose of learning is achieved, 5000 human face pictures are adopted as a training sample level in the application, 5000 human face pictures are selected as a test sample set, in order to ensure the training effect, after each training, the training result is compared with the last training result, if the error is increased, the weight value is adjusted towards the negative direction, if the error is decreased, the weight value is adjusted towards the positive direction, therefore, the identification accuracy is continuously improved, and the training process is completed when the preset training times or the preset convergence condition is reached. And finally, putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
In the embodiment, different degrees of the same expression are distinguished more accurately according to facial expression recognition, facial expression recognition is achieved through four steps, image preprocessing adopts a method of converting RGB space into gray level images, feature extraction adopts texture analysis technology of gray level co-occurrence matrixes, expression space-time extraction based on difference images is achieved, a BP neural network method is adopted as a classifier, the BP neural network has three layers, feature vectors are used as input, expressions after fine classification are used as coding output, a learning model is established through continuous learning training, a convolutional neural network is constructed by combining a deep learning technology, and accuracy of expression recognition is further improved.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (10)
1. A vision-based facial expression recognition method is characterized by comprising the following steps;
acquiring a face image sample set, and preprocessing the face image sample set;
extracting the characteristics of the preprocessed face image sample set to obtain an expression characteristic sample set;
introducing the expression feature sample set into an expression recognition classifier for learning training to obtain a trained expression recognition classifier;
and putting the test sample into the trained expression recognition classifier, and evaluating the test sample by the expression recognition classifier.
2. A vision-based facial expression recognition method according to claim 1, wherein the face image sample set is obtained, and the preprocessing of the face image sample set comprises converting color images of the face image sample set into grayscale images:
r, G, B is the color component of each color image pixel point, and g is the converted gray value.
3. The vision-based facial expression recognition method of claim 1, wherein the expression feature samples obtained by extracting the features of the preprocessed facial image sample set are composed of facial geometric features.
4. A vision-based facial expression recognition method as claimed in claim 3, wherein the geometric features of the human face include glasses, eyebrows, nose and mouth.
5. A visual-based facial expression recognition method as claimed in claim 3 or 4, wherein the changes in the expressive features include changes in the texture and shape of geometric features of the face.
6. The vision-based facial expression recognition method of claim 3, wherein the facial geometric feature extraction method of the expression feature samples of the expression feature sample set is based on the spatiotemporal features of the difference image.
7. The vision-based facial expression recognition method of claim 1, wherein the expression recognition classifier is a BP neural network classifier.
8. The vision-based facial expression recognition method of claim 7, wherein the BP neural network classifier takes expression feature samples as input, linear combination is performed in the BP neural network, each neuron outputs through a nonlinear activation function, and each neuron obtains a calculation result.
9. The vision-based facial expression recognition method of claim 8, wherein the working process of the BP neural network classifier comprises two stages:
forward propagation: firstly, inputting an expression feature sample set into an input layer, performing weighting calculation through a hidden layer, and finally outputting the expression feature sample set by an output layer, wherein the former layer is equivalent to the input layer of the next layer in the processing of each layer;
and (3) back propagation: when the information is transmitted to an output layer, comparing the result with a given label, judging whether a convergence condition is reached, and if so, ending the training process; if not, the layer-by-layer backward propagation is carried out, and the weights are sequentially adjusted until the convergence condition is met.
10. A method for visual-based facial expression recognition as recited in claim 8, wherein the activation function:
where x is the actual output.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110234838.7A CN112836680A (en) | 2021-03-03 | 2021-03-03 | Visual sense-based facial expression recognition method |
| PCT/CN2022/079035 WO2022184133A1 (en) | 2021-03-03 | 2022-03-03 | Vision-based facial expression recognition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110234838.7A CN112836680A (en) | 2021-03-03 | 2021-03-03 | Visual sense-based facial expression recognition method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112836680A true CN112836680A (en) | 2021-05-25 |
Family
ID=75934489
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110234838.7A Pending CN112836680A (en) | 2021-03-03 | 2021-03-03 | Visual sense-based facial expression recognition method |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112836680A (en) |
| WO (1) | WO2022184133A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022184133A1 (en) * | 2021-03-03 | 2022-09-09 | 郑州航空工业管理学院 | Vision-based facial expression recognition method |
| CN115331292A (en) * | 2022-08-17 | 2022-11-11 | 武汉元紫东科技有限公司 | Facial image-based emotion recognition method and device and computer storage medium |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117038055B (en) * | 2023-07-05 | 2024-04-02 | 广州市妇女儿童医疗中心 | Pain assessment method, system, device and medium based on multi-expert model |
| CN116825365B (en) * | 2023-08-30 | 2023-11-28 | 安徽爱学堂教育科技有限公司 | Mental health analysis method based on multi-angle micro-expression |
| CN117275070A (en) * | 2023-10-11 | 2023-12-22 | 中邮消费金融有限公司 | Video facial mask processing method and system based on micro-expressions |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108304826A (en) * | 2018-03-01 | 2018-07-20 | 河海大学 | Facial expression recognizing method based on convolutional neural networks |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10417483B2 (en) * | 2017-01-25 | 2019-09-17 | Imam Abdulrahman Bin Faisal University | Facial expression recognition |
| CN110110653A (en) * | 2019-04-30 | 2019-08-09 | 上海迥灵信息技术有限公司 | The Emotion identification method, apparatus and storage medium of multiple features fusion |
| CN112836680A (en) * | 2021-03-03 | 2021-05-25 | 郑州航空工业管理学院 | Visual sense-based facial expression recognition method |
-
2021
- 2021-03-03 CN CN202110234838.7A patent/CN112836680A/en active Pending
-
2022
- 2022-03-03 WO PCT/CN2022/079035 patent/WO2022184133A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108304826A (en) * | 2018-03-01 | 2018-07-20 | 河海大学 | Facial expression recognizing method based on convolutional neural networks |
Non-Patent Citations (3)
| Title |
|---|
| 梁瑞奇: "基于神经网络的人脸表情识别", 《电子制作》, no. 20, pages 1 * |
| 董瑞霞: "三维人脸数据的预处理及深度图获取", 《福建电脑》, vol. 33, no. 2, pages 1 * |
| 董薇: "基于3D面部数据及人工智能算法的情感数据采集分析与心理预警系统", 《电子设计工程》, vol. 28, no. 5, pages 152 - 156 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022184133A1 (en) * | 2021-03-03 | 2022-09-09 | 郑州航空工业管理学院 | Vision-based facial expression recognition method |
| CN115331292A (en) * | 2022-08-17 | 2022-11-11 | 武汉元紫东科技有限公司 | Facial image-based emotion recognition method and device and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022184133A1 (en) | 2022-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112836680A (en) | Visual sense-based facial expression recognition method | |
| US20210174072A1 (en) | Microexpression-based image recognition method and apparatus, and related device | |
| CN106960202B (en) | Smiling face identification method based on visible light and infrared image fusion | |
| CN110348330B (en) | Face pose virtual view generation method based on VAE-ACGAN | |
| Ramanathan et al. | Face verification across age progression | |
| CN107169455B (en) | Face attribute recognition method based on depth local features | |
| CN108629336B (en) | Face characteristic point identification-based color value calculation method | |
| US20230044644A1 (en) | Large-scale generation of photorealistic 3d models | |
| CN107832740B (en) | Teaching quality assessment method and system for remote teaching | |
| CN110909680A (en) | Facial expression recognition method and device, electronic equipment and storage medium | |
| CN111783748A (en) | Face recognition method and device, electronic equipment and storage medium | |
| CN112784736A (en) | Multi-mode feature fusion character interaction behavior recognition method | |
| CN110991258B (en) | Face fusion feature extraction method and system | |
| CN113239839B (en) | Expression recognition method based on DCA face feature fusion | |
| CN111028319A (en) | Three-dimensional non-photorealistic expression generation method based on facial motion unit | |
| CN107194364B (en) | Huffman-L BP multi-pose face recognition method based on divide and conquer strategy | |
| CN111652798A (en) | Human face pose migration method and computer storage medium | |
| CN115205933A (en) | Facial expression recognition method, device, equipment and readable storage medium | |
| CN107895154B (en) | Method and system for forming facial expression intensity calculation model | |
| CN117333604A (en) | Character face replay method based on semantic perception nerve radiation field | |
| CN114219920B (en) | Method and device for constructing three-dimensional face model, storage medium and terminal | |
| CN113468923B (en) | Human-object interaction behavior detection method based on fine-grained multi-modal common representation | |
| Lv et al. | A 3D face recognition method using region-based extended local binary pattern | |
| KR102472110B1 (en) | Face image generation system based on emotion and method thereof | |
| CN114862716A (en) | Image enhancement method, device and equipment for face image and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |