CN110175588A - A kind of few sample face expression recognition method and system based on meta learning - Google Patents

Abstract

The present disclosure proposes a kind of few sample face expression recognition method and system based on meta learning, comprising: receive facial sample set, carry out data prediction；Construct the master cast of Expression Recognition；Facial sample set is divided based on the method for meta learning: for seven class expressions, randomly choosing three classes expression and is used for training set, then random selection three classes expression is used for testing set in remaining four class expressions；Using the method construct C-way K-shot of episode-based, i.e., sample set and query set are constructed using ready-portioned training set；The facial sample set constructed is inputted into master cast, parameter optimization is carried out to master cast；Face data to be identified is received, human facial expression recognition is carried out according to the master cast after optimization.The disclosure mainly considers few sample face Expression Recognition problem based on meta learning, using the Research on Methods training set of episode-based, constructs a convolutional neural networks and extracts the knowledge of transferability to realize the identification to expression.

Description

A Few-Sample Facial Expression Recognition Method and System Based on Meta-learning

technical field

The present disclosure relates to the technical field of image recognition of computer vision, in particular to a method and system for recognizing facial expressions with few samples based on meta-learning.

Background technique

With the development of artificial intelligence and deep learning technology, facial expression recognition, as an important problem in the field of image recognition, has received more and more attention. Facial expression is one of the most favorable, natural and common signals for people to convey their emotions and intentions, and facial expression recognition is based on seven facial expressions (angry, disgusted, scared, happy, sad, surprised and neutral) to determine human emotion. Facial expression recognition, as one of the deep learning technologies, has made great progress and is widely used in many fields such as mentality analysis, medical diagnosis, and advertising effect research.

The inventor found in the research that in recent years, deep learning technology has made great progress in the field of computer vision, such as target detection, image segmentation and image classification. Deep neural networks can automatically extract advanced semantic features from input images end-to-end, which is considered to be one of the artificial intelligence technologies most likely to be close to human level. These supervised learning models require a large amount of labeled data and multiple iterations to train a large number of model parameters. Due to the time-consuming and labor-intensive labeling, this limits the scalability when facing new categories. More importantly, the applicability is limited due to the absence of a large number of labeled samples for emerging or rare categories. Therefore, how to better use a small number of labeled samples to train a classification model with high recognition accuracy and good generalization performance is a very meaningful research direction, that is, the study of the few-sample problem.

Few-sample learning problems generally contain three data sets: training set, support set, and query set. For the support set containing C-type samples, and each type of sample contains K samples, it is called the C-way K-shot problem. On the contrary, humans are very good at recognizing objects without direct supervision information, even categories that have not appeared before, which is the innate ability of meta-learning.

Contents of the invention

The purpose of the embodiment of this description is to provide a method for facial expression recognition with few samples based on meta-learning. The model trained by this method can use very few training samples to achieve satisfactory recognition accuracy, thereby solving the need for training models. The time-consuming and labor-intensive problem of a large number of labeled samples.

The implementation mode of this specification provides a method for recognizing facial expressions with few samples based on meta-learning, which is realized through the following technical solutions:

include:

Receive the face sample set and perform data preprocessing;

Construct the main model of expression recognition;

The data set is divided based on the meta-learning method: for the seven types of expressions, three types of expressions are randomly selected for the training set, and then three types of expressions are randomly selected for the test set from the remaining four types of expressions;

Use the episode-based method to construct C-way K-shot, that is, use the divided training set to construct sample set and query set;

Input the constructed face sample set into the main model, and optimize the parameters of the main model;

Receive facial data to be recognized, and perform facial expression recognition based on the optimized master model.

The implementation mode of this specification provides a few-sample facial expression recognition system based on meta-learning, which is realized through the following technical solutions:

include:

The data preprocessing module is configured to: receive a face sample set and perform data preprocessing;

The main model building module is configured to: build a main model for facial expression recognition;

The data set construction module is configured to: divide the data set based on the method of meta-learning: for seven types of expressions, randomly select three types of expressions for training set, and then randomly select three types of expressions from the remaining four types of expressions for testing set;

Use the episode-based method to construct C-way K-shot, that is, use the divided training set to construct sample set and query set;

The main model optimization module is configured to: input the constructed face sample set into the main model, and optimize the parameters of the main model;

The facial expression recognition module is configured to: receive facial data to be recognized, and perform facial expression recognition according to the optimized main model.

Compared with the prior art, the beneficial effects of the present disclosure are:

This disclosure mainly considers the problem of few-sample facial expression recognition based on meta-learning, explores the training set with an episode-based method, and constructs a convolutional neural network to extract transferable knowledge to realize expression recognition.

This disclosure considers that the original facial expression recognition requires a large amount of labeled data, and labeling is time-consuming and labor-intensive. How to use a small amount of labeled data more effectively, and for the research of few-sample expression recognition, this disclosure will use meta-learning The training strategy introduced to facial expression recognition can use a small number of facial expression samples to identify sample types that have not appeared in the training process; use the episode-based exploration method for the training set to make full use of the training set more effectively, More efficient extraction of transitive knowledge leads to better performance in the support set and better classification of the testing set.

Description of drawings

The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure, and the exemplary embodiments and descriptions of the present disclosure are used to explain the present disclosure, and do not constitute improper limitations to the present disclosure.

Fig. 1 is a flow chart of a method for recognizing facial expressions with few samples based on meta-learning according to one or more embodiments;

Figure 2(a)-Figure 2(b) is a schematic diagram of the main model of the facial expression recognition network according to one or more embodiments;

Fig. 3 is a schematic diagram of a framework of a facial expression recognition method according to one or more embodiments.

Detailed ways

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the present disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It should be noted that the terminology used herein is only for describing specific embodiments, and is not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

Implementation example one

As shown in FIG. 1 , according to an aspect of one or more embodiments of the present disclosure, a flow chart of a meta-learning-based method for recognizing facial expressions with few samples is provided.

A meta-learning based few-sample facial expression recognition method, the method comprising:

S101: Receive face sample set, carry out data preprocessing;

S102: Construct the network main model of expression recognition;

S103: Divide the data set based on the method of meta-learning to form training and support/query set, and use the method of episode-based to construct the sample set and query set that satisfy the C-way K-shot;

S104: Input the preprocessed sample into the main network model, and optimize the model;

S105: Receive facial data to be recognized, and perform facial expression recognition according to the optimized model.

In step S101 of this embodiment, the face sample data in the face sample set is a face sample picture, and performing the data preprocessing on the face sample picture includes normalizing each face sample picture and performing normalization on each face sample picture. Each pixel is normalized. The specific operation steps for data preprocessing in this embodiment:

S1011 normalize each picture: subtract the mean value from each picture, and then set the standard deviation to 3.125;

S1012 Normalize each pixel: first calculate a mean pixel value picture, and then subtract the mean pixel value corresponding to each picture; then set the standard deviation of each pixel of all training set pictures to 1.

In step S102 of the present embodiment, as shown in Fig. 2 (a)-Fig. 2 (b), described facial expression recognition network master model comprises 4 Convolution Block blocks and last Flatten layer of series connection;

The Convolution Block block mainly includes: a convolutional layer, a batch normalization layer, a Relu activation function, and a pooling layer. Parameter settings of the convolution layer: the convolution kernel is 3*3, padding=SAME; the pooling layer adopts the maximum pooling.

The Flatten layer is used to one-dimensionalize the features extracted by the training model to obtain n=64-dimensional features.

Figure 3 shows the framework of a few-sample facial expression recognition method based on meta-learning: the main innovation in the whole method lies in the research on few-sample facial expression recognition, which is mainly divided into three parts: data set construction, feature extraction, and model optimization.

Few sample learning is divided into three data sets: training, support, test. Among them, training is used for training, support is used for verification, and test is used for final model testing. In the training process, use training to construct the sample set and query set in the training process, which are used for the expansion of the next algorithm.

The specific steps of constructing the S103 dataset are:

S1031 uses the original data set to construct training/support/testing according to the idea of meta-learning (that is, training and support/test have different label spaces, and support/test has the same label space). In line with the standard of maximizing the use of the dataset, the dataset is divided as follows: randomly select three types of expressions from the seven types of expressions for the training set, and randomly select three types of expressions from the remaining four types of expressions for support/query set, the sample categories used for training and testing are not the same each time the program is run.

In the implementation examples of this application, the slashes uniformly represent "and".

Using the episode-based method to construct C-way K-shot, select C category among the seven categories of expressions, select K samples in each category to construct the sample set in the training process, and select q samples in the C categories Samples are used to construct the queryset.

During specific implementation, S1032 uses an episode-based method to construct a sample set and a query set that satisfy the C-way K-shot. Among the three types of expression pictures with C=3, select K pictures, K=1 or K>1 to construct the sampleset; among the remaining pictures of the three categories, select q pictures to construct the query set, q=5 ,15,20 etc.

Feature extraction is to obtain the 64-dimensional features of the input samples through the output of the main network model through the Flattern layer.

S104 Model optimization is to use the designed loss function to optimize the model parameters using the stochastic gradient descent SGD method to minimize the total loss. The specific process is as follows:

S1041 Calculate the prototype of each type of sample in the sample set according to the extracted features. In the 3-wayK-shot sample set: if K=1, the prototype of this type of sample is the 64-dimensional feature p _k = f( _xi ) of the sample; if K>1, according to the principle of Bregman divergence , then the prototype of the kth class sample is Among them, N _s is the number of samples selected for each type of expression, S _k is the set of samples of this type in the sample set, ( _xi , y _i ) is the labeled sample in the set, f( _xi ) is the Extracted features.

S1042 Calculate the distance d(x _q ,p _k ) between the sample x _q in the query set and each type of prototype p _k in turn, and then calculate the possibility of belonging to this type according to the distance to the type of prototype The calculation of the distance here can use the Euclidean distance or the cosine distance.

S1043 Use the loss function L(x _q )=-logp(x _q ,p _k ), and optimize using the stochastic gradient descent method SGD.

Table 1 gives the pseudocode of data set construction and model optimization.

Table 1

Implementation Example 2

According to an aspect of one or more embodiments of the present disclosure, a meta-learning based few-shot facial expression recognition device is provided.

A meta-learning based few-sample facial expression recognition device, based on a meta-learning few-sample facial expression recognition method, comprising: sequentially connected data preprocessing module, main model building module, data set building module, model Optimization module and facial expression recognition module.

The data preprocessing module is used to receive a face sample set and perform data preprocessing;

Described main model construction module is used for constructing facial expression recognition network main model;

The data set construction module is used to construct the training set and support set/testing set based on meta-learning, so that the training set and support set/test set have different label spaces; the C-way K for constructing episode-based -shot sample set and query set;

The model optimization module is used to optimize the model according to the constructed data set, according to the determined loss function and stochastic gradient descent method.

The facial expression recognition module is used to receive facial data to be recognized, and perform facial expression recognition according to the optimized model.

It should be noted that although several modules or sub-modules of the device have been mentioned in the above detailed description, this division is only exemplary and not mandatory. Actually, according to an embodiment of the present disclosure, the features and functions of two or more modules described above may be embodied in one module. Conversely, the features and functions of one module described above may be further divided to be embodied by a plurality of modules.

Implementation example three

A computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor implements a meta-learning-based few-sample facial expression when executing the program Identify method steps.

For the steps of a meta-learning-based few-sample facial expression recognition method in this embodiment example, please refer to the specific technical content in Embodiment 1, and will not be described in detail here.

Implementation Example 4

A computer-readable storage medium, on which a computer program is stored, is characterized in that, when the program is executed by a processor, the steps of a meta-learning-based few-sample facial expression recognition method are implemented.

For the steps of a meta-learning-based few-sample facial expression recognition method in this embodiment example, please refer to the specific technical content in Embodiment 1, and will not be described in detail here.

It can be understood that, in the description of this specification, references to the terms "an embodiment", "another embodiment", "other embodiments", or "the first embodiment to the Nth embodiment" mean that A specific feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (10)

1. a kind of few sample face expression recognition method based on meta learning, characterized in that include:

Facial sample set is received, data prediction is carried out；

Construct the master cast of Expression Recognition；

Facial sample set is divided based on the method for meta learning: for seven class expressions, randomly choosing three classes expression and is used for Training set, then random selection three classes expression is used for testing set in remaining four class expressions；

Using the method construct C-way K-shot of episode-based, i.e., constructed using ready-portioned training set Sample set and query set；

The facial sample set constructed is inputted into master cast, parameter optimization is carried out to master cast；

Face data to be identified is received, human facial expression recognition is carried out according to the master cast after optimization.

2. a kind of few sample face expression recognition method based on meta learning as described in claim 1, characterized in that the face Facial sample data in portion's sample set is facial samples pictures, and carrying out data prediction to facial samples pictures includes to every width Facial samples pictures are normalized and each pixel in facial samples pictures are normalized.

3. a kind of few sample face expression recognition method based on meta learning as claimed in claim 2, characterized in that every width Facial samples pictures are normalized: subtracting average value from every picture, standard deviation is then arranged.

4. a kind of few sample face expression recognition method based on meta learning as claimed in claim 2, characterized in that face Each pixel in samples pictures is normalized: a mean value pixel value picture is calculated first, then for each width picture Subtract the mean value pixel of corresponding position；Then the standard deviation of each pixel of all training set pictures is set.

5. a kind of few sample face expression recognition method based on meta learning as described in claim 1, characterized in that the table The master cast of feelings identification includes 4 concatenated Convolution Block blocks and the last one Flatten layers；

The Convolution Block block specifically includes that convolutional layer, batch normalization layer, Relu activation primitive, pond layer；

The parameter setting of convolutional layer: convolution kernel 3*3, padding=SAME mode；Pond layer uses maximum value pond；

The Flatten layer is used for the feature one-dimensional for extracting training pattern, obtains the feature of n=64 dimension.

6. a kind of few sample face expression recognition method based on meta learning as described in claim 1, characterized in that utilize The method of episode-based, construction meets sample set and the query set of C-way K-shot, in the three classes of C=3 In expression picture, select K picture, K=1 or K > 1 for constructing sample set；In the remaining picture of the three classes, q is selected Picture is for constructing query set.

7. a kind of few sample face expression recognition method based on meta learning as described in claim 1, characterized in that expression is known Other master cast exports to obtain the feature of 64 dimensions of input sample by the Flattern number of plies；

According to the feature of extraction, the prototype of every a kind of sample in sample set is calculated, in the sample of 3-way K-shot In set: if K=1, then the prototype of such sample is the feature p of 64 dimension of sample_k=f (x_i)；If K > 1, according to Bregman The principle of divergence, then the prototype of kth class sample beWherein K is the sample that every a kind of expression is chosen Number, S_kIt is sample set, f (x_i) it is the feature extracted by master cast；

Successively calculate the sample x in query set_qWith every a kind of prototype p_kDistance d (x_q,p_k), then basis arrives such prototype The far and near of distance calculates a possibility that belonging to such, and the calculating of distance herein can use Euclidean distance or COS distance.

Using loss function L (x_q)=- logp (x_q,p_k), it is optimized using stochastic gradient descent method SGD.

8. a kind of few sample face Expression Recognition system based on meta learning, characterized in that include:

Data preprocessing module is configured as: being received facial sample set, is carried out data prediction；

Master cast constructs module, is configured as: constructing the master cast of Expression Recognition；

Dataset construction module, is configured as: the method based on meta learning divides data set: for seven class expressions, with Machine selects three classes expression to be used for training set, then random selection three classes expression is used for testing in remaining four class expressions set；

Using the method construct C-way K-shot of episode-based, i.e., constructed using ready-portioned training set Sample set and query set；

Master cast optimization module, is configured as: the facial sample set constructed being inputted master cast, it is excellent to carry out parameter to master cast Change；

Human facial expression recognition module, is configured as: receiving face data to be identified, carries out facial table according to the master cast after optimization Feelings identification.

9. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor Calculation machine program, which is characterized in that the processor realizes that one kind as claimed in claim 1 to 7 is based on when executing described program Few sample face expression recognition method step of meta learning.

10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor A kind of few sample face expression recognition method step based on meta learning as claimed in claim 1 to 7 is realized when execution.