CN111523404A - Partial face recognition method based on convolutional neural network and sparse representation - Google Patents

Abstract

The invention discloses a partial face recognition method based on convolutional neural network and sparse representation, and a partial face recognition method based on convolutional neural network and sparse representation. To achieve sample expansion, use convolutional neural network to extract features from the training set and the images to be tested, obtain their corresponding feature maps to construct feature vectors, use sparse representation and sample correction methods to calculate residuals, and based on the score minimization criterion. sort. Compared with the prior art, the invention has the advantages of reducing the influence of changes such as occlusion, illumination and expression of the image to be measured, improving the accuracy of classification, extracting features more accurately, and effectively enhancing the robustness to different changes in the image. , which better solves the recognition problem with less face information.

Description

A partial face recognition method based on convolutional neural network and sparse representation

technical field

The invention relates to the technical field of face recognition, in particular to a partial face recognition method based on a convolutional neural network and sparse representation.

Background technique

Due to the wide application of artificial intelligence technology, face recognition has become a research hotspot in the field of computer vision and image processing in recent years. With the vigorous development of neural networks and deep learning, and the significant increase in computing power, face recognition has been applied in real life. Parkhi proposed VGGFace in 2015, which is based on VGGNet proposed by Simonyan and Zisserman in 2014, and has good transferability and excellent recognition effect. Most of the previous face recognition technologies are mainly for complete face images. In real life, due to the influence of factors such as occlusion, illumination, different postures and expressions, the face images to be recognized are often incomplete. Therefore, some face recognition is a hot issue worthy of discussion, and its recognition technology and recognition accuracy need to be improved urgently. Partial face images tend to lack more information than complete face images, and because of their small size, they are generally not directly matched with images in the image library. He et al. proposed dynamic feature matching based on Fully Convolutional Network (FCN), which has greatly improved the recognition accuracy, but when the image to be tested contains changes such as occlusion, illumination and expression, the robustness is not enough. it is good.

SUMMARY OF THE INVENTION

The purpose of the present invention is to design a partial face recognition method based on convolutional neural network and sparse representation designed for the deficiencies of the prior art. To improve the recognition accuracy, the convolutional neural network is used to extract features from the training set and the image to be tested, and the corresponding feature maps are obtained to construct feature vectors, and sparse representation and sample correction methods are used to calculate residuals. The input image is classified, and the unknown category of the image to be tested is predicted, which effectively reduces the influence of changes in the occlusion, illumination and expression of the image to be tested, and greatly improves the accuracy of the classification. The robustness of different changes can be applied to face recognition in various scenarios. The method is simple and the extracted features are more accurate. It can better solve some recognition problems with less face information, and has broad application prospects.

The specific technical scheme for realizing the purpose of the present invention is: a partial face recognition method based on convolutional neural network and sparse representation, which is characterized in that the mirror image of the image to be tested is added to realize sample expansion, and the convolutional neural network is used for training. Perform feature extraction on the set and the image to be tested, obtain its corresponding feature map to construct a feature vector, use the sparse representation and sample correction method to calculate the residual, and classify the input image based on the score minimization criterion. The specific process includes the following steps:

Step a: Make an image training set;

Step b: obtaining a mirror image of the input image to be tested for sample expansion;

Step c: Calculate residuals using sliding window, sparse representation and sample correction methods;

Step d: Repeat step c with the mirror image of the image to be tested to calculate the residual;

Step e: Combine the two residuals obtained after the sample expansion to calculate a new score, and predict the unknown category of the image to be tested based on the score minimization criterion.

What is required in the step a is a labeled sample image set (if necessary, the original image needs to be cropped to eliminate the interference of the background and only retain the complete face part), use CNN to extract the features of all training images, and construct a feature matrix.

In the step b, the input original image to be tested is mirror-inverted, and as a new image to be tested, its residual will be combined with the residual of the original image to be tested, and the two will jointly participate in the identification.

The local matching performed on the image to be tested in the step c includes the following steps:

Step c1 _: use CNN to generate the feature map and feature vector of the image to be tested;

Step c2 _: for each training image in the training set, use a sliding window to generate a subset of the same size as the feature map of the image to be tested, thereby obtaining a new training set;

Step c3: Calculate the sparse coefficient based _on the newly generated training set and the sparse representation method;

Step _c4 : Construct a change dictionary using the newly generated training set;

Step _c5 : Using the idea of sparse representation and linear additive model, subtract the corresponding change part existing in the change dictionary from the image to be tested to complete the sample correction;

Step _c6 : Calculate the residual between the image to be tested and each training image in the training set by using the sparse coefficient and the corrected image.

In the step e, the two residuals obtained after the sample expansion are combined to calculate a new score, and the specific steps are to assign different weights to the residuals of the original image to be tested and the mirror image respectively, and the two are weighted and added; Based on the score minimization criterion, the category of the image to be tested is predicted, and the specific steps are to calculate the minimum score between the image to be tested and each training image in the training set, and use the category of the corresponding training image as the prediction result.

Compared with the prior art, the invention has the advantages of reducing the influence of changes such as occlusion, illumination and expression of the image to be tested, improving the accuracy of classification, and the introduced change dictionary effectively enhances the robustness to different changes in the image, and can be applied For face recognition in a variety of scenes, the method is simple, the extracted features are more accurate, and the recognition problem with less face information is better solved, and it has a wide range of application prospects.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Figure 2 is an example of a training image;

Fig. 3 is an example diagram of an image to be tested;

4 is an example diagram of mirror images corresponding to the images to be tested respectively;

Figure 5 is a schematic diagram of the first sliding on the feature map of a training image using a sliding window.

Detailed ways

The present invention will be further described in detail below with reference to some specific embodiments of face recognition.

Referring to Fig. 1, the present invention includes four parts: making training set, sample expansion, sparse representation and fusion classification, and the specific steps of partial face recognition are as follows:

Step a: Make a training set of images

表示具有l_k张图的第k类训练图像，而且k＝[1，2，...，L]，

Refer to Figure 2, the image training set is a labeled sample image set. For the initial image with many background parts, it needs to be cropped according to the coordinates of the eyes and the facial features ratio of "three courts and five eyes" to eliminate background interference and keep the complete face as much as possible An image set of images, and unified image size. Then use CNN to extract the feature map of each original training image, and construct a feature matrix (convert into feature vector), and combine the feature vectors of N training images to form a training set G=[G ₁ , G ₂ ,...,G _N ]. If training images belonging to the same class are considered, assuming that there are L different classes in the training set, then the training set G can be rewritten as F=[F ₁ , F ₂ , . . . , F _L ], where

represents the k-th training image with lk images, and _k = [1, 2, ..., L],

Step b: Obtain a mirror image of the input image to be tested for sample expansion

Referring to Fig. 3, input the image to be tested y of part of the face image.

Referring to FIG. 4 , the image to be tested y is centered on the longitudinal axis of the image, and its left and right parts are mirror-transformed to obtain a mirror image y′ of the image to be tested y, so as to achieve sample expansion. Throughout the identification process, these two images are processed in exactly the same way, and the residuals of y and y' are weighted and summed to obtain the final score.

Step c: Calculate residuals using sliding window, sparse representation and sample correction methods

When the dimensions of the image to be tested y and the image in the training set are scaled to the same scale, the size of the feature map extracted from the image to be tested by CNN is often smaller than that of the image in the training set G.

Referring to FIG. 5 , the feature map p of the image to be tested slides for the first time on the feature map g ₁ of a training image using a sliding window. Using the sliding window to extract the feature map p of the image to be tested y, slide on the feature map of each image in the training set with the step size as step, and obtain M sub-feature maps of the same size, as a new training set G', where G ′ _{i =} [9 ₁ , 9 ₂ , . Width, take step=1, the smaller the value of step step, the finer the image features of the training set obtained through the sliding window.

Take the average value of the images belonging to the same class in G', construct the prototype dictionary P=[μ ₁ , μ ₂ , . . . , μ _L ]; then construct the change dictionary y=[F ₁ -μ ₁ e ₁ , F ₂ - μ ₂ e ₂ , ..., F _L - μ _L e _L ], where _ek is an _lk -dimensional row vector of all ones.

Based on the sparse representation method, the image to be tested p can be expressed as p=G′α,

d＝[d₁，d₂，...，d_N]，d_i＝||p-g_i||₂是待测图像与训练集G′中第i个图像的相似度；λ₁和λ₂是两个根据已有理论和方法设置的常数项，取λ₁＝3.1、λ₂＝0.4。where:

_{d = [ d 1} , _{d 2} , _. are two constant terms set according to existing theories and methods, and take λ ₁ =3.1 and λ ₂ =0.4.

The method based on sample correction gets:

λ₃是一个根据已有理论和方法设置的常数项，取λ₃＝3.6。where:

λ ₃ is a constant term set according to existing theories and methods, and takes λ ₃ =3.6.

Then, the residual r _i (y) is calculated according to the following formula a:

Step d: Repeat step c to calculate the residual _ri (y') using the mirror image y' of the image to be tested.

Step e: When calculating the weighted score, considering that the left and right halves of a person's face cannot be exactly the same, the residual weight of the image y to be tested is ω ₁ , and the residual weight of the mirror image y' is 1-ω ₁ is also different. , calculate the new score according to the following formula b:

score _i (y)=ω ₁ ×r _i (y)+(1-ω ₁ )× _ri (y′) (b);

Considering that the left and right halves of a person's face cannot be exactly the same, the original image to be tested should be paid more attention, so the weights of the residuals of y and y' are set as ω ₁ =0.6 and 1-0.6=0.4, respectively.

Based on the score minimization criterion, the label label of the image to be tested with the minimum score between the image to be tested and each training image in the training set is calculated according to the following formula c, and the category of the corresponding training image is used as the prediction result:

where z _i denotes the label of the ith image in the training set.

The above embodiments are only to further illustrate the present invention, and are not intended to limit the patent of the present invention. All equivalent implementations of the present invention should be included within the scope of the claims of the patent of the present invention.

Claims (6)

1. a partial face recognition method based on convolutional neural network and sparse representation, it is characterized in that adopting the mirror image that increases the image to be tested to realize sample expansion, utilizes the convolutional neural network to carry out feature extraction to training set and the image to be tested , obtain its corresponding feature map to construct a feature vector, use the sparse representation and sample correction method to calculate the residual, and classify the input image based on the score minimization criterion. The specific process includes the following steps: Step a: Make an image training set; Step b: obtaining a mirror image of the input image to be tested for sample expansion; Step c: Calculate residuals using sliding window, sparse representation and sample correction methods; Step d: use the mirror image of the image to be tested to repeat step c to calculate the residual; Step e: Combine the two residuals obtained after the sample expansion to calculate a new score, and predict the unknown category of the image to be tested based on the score minimization criterion. 2. The partial face recognition method based on convolutional neural network and sparse representation according to claim 1, wherein the image training set is a labeled sample image set, or the original image is trimmed to eliminate background interference, only A set of images with full face parts is preserved, features are extracted from all training images using CNN, and feature matrices are constructed. 3. the partial face recognition method based on convolutional neural network and sparse representation according to claim 1, it is characterized in that described sample expansion is that the original image to be tested of input is mirror-transformed to new image to be tested, and its residual The residuals of the original image to be tested are weighted and added together to participate in the identification. 4. the partial face recognition method based on convolutional neural network and sparse representation according to claim 1, it is characterized in that described utilizing sliding window, sparse representation and sample correction method to calculate residual error, and its concrete calculation comprises the steps: Step c1 _: use CNN to generate the feature map and feature vector of the image to be tested; Step c2 _: for each training image in the training set, use a sliding window to generate a subset of the same size as the feature map of the image to be tested, thereby obtaining a new training set; Step c3: Calculate the sparse coefficient based _on the newly generated training set and the sparse representation method; Step _c4 : Construct a change dictionary using the newly generated training set; Step _c5 : use the sparse representation and the linear additive model to subtract the corresponding change part existing in the change dictionary from the image to be tested to complete the sample correction; Step _c6 : Calculate the residual between the image to be tested and each training image in the training set by using the sparse coefficient and the corrected image. 5. the partial face recognition method based on convolutional neural network and sparse representation according to claim 1, it is characterized in that described two residuals are combined as the two residuals that assign different weights to original image to be tested and mirror image respectively Weighted addition of differences. 6. the partial face recognition method based on convolutional neural network and sparse representation according to claim 1, it is characterized in that the described image to be tested of unknown category is predicted to calculate the image to be tested and each training image in training set The minimum score between , and the class corresponding to the training image is used as the prediction result.

Images