CN115862097A - Method and device for identifying shielding face based on multi-attention and multi-scale feature learning - Google Patents

Abstract

The invention discloses a method and a device for identifying an occluded face based on multi-attention and multi-scale feature learning. The method is based on a channel attention and space attention mechanism, reduces the influence of a shelter on face recognition, and solves the problem that the face recognition accuracy is reduced under the situation of shelter. Firstly, adding a multi-level attention network on the basis of a traditional convolutional neural network, and extracting a channel attention diagram and a space attention diagram of a face image; secondly, constructing a multi-scale feature fusion device, and integrating local and global information of the face image; next, positioning an occlusion region and generating an occlusion mask through an occlusion mask generator, and reducing the influence of the occlusion region; and finally, carrying out shielding category classification and face identity classification through a multi-task learning network to obtain a final face recognition result. The method is simple and easy to implement, the model depth is shallow, the cost is low, and the identification efficiency is ensured while the accurate identification of the shielded face is realized.

Description

Occluded face recognition method and device based on multi-attention multi-scale feature learning

Technical Field

The present invention belongs to the field of artificial intelligence face recognition, and in particular relates to an occluded face recognition method and device based on multi-attention multi-scale feature learning.

Background Art

As a non-invasive identification and verification method, facial recognition is more popular and accepted by the public than other biometric technologies. With the development and advancement of recognition technology, facial recognition technology has been widely deployed in various scenarios such as monitoring systems, security systems, industrial production, home monitoring, etc., making it convenient for people in all aspects of their lives.

The accuracy of face recognition technology depends largely on the model's extraction of key facial features, and whether the facial area is complete has a great impact on the feature extraction process. With the global pandemic of the new coronavirus, wearing masks has become an indispensable requirement for people to travel. Masks, as external interference factors, cause occlusion of face images, which in turn damages some features. In this case, the commonly used face recognition algorithms will lose their high accuracy and ultimately fail to complete the mask face recognition task. Therefore, it is urgent to propose new research algorithms for the recognition of occluded faces.

Summary of the invention

The purpose of the present invention is to provide a method and device for occluded face recognition based on multi-attention multi-scale feature learning. The present invention extracts and fuses multi-level features of face images, and uses channel attention and spatial attention mechanisms to eliminate the influence of occluded areas on face recognition, thereby providing a method for effectively improving the accuracy of occluded face recognition. The present invention has simple logic and significant effects, and can effectively shield the adverse effects of partial occlusion on face recognition, while also supporting face recognition tasks in unobstructed scenes.

The present invention is achieved by adopting the following technical solutions:

An occluded face recognition method based on multi-attention and multi-scale feature learning is proposed. First, a multi-level attention network is added to the convolutional neural network to extract the channel attention map and spatial attention map of the face image; secondly, a multi-scale feature fuser is constructed to integrate the local and global information of the face image to obtain more robust face features; then, the occluded area is located and the occlusion mask is generated based on the occlusion mask generator to reduce the impact of the occluded area; finally, the occlusion category and face identity are classified simultaneously through the multi-task learning network to achieve the best face recognition generalization effect.

The present invention adopts an occluded face recognition method based on multi-attention multi-scale feature learning, which specifically includes the following steps:

Extract facial image features: Use residual neural network as facial image feature extraction module to extract facial image features;

Obtain channel attention map of face image: After obtaining the features of face image, calculate the attention weight for the feature map of each channel to obtain the relevance of each channel to the key information;

Obtain the spatial attention map of the face image: take the feature map after the channel attention map is refined as input, calculate the correlation between different pixels and key information, and obtain the multi-level features of the face image;

Multi-scale feature fusion of face images: Based on the multi-level features of face images, a three-layer deconvolution structure is used to build a multi-scale feature fusion device. By adding feature maps of different scales element by element, multi-scale feature information of face images with different resolutions and semantic strengths is obtained.

Occlusion mask generation: Learn feature masks that are highly sensitive to the occlusion positions of the input image, calculate the weights corresponding to the occluded areas, and eliminate the impact of damaged features on face recognition by assigning different weights to features;

Occlusion category classification: The feature mask learned by the occlusion mask generator is taken as input and classified into occlusion categories to supervise the learning of the occlusion mask generator;

Face category classification: Multiply the regional weights obtained by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face features, which are used as the input of the face category classifier to obtain the classification result of the face category.

A further improvement of the present invention is to obtain a channel attention map of a face image, specifically including:

The extracted facial features are subjected to average pooling and maximum pooling operations to aggregate spatial information, and then input into two shared fully connected layers to fit the correlation between channel features, thus obtaining two channel feature maps.

The corresponding elements in the two channel feature maps are added and processed using the Sigmoid activation function to obtain the channel attention map of the face image. The weight in the map reflects the degree of relevance of the channel to the key information.

A further improvement of the present invention is to obtain a spatial attention map of a face image, including:

Perform maximum pooling and average pooling operations on the extracted facial features along the channel direction to obtain two spatial feature maps;

The two spatial feature maps are spliced together, and the feature correlation in the spatial dimension is fitted through convolution operation to obtain a spatial attention map. The weights in the map reflect the degree of correlation between different pixels and key information.

A further improvement of the present invention is that the multi-scale feature fusion of the face image specifically includes:

The facial image feature extraction module is used as the main body of the multi-scale feature fusion device, and a pyramid structure model is constructed using a top-to-bottom horizontal connection architecture.

The input of the pyramid structure model is a preprocessed face image, and multi-scale feature information of the face image with different resolutions and semantic strengths is obtained through convolution and upsampling operations.

A further improvement of the present invention is that the occlusion mask generation specifically includes:

The input contains facial features of different scales and global information, and the final occlusion mask is obtained through the convolutional network combined with the PReLu activation function, batch normalization layer and Sigmoid function to clean up the original facial features damaged by partial occlusion.

A further improvement of the present invention is that the occlusion category classification specifically includes:

The face image is divided into several rectangular grids, and the occlusion area is simulated by combining the rectangles and a new occlusion category is constructed. Based on this, an occlusion dictionary of all occlusion categories is obtained, which still includes the unoccluded case.

Select different types of mask images as occluders, randomly select the occluder center and integrate the occluder image on the face image;

Calculate the corresponding occlusion matrix according to whether each square is occluded, and look up the corresponding occlusion category in the generated occlusion dictionary as the label of the occluded face image;

The labeled occluded face images are fed into the occlusion mask generator to learn the mask associated with the occlusion category;

The learned mask is fed into the occlusion category classifier for classification, and the cross entropy is used as the loss function to supervise the learning process of the occlusion mask generator to obtain a more accurate occlusion mask.

A further improvement of the present invention is that the face category classification specifically includes:

Input the facial features after occlusion mask processing, and use the edge-based loss function LMCL supervised model to learn the identity-related facial features;

Finally, the loss function of the face recognition task and the loss function of the occlusion category recognition task are added together as the final loss function to supervise the model so that it converges faster and completes the face category classification.

The occluded face recognition device based on multi-attention multi-scale feature learning comprises:

The face image feature extraction module uses a residual neural network as the face image feature extraction module to extract face image features;

The channel attention map construction module calculates the attention weight for the feature map of each channel after obtaining the facial image features, and obtains the correlation between each channel and the key information;

The spatial attention map construction module takes the feature map refined by the channel attention map as input, calculates the correlation between different pixels and key information, and obtains the multi-level features of the face image;

The multi-scale feature fusion module uses a three-layer deconvolution structure to construct a multi-scale feature fuser for the multi-level features of facial images. By adding feature maps of different scales element by element, multi-scale feature information of facial images with different resolutions and semantic strengths is obtained.

The occlusion mask generation module learns feature masks that are highly sensitive to the occlusion positions of the input image, calculates the weights corresponding to the occluded areas, and eliminates the impact of damaged features on face recognition by assigning different weights to the features;

The occlusion category classification module takes the feature mask learned by the occlusion mask generator as input and classifies it into occlusion categories to supervise the learning of the occlusion mask generator;

The face category classification module multiplies the regional weight obtained by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face features, and uses them as the input of the face category classifier to obtain the classification result of the face category.

The present invention has at least the following beneficial technical effects:

The present invention provides an occluded face recognition method based on multi-attention multi-scale feature learning. The method first adds a multi-level attention mechanism on the basis of the face feature extraction network to extract the channel attention map and spatial attention map of the face image; secondly, a multi-scale feature fuser is constructed to integrate the local and global information of the image; next, the occluded area is located and the occlusion mask is generated by the occlusion mask generator to reduce the influence of the occluded area; finally, the occlusion category is classified and the face identity is classified by a multi-task learning network to obtain the final face recognition result. Compared with ordinary face recognition algorithms, this method can reduce the influence of occluders such as masks on the accuracy of face recognition. Experiments show that the method achieves an accuracy of 97.76% in the occluded face task, which is better than the existing occluded face recognition algorithm. In the unoccluded face recognition task, its accuracy is comparable to that of the existing method.

The present application also provides an occluded face recognition device based on multi-attention multi-scale feature learning, which includes seven modules: face feature extraction module, channel attention map construction module, spatial attention map construction module, multi-scale feature fusion module, occlusion mask generation module, occlusion classification module, and face classification module. The face feature extraction module provides deep face features for subsequent modules; the channel attention and spatial attention modules provide extraction methods for channel attention maps and spatial attention maps; the multi-scale feature fusion module can fuse local and global information of the image to obtain multi-scale feature information of the face image that is more conducive to recognition; the occlusion mask generation module can locate the occlusion area and generate the occlusion mask, reduce the influence of the occlusion area, and improve the classification accuracy; the occlusion classification module is used to classify the occlusion mask and effectively supervise the occlusion mask generation process; the final face classification module can achieve effective classification of face categories by inputting the cleaned face features, thereby improving the classification efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is an overall processing flow chart of the occluded face recognition algorithm based on the attention mechanism;

FIG2 is a schematic diagram of the channel attention map extraction process;

FIG3 is a schematic diagram of the spatial attention map extraction process;

Fig. 4 is a schematic diagram of a multi-level feature fusion module;

Figure 5 is the confusion matrix of different algorithms on the LFW and Occ-LFW datasets;

FIG6 is a functional module diagram of an occluded face recognition device based on multi-attention scale feature learning provided by the present invention;

FIG7 is a schematic diagram of the structure of an electronic device for implementing the occluded face recognition algorithm based on multi-attention scale feature learning provided by the present invention.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to be able to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

Referring to Figure 1, the present invention provides occluded face recognition based on multi-attention multi-scale feature learning. The method first extracts features from the input face image, and calculates the channel attention map and spatial attention map of the face image based on the multi-level attention mechanism; then obtains feature information containing different resolutions and semantic strengths through a multi-scale feature fuser; the occlusion category recognition task branch learns the occlusion mask of the face image through an occlusion mask generator, and sends this mask to the occlusion category classifier to supervise the learning process of the mask generator; the face recognition task branch superimposes the occlusion mask learned by the occlusion category recognition task branch on the original face features to eliminate the adverse effects of occlusion on recognition, and completes the occluded face recognition task. Specifically includes the following modules:

1. Multi-level feature extraction of face images: including face image feature extraction, channel attention map acquisition and spatial attention map acquisition. The specific steps are as follows:

Step 1: Extract facial image features: Use residual neural network as facial image feature extraction network to extract facial image features;

Step 2: Obtain the channel attention map of the face image: Referring to Figure 2, after obtaining the features of the face image, calculate the attention weight for the feature map of each channel to obtain the degree of relevance between each channel and the key information; the specific steps are as follows:

分别进行平均池化和最大值池化聚合空间信息，得到特征图

和

其次，将

和

输入两个共享的全连接层，以拟合通道特征之间的相关性：First, for the input

Average pooling and maximum pooling are performed to aggregate spatial information and obtain feature maps.

and

Secondly,

and

Enter two shared fully connected layers to fit the correlation between channel features:

其中r为压缩比，以减少参数量。In the formula, (W ₀ ,b ₀ ,W ₁ ,b ₁ ) represent the weights and biases of the two fully connected layers respectively.

Where r is the compression ratio to reduce the number of parameters.

Finally, the corresponding elements of equations (1) and (2) are added and processed using the Sigmoid activation function to obtain the final channel attention map _Mc (F). The process is shown in the following equation.

即Sigmoid激活函数。In the formula,

That is the Sigmoid activation function.

Step 3: Obtain the spatial attention map of the face image: Referring to Figure 3, take the feature map after the channel attention map is refined as input and calculate the correlation between different pixels and key information. The specific steps are as follows:

输出为空间权重图M_s。首先，沿着通道方向对F进行最大池化和平均池化操作，得到特征图

和

然后，将平均池化结果

和最大池化结果

进行拼接，得到新的通道数为2的数据，并将其送入一层卷积神经网络以拟合空间维度上的特征相关性。该过程下式所示。The input of this module is the feature map after the channel attention map is refined.

The output is the spatial weight map _Ms. First, perform maximum pooling and average pooling operations on F along the channel direction to obtain the feature map

and

Then, the average pooling result is

And the maximum pooling result

After concatenation, new data with 2 channels is obtained, which is then fed into a convolutional neural network to fit the feature correlation in the spatial dimension. The process is shown in the following formula.

代表卷积核为k₁×k₂的卷积操作。in,

Represents a convolution operation with a convolution kernel of k ₁ ×k ₂ .

2. Multi-scale feature fusion of face images: Referring to Figure 4, the face image feature extraction module in step 1 is used as the main body of the multi-scale feature fusion device, and a pyramid structure is constructed using a top-to-bottom horizontal connection architecture. The model takes the processed face image as input and outputs face features x ₁ , x ₂ , x ₃ of different scales. Among them, x ₁ is the underlying face recognition feature that needs to be cleaned, and x ₂ , x ₃ contain local and global information of different scales. The process can be formally expressed as follows

x ₂ =conv(upsample(conv(x ₁ ))+conv(C ₂ )) (5)

x ₃ =conv(upsample(conv(x ₂ ))+conv(C ₃ )) (6)

Among them, conv is the convolution operation and upsample is the upsampling operation.

3. Occlusion mask generation: Learn a feature mask that is highly sensitive to the occlusion position of the input image, calculate the weights corresponding to the occluded area, and eliminate the impact of damaged features on face recognition by assigning different weights to the features.

4. Multi-task occluded face classification model: It includes two subtasks, face category classification and occlusion category classification. The specific steps are as follows:

Step 1 Occlusion Category Classifier: Take the feature mask learned by the occlusion mask generator as input and classify it into occlusion categories to supervise the learning of the occlusion mask generator.

Step 2 Face category classification: Multiply the regional weight calculated by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face features, which are used as the input of the face category classifier to obtain the face category classification result.

Referring to Table 1, compared with other algorithms in the field of face recognition, the algorithm proposed in the present invention is basically consistent with Arcface in terms of unobstructed face comparison, and is 0.4% higher than the FROM algorithm in the field of occluded face recognition. In terms of occluded face comparison, the accuracy of the algorithm in this paper is 1.2% higher than Arcface and 1% higher than the occluded face recognition algorithm FROM. This shows the effectiveness of the occluded face recognition algorithm based on the attention mechanism in face comparison.

Table 1: Comparison of face accuracy of the present invention and other face recognition algorithms ArcFace and FROM on LFW and Occ-LFW datasets.

In order to further evaluate the algorithm compared with other algorithms, the confusion matrices of the three algorithms on the LFW dataset and the Occ-LFW dataset were further obtained, refer to Figure 5. On the LFW dataset, FROM is most likely to confuse the same face pairs with different face pairs, while Arcface is least likely to confuse the two. In terms of same face comparison, the performance of the present invention is better than FROM and similar to that of Arcface; in terms of different face comparison, the performance of the present invention is significantly better than FROM, but slightly worse than Arcface. On the Occ-LFW dataset, Arcface is most likely to confuse the same face pairs with different face pairs, while the present invention is least likely to confuse the two. In terms of same face comparison and different face comparison, the performance of the present invention is better than FROM and Arcface, which fully demonstrates the effectiveness of the occluded face recognition algorithm based on the attention mechanism.

The occluded face recognition device based on multi-attention and multi-scale feature learning provided by the present invention comprises a face feature extraction module, a channel attention map construction module, a spatial attention map construction module, a multi-scale feature fusion module, an occlusion mask generation module, an occlusion classification module and a face classification module.

1. Face image feature extraction module, using residual neural network as face image feature extraction module to extract face image features;

2. Channel attention map construction module, after obtaining the facial image features, calculates the attention weight for the feature map of each channel to obtain the degree of relevance between each channel and the key information;

3. The spatial attention map construction module takes the feature map after the channel attention map is refined as input, calculates the correlation between different pixels and key information, and obtains the multi-level features of the face image;

4. Multi-scale feature fusion module: Based on the multi-level features of face images, a multi-scale feature fusion module is constructed using a three-layer deconvolution structure. By adding feature maps of different scales element by element, multi-scale feature information of face images with different resolutions and semantic strengths is obtained;

5. The occlusion mask generation module learns the feature mask that is highly sensitive to the occlusion position of the input image, calculates the weight corresponding to the occlusion area, and eliminates the influence of damaged features on face recognition by assigning different weights to the features;

6. The occlusion category classification module takes the feature mask learned by the occlusion mask generator as input and classifies it into occlusion categories to supervise the learning of the occlusion mask generator;

7. The face category classification module multiplies the regional weight obtained by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face features, and uses them as the input of the face category classifier to obtain the classification result of the face category.

Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims (9)

1. An occluded face recognition method based on multi-attention multi-scale feature learning is characterized in that, firstly, a multi-level attention network is added on the basis of a convolutional neural network to extract the channel attention map and spatial attention map of the face image; secondly, a multi-scale feature fuser is constructed to integrate the local and global information of the face image to obtain more robust face features; then, based on the occlusion mask generator, the occlusion area is located and the occlusion mask is generated to reduce the influence of the occlusion area; finally, the occlusion category classification and face identity classification are simultaneously performed through a multi-task learning network to achieve the best face recognition generalization effect. 2. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 1, characterized in that the method specifically comprises the following steps: Extract facial image features: Use residual neural network as facial image feature extraction module to extract facial image features; Obtain channel attention map of face image: After obtaining the features of face image, calculate the attention weight for the feature map of each channel to obtain the relevance of each channel to the key information; Obtain the spatial attention map of the face image: take the feature map after the channel attention map is refined as input, calculate the correlation between different pixels and key information, and obtain the multi-level features of the face image; Multi-scale feature fusion of face images: Based on the multi-level features of face images, a three-layer deconvolution structure is used to build a multi-scale feature fusion device. By adding feature maps of different scales element by element, multi-scale feature information of face images with different resolutions and semantic strengths is obtained. Occlusion mask generation: Learn feature masks that are highly sensitive to the occlusion positions of the input image, calculate the weights corresponding to the occluded areas, and eliminate the impact of damaged features on face recognition by assigning different weights to features; Occlusion category classification: The feature mask learned by the occlusion mask generator is taken as input and classified into occlusion categories to supervise the learning of the occlusion mask generator; Face category classification: Multiply the regional weights obtained by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face features, which are used as the input of the face category classifier to obtain the classification result of the face category. 3. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 2 is characterized in that obtaining a channel attention map of a face image specifically comprises: The extracted facial features are subjected to average pooling and maximum pooling operations to aggregate spatial information, and then input into two shared fully connected layers to fit the correlation between channel features to obtain two channel feature maps; The corresponding elements in the two channel feature maps are added and processed using the Sigmoid activation function to obtain the channel attention map of the face image. The weight in the map reflects the degree of relevance of the channel to the key information. 4. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 2, characterized in that obtaining a spatial attention map of a face image comprises: Perform maximum pooling and average pooling operations on the extracted facial features along the channel direction to obtain two spatial feature maps; The two spatial feature maps are concatenated, and the feature correlation in the spatial dimension is fitted through convolution operation to obtain a spatial attention map. The weights in the map reflect the degree of correlation between different pixels and key information. 5. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 2 is characterized in that the multi-scale feature fusion of the face image specifically includes: The facial image feature extraction module is used as the main body of the multi-scale feature fusion device, and a pyramid structure model is constructed using a top-to-bottom horizontal connection architecture. The input of the pyramid structure model is a preprocessed face image, and multi-scale feature information of the face image with different resolutions and semantic strengths is obtained through convolution and upsampling operations. 6. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 5, characterized in that the occlusion mask generation specifically comprises: The input contains facial features of different scales and global information, and the final occlusion mask is obtained through the convolutional network combined with the PReLu activation function, batch normalization layer and Sigmoid function to clean up the original facial features damaged by partial occlusion. 7. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 2 is characterized in that the occlusion category classification specifically includes: The face image is divided into several rectangular grids, and the occlusion area is simulated by combining the rectangles and a new occlusion category is constructed. Based on this, an occlusion dictionary of all occlusion categories is obtained, which still includes the unoccluded case. Select different types of mask images as occluders, randomly select the occluder center and integrate the occluder image on the face image; Calculate the corresponding occlusion matrix according to whether each square is occluded, and look up the corresponding occlusion category in the generated occlusion dictionary as the label of the occluded face image; The labeled occluded face images are fed into the occlusion mask generator to learn the mask associated with the occlusion category; The learned mask is fed into the occlusion category classifier for classification, and the cross entropy is used as the loss function to supervise the learning process of the occlusion mask generator to obtain a more accurate occlusion mask. 8. The occluded face recognition method based on multi-attention multi-scale feature learning according to claim 2, characterized in that the face category classification specifically includes: Input the facial features after occlusion mask processing, and use the edge-based loss function LMCL supervised model to learn the identity-related facial features; Finally, the loss function of the face recognition task and the loss function of the occlusion category recognition task are added together as the final loss function to supervise the model so that it converges faster and completes the face category classification. 9. An occluded face recognition device based on multi-attention multi-scale feature learning, characterized by comprising: The face image feature extraction module uses a residual neural network as the face image feature extraction module to extract face image features; The channel attention map construction module calculates the attention weight for the feature map of each channel after obtaining the facial image features, and obtains the correlation between each channel and the key information; The spatial attention map construction module takes the feature map refined by the channel attention map as input, calculates the correlation between different pixels and key information, and obtains the multi-level features of the face image; The multi-scale feature fusion module uses a three-layer deconvolution structure to construct a multi-scale feature fuser for the multi-level features of facial images. By adding feature maps of different scales element by element, multi-scale feature information of facial images with different resolutions and semantic strengths is obtained. The occlusion mask generation module learns feature masks that are highly sensitive to the occlusion positions of the input image, calculates the weights corresponding to the occluded areas, and eliminates the impact of damaged features on face recognition by assigning different weights to features; The occlusion category classification module takes the feature mask learned by the occlusion mask generator as input and classifies it into occlusion categories to supervise the learning of the occlusion mask generator; The face category classification module multiplies the regional weight obtained by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face features, and uses them as the input of the face category classifier to obtain the classification result of the face category.

Images