CN106778589A - A kind of masked method for detecting human face of robust based on modified LeNet - Google Patents

Abstract

A Robust Masked Face Detection Method Based on Improved LeNet, which involves masked face detection. Including the following steps: 1) expanding the training data by flipping the original training picture horizontally; 2) proposing a new MLeNet model by modifying the structure of the traditional LeNet model to make it suitable for the detection of masked humans. The specific method can be: Adjust the size of the convolution kernel and the number of feature maps. In addition, change the number of nodes in the original output layer from 10 to 2 to make it suitable for the 2-category problem of human detection; 3) Borrow the parameters in the original LeNet model to pre-train the MLeNet structure , and fine-tune the MLeNet model to obtain a detector suitable for masked faces; 4) Combine the sliding window and non-maximization suppression technology to accurately locate the masked face. It can accurately detect the face of the masked person, and the model still has strong robustness under the interference conditions such as scattered background and environmental changes.

Description

A Robust Masked Face Detection Method Based on Improved LeNet

technical field

The invention relates to masked human face detection, in particular to a robust masked human face detection method based on an improved LeNet.

Background technique

With the development of society, the improvement of science and technology, and the popularization of multimedia technology, more and more people are uploading various online videos on the Internet, including many criminals trying to use multimedia channels to spread violence. Horror videos, this kind of behavior has affected the stable development of society to a certain extent. If terrorists can be quickly and accurately located in a large number of video frames, it will greatly reduce human resources and maintain social stability.

As a basic need for the management of a large-scale video library, accurately retrieving violent and terrorist video frames with terrorists plays a major role in the stability of the entire society. How to accurately define the presence of terrorists in a given video frame is a difficult problem because terrorists manifest in various forms. Usually, terrorists are masked, so in the present invention, terrorists are considered as people with masked features. As a special task of face detection, masked face detection is different from traditional face detection technology in that it faces more challenges. On the one hand, masked person face detection includes pose changes, lighting and other influencing conditions that traditional face detection technology cannot handle. On the other hand, the masked person's face is severely occluded, which greatly loses the normal structure of the original face, making the traditional algorithm invalid for masked person's face detection.

At present, a large number of face detection technologies rely on manually set features, such as: widely used Fisherface [1], cascade classifier based on Haar-like features [2], AdaBoost detector based on Gabor-like high-dimensional features [3]. Since this manually set feature requires a large number of training samples and the masked person loses the complete face structure, the manually designed features cannot accurately represent the masked person's face structure, and ultimately these methods cannot accurately detect the masked person. human face. Recently, the template-based (exemplar-based) face detection method [4] has shown better results, mainly because the huge template database covers all possible visual variations of the face, including occlusion, lighting , face pose and other changes, but this method requires a large template data set, and in the case of a highly scattered background, it is easy to produce false alarm results. In order to reduce the number of required templates, literature [5] proposed an effective template face detection method based on lifting. This method can further improve the face detection rate, accelerate the detection process, and greatly save memory overhead by combining discriminative training and effective templates as weak classifiers.

In recent years, due to the rise of deep learning, convolutional neural networks (CNN) with powerful GPU computing capabilities have also made great breakthroughs in the face field, such as LFW[6][7][ 8]. In particular, convolutional networks are able to automatically learn effective feature representations from training samples. In the Large Scale Visual Recognition Challenge in 2012, literature [9] made a breakthrough using deep convolutional neural networks. In addition, in order to further deal with the situation of only a small number of training samples, literature [10] introduces pre-training to initialize the weight of the deep network to speed up the convergence of the network and obtain a better local solution. Literature [11] proposed the LeNet model, which showed good performance in handwritten character recognition. With the development of these deep learning techniques, face detection methods based on deep learning have become possible.

references:

[1] H.J.P. Belhumeur P N, K.D.J. Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 711-720.

[2]P.Viola,M.Jones,Rapid object detection using a boosted cascade of simple features.in Proceedings of CVPR,2001.

[3]C.Liu,H.Wechsler,Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition.IEEE Transactions on Image Processing,2002,11(4):467-476.

[4] X.Shen, Z.Lin, J.Brandt, et al. Detecting and aligning faces by image retrieval. in Proceedings of CVPR, 2013: 3460-3467.

[5] H.Li, Z.Lin, J.Brandt, et al. Efficient boosted exemplar-based face detection. In Proceedings of CVPR, 2014: 1843-1850.

[6]X.W.Yi Sun,X.Tang.Deep learning face representation from predicting10,000classes.in Proceedings of CVPR,2014:1891-1898.

[7]Y.Sun, X.Wang, X.Tang.Deeply learned face representations aresparse,selective,and robust.arXiv preprint arXiv:1412.1265.

[8] Y. Sun, X. Wang, X. Tang. Hybrid deep learning for face verification. in Proceedings of ICCV, 2013: 1489-1496.

[9] A. Krizhevsky, I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutional neural networks. in Proceedings of NIPS, 2012: 1097-1105.

[10]G.E.Hinton,R.R.Salakhutdinov.Reducing the dimensionality of data with neural networks.Science,2006,313:504-507.

[11] Y.LeCun, L.Bottou, Y.Bengio, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324.

Contents of the invention

The purpose of the present invention is to provide MLeNet with methods such as pre-training and fine-tuning, combined with the sliding window method, which can quickly A robust masked face detection method based on an improved LeNet that accurately locates the masked person's face position.

The present invention comprises the following steps:

1) Expand the training data by flipping the original training picture horizontally;

2) By modifying the structure of the traditional LeNet model, a new MLeNet model is proposed to make it suitable for the detection of masked humans. The specific method can be: adjust the size of the convolution kernel and the number of feature maps, and change the original output The number of nodes in the layer 10 is 2, making it suitable for the 2-classification problem of human detection;

3) Borrow the parameters of the original LeNet model to pre-train the MLeNet structure, and fine-tune the MLeNet model to obtain a detector suitable for masked faces;

4) Combining sliding window and non-maximization suppression technology to accurately locate the position of the masked person's face.

The present invention has the following outstanding advantages:

Based on the original LeNet model, the present invention proposes a new MLeNet model by modifying the size of the convolutional filter, the number of feature maps and the number of nodes in the fully connected layer of the convolutional layer. . At the same time, the performance of MLeNet is further improved by expanding the training samples and combining pre-training and fine-tuning. Finally, the location of the masked person's face is accurately located by combining sliding windows and non-maximum suppression. In the present invention, the requirements for equipment are relatively low, only an 8G USB disk is needed to store the data set for training the MLeNet model, and a high-performance CPU is also needed to calculate various convolution calculations in the MLeNet model.

Technical effect of the present invention is as follows:

Through the modified LeNet model, a new MLeNet model is proposed, using technologies such as pre-training, fine-tuning, and data expansion, and introducing some post-processing techniques, the model proposed by the invention can accurately detect the masked human face, and in the background The model still has strong robustness under disturbance conditions such as scattered and environmental changes.

The MLeNet model can effectively solve the model overfitting problem caused by the small sample problem, and can accurately locate the face position of the masked person in the natural environment. It has a large number of application prospects in the fields of video surveillance and public security. The invention establishes the MLeNet model, which modifies the original LeNet model, so that the model is more suitable for masked person face detection. In the case of fewer training samples, training the model can easily lead to overfitting. Therefore, by expanding the training data set and combining pre-training, fine-tuning and other technologies, the over-fitting problem and the classification of the improved MLeNet model have been overcome. Accuracy. The use of post-processing methods, such as non-maximum suppression, makes the detection of masked faces more accurate.

Description of drawings

Figure 1 is a general flowchart of specific masked face detection.

Figure 2 shows the modified convolutional neural network MLeNet model: the MLeNet output layer has only two nodes, has a smaller convolution kernel size in all convolution layers, and each layer has a larger number of feature maps.

Figure 3 shows the LeNet loss function value (including the function loss value in the training and verification stages).

Figure 4 shows the LeNet classification error rate (including the classification error rate of positive and negative samples).

Figure 5 shows the MLeNet loss function value without pre-training and fine-tuning (including the function loss value in the training and verification stages).

Figure 6 shows the classification error rate of MLeNet without pre-training and fine-tuning (including the classification error rate of positive and negative samples).

Figure 7 shows the MLeNet loss function value with pre-training and fine-tuning (including the function loss value in the training and verification stages).

Figure 8 shows the classification error rate of MLeNet with pre-training and fine-tuning (including the classification error rate of positive and negative samples).

Figure 9 shows some results of face detection of masked terrorists (in order to protect privacy, the face area of the masked person is processed by mosaic).

detailed description

The purpose of the present invention is to provide an MLeNet model for the lack of training samples and to improve the traditional manual adjustment of face features, and to obtain an accurate and robust face model through simple expansion of samples, pre-training and fine-tuning. Combining sliding window, non-maximization suppression methods, a fast, robust and accurate face detector is obtained. The specific algorithm flow is shown in Figure 1. The specific modules are as follows:

1. Expand the data set

The training and test data sets used in the present invention are composed of some key frames in the violent terror videos provided by the Ministry of Public Security. It contains a total of 1140 pictures, of which 240 are positive samples (that is, contain masked faces), and 900 negative samples (that is, do not contain masked faces). The experiment randomly selects 150 positive samples and 750 negative samples as In the training set (trainingset), 50 positive samples and 50 negative samples are used as the validation set (validation set), leaving 140 pictures as the test set (test set). Considering the special symmetry information of the human face, the present invention uses a horizontal reflection technology to expand the original data set twice.

2. MLeNet model

The MLeNet model is an improvement of the original LeNet model. The LeNet model has a total of 5 layers, including 3 convolutional layers and 2 fully connected layers. The convolutional layer contains convolution and downsampling operations. First consider whether there is a masked face, which is a two-category problem. By modifying the number of nodes in the last fully connected layer, from the original 10 to 2, and convolving the original LeNet The kernel size is reduced to 3×3, but the number of feature maps per layer is increased. In particular, the number of nodes in the first fully connected layer (FC4) is changed from 84 to 500. The information of each layer of the MLeNet and LeNet models are listed in Table 1 in detail. In addition, the final MLeNet model is shown in Figure 2.

See Table 1 for MLeNet and LeNet models: Each model contains 3 convolutional layers and 2 fully connected layers. The detailed parameters of each layer of each model are listed in the last two lines, where the convolution kernel size is "num×size×size" , the convolution kernel movement interval "st.", the space filling "pad", and the maximum pooling factor.

Table 1

Let the N training samples be Wherein the label y _i is a label variable (the value is 0 or 1 in the present invention). The final loss function is the Softmax loss function (ie, the error between the predicted value and the label), defined as:

in, is the probability value output by the model, l{y _i =j} is an indicative function, which can be defined as

If the model output value The closer to the real label value, the smaller the error output. w and b are the weight and bias of each layer respectively. predicted label It can be obtained by a series of w,b forward propagation. In addition, each parameter of the network can be combined with back-propagating errors of each layer and stochastic gradient descent to update all parameters.

Specifically, the present invention uses the gradient descent method to train the MLeNet model (that is, update the variables w,b of each layer), set the batch size to 20, set the momentum to 0.9, and weight decay (weight decay) It is set to 0.0005, the learning rate is set to 0.001, and the number of training rounds (epoch) is 100. The weight w and bias b update rules are as follows:

Among them, i is the iteration index value, u, v are momentum variables, Expressed as the partial derivative of the objective function corresponding to the i-th batch image D _i to the weight w, Expressed as the partial derivative of the objective function corresponding to the i-th batch image D _i to the weight b. The update rule shows that the update method of each layer of variables (weight w and bias b) is to make the target loss function move along the direction of the local minimum, and finally obtain a local optimal solution. The weights and bias values initialized in the present invention are directly from the trained LeNet model parameters, and the random gradient descent method is used to fine-tune the MLeNet. On an ordinary PC with 6GB of memory and 1.90GHz AMD A8-4500MAPU, the MLeNet model can be trained for 100 rounds without using a GPU, and the training time only takes 10 minutes.

3. Skills to improve detection accuracy: pre-training, fine-tuning

The present invention learns the MLeNet model through pre-training and fine-tuning means. First, the LeNet model is pre-trained using the MNIST dataset, and then the MLeNet parameters are initialized by the learned LeNet parameters. Finally, the parameters of MLeNet are fine-tuned using stochastic gradient descent.

4. Detection of masked faces

Using the training MLeNet method described above, a masked face detector with high accuracy can be obtained to determine whether there is a masked face in a given window. However, the problems of multi-scale and overlapping detection windows are not considered, so the present invention utilizes an image pyramid matching scheme combined with non-maximum value suppression to post-process such problems.

In short, in order to perform pyramid matching, it is necessary to collect target images at different positions in the multi-scale image, and each sampled image is put into the trained MLeNet masked face detector, and the MLeNet detector can give each The window produces a score value for the presence or absence of a face. Then, some high-scoring sub-windows are fused using non-maximum suppression, and finally, the detection is done.

Masked face detection technology based on a new MLeNet model. MLeNet introduces pre-training and fine-tuning methods, combined with the sliding window method, to quickly and accurately locate the masked person's face position.

The specific experimental results are as follows:

With the development of society, the improvement of science and technology, and the popularization of multimedia technology, more and more people are uploading various online videos on the Internet, including many criminals trying to use multimedia channels to spread violence. Horror videos, this kind of behavior has affected the stable development of society to a certain extent. If terrorists can be quickly and accurately located in a large number of video frames, it will greatly reduce human resources and maintain social stability. How to accurately define the presence of terrorists in a given video frame is a difficult problem because terrorists manifest in various forms. Usually, terrorists are masked, so in the present invention, terrorists are considered as people with masked features. Therefore, whether the position of the masked person's face can be accurately located is the key to judging whether there are terrorists in the video frame. Given a small number of training samples and the masked person cannot obtain the complete face structure, the traditional face detection technology cannot accurately locate the masked person's face position.

Face detection is an important application in the direction of computer vision. Traditional face detection algorithms can more accurately detect frontal, unoccluded faces, but for occluded, especially low-resolution, masked faces, it is Less than a good detection effect. In the present invention, a new model is proposed for masked person face detection, which can obtain good performance. The present invention can be used in fields such as video surveillance, human-computer interaction, video retrieval of violent terrorism, and public security.

Figures 3 and 4 present the performance of the LeNet model on a given dataset of masked faces. Figures 5 and 6 show the performance of MLeNet without pre-training and fine-tuning, and Figures 7 and 8 show the training results of MLeNet with pre-training and fine-tuning. From the graph of the experiment, it can be seen that the MLeNet model trained by means of pre-training and fine-tuning has greatly improved the masked face classification results.

The experimental results of detecting masked faces in the self-created masked person dataset are shown in Table 2. As can be seen from Table 2, compared to the traditional AdaBoost algorithm, LeNet model, and the MLeNet model without pre-training and fine-tuning, the method of the present invention is more suitable for on the masked face detection problem.

Table 2

Ours AdaBoost[2] LeNet[11] MLeNet recall 0.925 0.75 0.82 0.85 Precision 0.71 0.6 0.64 0.68 F ₁ -score 0.803 0.667 0.719 0.756

"Ours" means MLeNet with pre-training and fine-tuning; "MLeNet" means MLeNet model without pre-training and fine-tuning.

The formula description is as follows: (The defined formula variables and symbols can refer to the specific formula expression description)

Formula (1) defines the loss function of the model, which is used to measure the error between the output result of the model and the original label value.

Formula (2) is the definition of an indicative function, the purpose is to judge whether two values are equal, if they are equal, the value is set to 1, otherwise, it is set to 0.

Formula (3) defines the update rule of the stochastic gradient descent method. Its purpose is to update the variables (weight w and bias b) of each layer so that the target loss function moves along the direction of the local minimum to obtain the final local optimal solution.

Claims (2)

1. A robust mask face detection method based on an improved LeNet is characterized by comprising the following steps:

1) expanding training data by horizontally turning over the original training picture;

2) by modifying the structure of the traditional LeNet model, a new MLeNet model is provided, so that the MLeNet model is suitable for the detection problem of a masked human;

3) pre-training an MLeNet structure by using parameters in an original LeNet model, and finely adjusting the MLeNet model to obtain a detector suitable for a mask face;

4) and the position of the face of the masked person is accurately positioned by combining a sliding window and a non-maximization inhibition technology.

2. The robust masked face detection method based on the improved LeNet as claimed in claim 1, wherein in step 2), the specific method for proposing the new MLeNet model by modifying the structure of the traditional LeNet model to adapt to the masked human detection problem is: the size of the convolution kernel and the number of the characteristic graphs are adjusted, and in addition, the number 10 of the nodes of the original output layer is changed to be 2, so that the method is suitable for the 2-classification problem of human detection.