CN110569809A - A coal mine dynamic face recognition attendance method and system based on deep learning - Google Patents

Abstract

The present invention discloses a dynamic face recognition attendance method and system in a coal mine based on deep learning. The method is as follows: Step 1: Obtain the monitoring video at the entrance of the coal mine, and extract each frame of the monitoring video; Step 2: Comparing the pictures Perform blur detection. When the detected blur is lower than the set threshold, the picture will not be processed. If it is higher than the set threshold, go to Step 3; Step 3: Use the face detection algorithm based on deep learning to process the preprocessed image. Detect the face area in the picture, if there is a face, output the coordinates of the face in the picture; Step 4: Locate the key points of the face in the face area; Step 5: Use a method based on deep learning for face recognition , using the similarity comparison to calculate the image with the closest distance in the face database, that is, the face recognition is completed. By adopting the method, the attendance checking efficiency of the personnel in the coal mine is improved, and the behaviors of passing cards, missing cards, personnel identity authentication, and checking attendance in the coal mine checking attendance are avoided.

Description

A coal mine dynamic face recognition attendance method and system based on deep learning

technical field

The invention belongs to the technical field of face recognition, and in particular relates to a dynamic face recognition attendance method and system for coal mines based on deep learning.

Background technique

At present, there are mainly three types of attendance management methods commonly used in coal mines: 1. Traditional attendance punching and card swiping; 2. Fingerprint recognition; 3. Iris recognition. However, in the traditional attendance check-in, a card reader is installed at the entrance of the well, and the attendance is completed by detecting the information of the identification card. Fingerprint recognition checks attendance at the entrance of the mine by verifying the fingerprint information of the personnel who go down the mine. However, the fingerprints of front-line mine workers in coal mines are easily damaged, and the fingerprints are covered with dust and coal ash, which leads to a high recognition rate. The effect in the coal mining industry is not ideal. Iris recognition uses the characteristics of lifelong invariance and difference of iris to identify identities. It has stable performance, high recognition accuracy, and cannot be forged. It is non-contact and suitable for special industries such as coal mines. At present, more and more mines are beginning to use iris recognition. As an identification method for mine safety attendance, the technology has been used in coal mines in the three northeastern provinces, Shaanxi, Shanxi, Shandong and other provinces. However, there are still some deficiencies in iris time attendance: it is necessary for the personnel who go into the well to make cooperative behaviors, and the attendance efficiency is poor, which will cause crowded queues at the entrance of the well, as shown in Figure 1; some personnel did not go down the well after being authenticated on the iris time attendance machine; some Unreasonable situations such as not checking attendance when personnel go down the well, and checking attendance after returning to the wellhead after exiting the wellhead.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method and system based on deep learning for dynamic face recognition attendance in coal mines, which improves the efficiency of attendance checks for personnel in coal mines, and avoids the problems of checking in coal mines. Missing cards, personnel identity authentication, attendance falsification, etc.

In order to solve the above-mentioned technical problems, the technical solution adopted by the present invention is a method for dynamic face recognition attendance checking in coal mines based on deep learning, which method includes as follows:

Step 1: Obtain the surveillance video at the entrance of the coal mine, convert the surveillance video into a frame-by-frame picture, and extract each frame of picture;

Step 2: Preprocess the picture in step 1: perform blur detection on the picture. When the detected blur is lower than the set threshold, the picture will not be processed. If it is higher than the set threshold, perform step 3; threshold set to 0.7;

Step 3: Detect the face area in the picture: use the face detection algorithm based on deep learning to detect the face area in the preprocessed picture, if there is a face, output the coordinates of the face in the picture;

Step 4: Locate the key points of the face in the face area, use the method based on deep learning to align the face area, and obtain the aligned face picture;

Step 5: Use the method based on deep learning to carry out face recognition, and the aligned face image (picture) will output the feature vector of each face through the neural network, and use the similarity comparison to calculate the closest image with the face database. That is, face recognition is completed. When there is the closest image, it indicates arrival, and if there is no nearest image, it indicates absence.

Further, the specific process of step 3 is as follows: input the preprocessed picture into the feature extraction network to generate a feature map, on the output feature map, use the candidate region generation network to generate the image in the output feature map Candidate face area; then use the face detection network to classify the face area, and output the final face target position and probability; the face area classification is divided into face target and background; the feature extraction network uses 50 layers of residual Network Res50.

The candidate area generation network uses the anchor point window to determine the candidate face area in the picture. The anchor point window in the candidate area generation network adopts 8 scales, 3 kinds of ratios, and the 8 scales are [1, 2, 3, 4 , 6, 8, 16, 32, 64], the three ratios are [2:1, 1:1, 1:2]; the face detection network includes a 1*1 convolutional layer, a 2k A ² -dimensional convolutional layer and a pooling layer; where: k≥3, and the maximum value of k needs to be determined by the detection speed.

Further, the process of classifying faces by the face detection network is as follows: the face detection network first uses a 1*1 convolutional layer to reduce the dimension on the generated feature map, and then generates Position-sensitive score map; the position-sensitive score map and the candidate face area generated by the candidate region generation network are sent to the position-sensitive pooling layer to generate a category score map and a range box prediction map, and the category score map and the range box prediction map An average pooling operation is used to aggregate the feature map into a vector, and finally output the probability and coordinate position of each candidate area containing a face.

Further, the specific process of step 4 is as follows: send the preprocessed picture and the coordinates of the human face area position into the first convolutional neural network, put the average human face shape in the human face area coordinate range frame, and This is to initialize the coordinates of the key points of the face, and output the offset of the coordinates of the key points of the face through the first neural network, that is, to obtain the key points of the face, and normalize the geometry of the face area to the standard pose through similar transformation, that is, to achieve Faces are aligned to obtain aligned face images.

Further, the specific process of step 5 is: using the framework of convolutional neural network plus loss function, using the loss function to supervise the training of the entire convolutional neural network; the convolutional neural network uses a 20-layer residual network;

The formula of the loss function is as follows:

Among them: L _A represents the A-softmax loss function, _LC represents the center loss function, λ is used to balance the weight of the center loss function and the A-softmax loss function, N represents the number of training set samples, and _xi represents the i-th training The characteristics of the set sample, m is the hyperparameter used to quantify the decision boundary, j represents the category, represents the feature center of the y _i -th category, is the weight vector and x _i , θ _j,i represents the angle between W _j and eigenvector x _i , and W _j represent the weights of the fully connected layer of the network corresponding to categories y _i and j, respectively.

The invention also discloses a coal mine dynamic face recognition attendance system based on deep learning, including an image acquisition module, a database, a video preprocessing module, a deep learning server, and a detection and identification management module; the image acquisition module: arranged at the coal mine entrance, It is used to collect face information of coal mine workers, and generate corresponding face feature information and store it in the database;

Database: used to receive and store facial feature information and identity information;

Video preprocessing module: Obtain real-time surveillance video from the surveillance camera, convert the video into a frame-by-frame picture, and perform blur detection on each frame of picture. When the blur degree is higher than the set threshold, the picture will not be processed. If it is lower than the set threshold, it will be sent to the deep learning server;

Deep learning server: including face detection module, face alignment module and face recognition module connected in sequence;

The preprocessed picture is first sent to the face detection module, and the face area in the preprocessed picture is detected by using the face detection algorithm based on deep learning. If there is a face, the coordinates of the face in the picture are output ; Then the whole picture (preprocessed picture) and the detected face area coordinates are sent to the face alignment module to detect the key point feature information of the face, and the face is aligned; finally the alignment The final face picture is sent to the face recognition module to extract the features of the face, compared with the information in the face feature database, to identify the identity information of each face to be detected;

Detection and recognition management module: used to manage the face identity information detected and recognized, and realize the query and recording of personnel attendance information.

The present invention has the following advantages: Compared with the traditional attendance checking method, the present invention has a faster recognition speed, and the whole recognition process does not require cooperative actions, and the whole process is senseless. Use the deep learning framework TensorFlow to complete the implementation of the algorithm and the training of the model. Using this method to realize face recognition can quickly deploy the model to the production environment without writing additional code, and the trained model can be quickly deployed to the depth In the learning server, sending a picture to the deep learning server can return the coordinates of the detected face, the key coordinates of the face, the features of the face and other information.

Description of drawings

Fig. 1 is a structure diagram of a coal mine dynamic face recognition attendance method based on deep learning according to the present invention.

Detailed ways

A kind of coal mine dynamic face recognition attendance checking method based on deep learning of the present invention, this method comprises as follows, as shown in Figure 1:

Step 1: Obtain the surveillance video at the mine entrance, convert the surveillance video into frame-by-frame pictures, and extract each frame of the surveillance video; specifically, install 4 to 6 high-definition cameras at different angles at the mine entrance to capture The photos of the faces of the personnel who go down the well when they pass through the mouth of the well can be 100% guaranteed through the real-time monitoring of multiple cameras.

Step 2: Preprocess the picture: perform blur detection on the picture. When the detected blur is higher than the set threshold, the picture will not be processed. If it is higher than the set threshold, perform step 3; the threshold is set to 0.7 ;

Step 3: Detect the face area in the picture: use the face detection algorithm based on deep learning to detect the face area in the preprocessed picture, if there is a face, output the coordinates of the face in the picture;

The specific process is as follows: take the preprocessed whole picture as input, input it into the feature extraction network, generate a feature map, on the output feature map, use the candidate area generation network to generate the candidate in the picture on the output feature map Face area; then use the face detection network to classify the face area, and output the final face target position and probability; the face area classification is divided into face target and background; the feature extraction network uses a 50-layer residual network Res50 .

The candidate region generation network uses an anchor point window to determine the candidate face region in the picture, and the anchor point window in the candidate region generation network adopts 8 scales and 3 ratios, and the 8 scales are [1, 2, 3 , 4, 6, 8, 16, 32, 64], the three ratios are [2:1, 1:1, 1:2]; the face detection network includes a 1*1 convolutional layer connected in sequence, A 2k ² -dimensional convolutional layer and a pooling layer; where: k≥3, and the maximum value of k is determined by the detection speed. If k is larger, the spatial division will be finer and the positioning will be more accurate. However, due to the increase of k, the spatial position grid to be processed will be increased, which will increase the memory usage and reduce the processing speed. Usually k is selected as 3.

The process of classifying faces by the face detection network is as follows: the face detection network first uses a 1*1 convolutional layer to reduce the dimension on the generated feature map, and then generates a position through a specific convolutional layer. Sensitive score map; the position-sensitive score map and the candidate face area generated by the candidate region generation network are sent to the position-sensitive pooling layer to generate a category score map and a range box prediction map, and on the category score map and the range box prediction map Use an average pooling operation to aggregate the feature map into a vector, and finally output the probability and coordinate position of each candidate area containing a face.

The main purpose of the position-sensitive score map here is to improve the performance of face detection, because the feature map output by the convolutional neural network has translation invariance, and the face detection task needs to locate the position of the face, which requires the network to have Good translation variability. In the present invention, the face detection network integrates the position information of the face into the feature by introducing a position-sensitive score map, so as to improve the detection performance. The specific method is as follows. First, a 1*1 convolutional layer is used for dimensionality reduction on the feature map generated by the feature extraction network. The feature map after dimensionality reduction undergoes a 2k ² -dimensional convolution operation (conv) and the output dimension is 2k ² position-sensitive score maps, that is, k×k position-sensitive score maps are generated for each face. Among them: 2 means category: face + background, so there are 2 categories in total; k×k means k×k relative spatial positions; therefore, k ² position-sensitive score maps are generated for each category, so a total of 2k ² position-sensitive map; k ² position-sensitive score maps encode k ² positions used to describe spatial position information. Taking k=3 as an example, 9 position-sensitive score maps encode 9 kinds of spatial position information of a face {top left corner, top center, top right..., bottom right}.

Step 4: Locate the key points of the face in the face area, perform alignment processing on the face area, and obtain the aligned face picture; the specific implementation of the face alignment method is as follows: the state of the posture, direction, and size of the face In the image, it is different, and the face poses are different, which will cause difficulties in recognition. Aligning the faces in the image to a unified template can improve the accuracy of face recognition. Therefore, in the present invention, in the process of face recognition Faces are aligned.

The face alignment method used is a face alignment method based on deep learning. The original picture and the coordinates of the face area detected by the face detection algorithm are sent to the convolutional neural network, and the face returned by the face detection algorithm is Put the average face shape in the frame of the coordinate range of the face area to initialize the coordinates of the key points of the face, output the offset of the coordinates of the key points of the face through the neural network, that is, obtain the key points of the face, and use the detected key points of the face to pass The similarity transformation normalizes the geometry of the face area to the standard pose, which realizes face alignment. In the present invention, using the entire face image as input can align face images with a large range of facial poses, thereby further improving the performance of face recognition.

Step 5: Use the method based on deep learning for face recognition, and the aligned face images will output the feature vector of each face through the neural network, and use the similarity comparison to calculate the image with the closest distance to the face database, that is, complete the face recognition process. face recognition. When there is the closest image, it indicates arrival, and if there is no nearest image, it indicates absence.

The specific process of this step 5 is: using the framework of convolutional neural network plus loss function, using the loss function to supervise the training of the entire convolutional neural network; the convolutional neural network uses a 20-layer residual network; using convolutional neural network The framework of network + loss function uses the loss function to supervise the training of the entire convolutional neural network, so that the network can learn facial features with strong discrimination. For the consideration of recognition speed, the convolutional neural network in the present invention uses a 20-layer residual network, and the loss function proposes an improved loss function in the present invention, which is jointly supervised by the joint center loss function and A-softmax loss function. The convolutional neural network is trained to obtain face features with a more compact intra-class distance, which ultimately improves the performance of face recognition. The central loss function can only compress the features within the class, and it needs to supervise the training convolutional neural network together with the softmax function to obtain highly discriminative face features; the A-softmax loss function defines a learning method with a large angle interval, which can After learning the face features with a large inter-class distance and a compact intra-class distance, the intra-class distance of the face can be further compressed through the combination of two loss functions.

The formula of the loss function is as follows:

Among them: L _A represents the A-softmax loss function, _LC represents the center loss function, λ is used to balance the weight of the center loss function and the A-softmax loss function, N represents the number of training set samples, and _xi represents the i-th training The characteristics of the set sample, m is the hyperparameter used to quantify the decision boundary, j represents the category, represents the feature center of the y _i -th category, is the weight vector and x _i , θ _j,i represents the angle between W _j and eigenvector x _i , and W _j represent the weights of the fully connected layer of the network corresponding to categories y _i and j, respectively.

The invention also discloses a coal mine dynamic face recognition attendance system based on deep learning, including an image acquisition module, a database, a video preprocessing module, a deep learning server, and a detection and identification management module; the image acquisition module: arranged at the coal mine entrance, It is used to collect face information of coal mine workers, and generate corresponding face feature information and store it in the database;

Database: used to receive and store facial feature information and identity information;

Video preprocessing module: Obtain real-time surveillance video from the surveillance camera, convert the video into a frame-by-frame picture, and perform blur detection on each frame of picture. When the blur degree is higher than the set threshold, the picture will not be processed. If it is lower than the set threshold, it will be sent to the deep learning server;

Deep learning server: including face detection module, face alignment module and face recognition module connected in sequence;

The preprocessed picture is first sent to the face detection module, and the face area in the preprocessed picture is detected by using the face detection algorithm based on deep learning. If there is a face, the coordinates of the face in the picture are output ; Then the whole picture (preprocessed picture) and the detected face area coordinates are sent to the face alignment module to detect the key point feature information of the face, and the face is aligned; finally the alignment The final face picture is sent to the face recognition module to extract the features of the face, compared with the information in the face feature database, to identify the identity information of each face to be detected;

Detection and recognition management module: used to manage the face identity information detected and recognized, and realize the query and recording of personnel attendance information.

In order to be able to realize dynamic face recognition attendance, that is, coal mine personnel who go down the mine do not need to make cooperative behaviors, and realize face recognition and sign-in when they walk through the coal mine entrance. Install 4 to 6 high-definition cameras at different angles at the entrance of the well, the purpose is to capture photos of the faces of the personnel who go down the well when they pass the entrance of the well. Through the real-time monitoring of multiple cameras, the capture of the face can be 100% guaranteed, regardless of whether the personnel going down the well are walking Any twisting during the process can guarantee the capture of the face.

Claims (6)

1. A coal mine dynamic face recognition attendance method based on deep learning is characterized by comprising the following steps:

Step 1: acquiring a monitoring video at a coal mine well entrance, converting the monitoring video into a frame of picture, and extracting each frame of picture;

Step 2: preprocessing the picture: carrying out ambiguity detection on the picture, when the detected ambiguity is lower than a set threshold value, the picture is not processed, and when the detected ambiguity is higher than the set threshold value, executing the step 3; the threshold was set to 0.7;

And step 3: detecting a face area in a picture: detecting a face region in the preprocessed picture, and outputting coordinates of the face in the picture if the face exists;

And 4, step 4: positioning face key points in the face area, and aligning the face area to obtain an aligned face picture;

And 5: adopting a deep learning-based method to carry out face recognition, outputting the feature vector of each face by the aligned face image through a neural network, and calculating the image closest to the face database by utilizing similarity comparison, namely finishing the face recognition; when the image with the nearest distance exists, the duty is indicated, and if the image with the nearest distance does not exist, the absence of duty is indicated.

2. The coal mine dynamic face recognition attendance method based on deep learning of claim 1, wherein the specific process of step 3 is as follows:

The method comprises the following specific steps: inputting the preprocessed picture into a feature extraction network to generate a feature mapping graph, and generating a candidate face region in the picture by utilizing a candidate region on the output feature mapping graph; then, classifying the face region by using a face detection network, and outputting the final face target position and probability; classifying the face area into a face target and a background; the feature extraction network adopts a residual error network Res50 with 50 layers;

The candidate area generation network determines the candidate face area in the picture by adopting an anchor point window, the anchor point window in the candidate area generation network adopts 8 scales and 3 proportions, the 8 scales are [1, 2, 3, 4, 6, 8, 16, 32, 64], and the 3 proportions are [ 2: 1,1: 1,1: 2 ];

The face detection network comprises a 1 x 1 convolution layer and a 2k convolution layer which are connected in sequence²a convolutional layer of dimensions and a pooling layer; wherein: k is equal to or more than 3, and the maximum value of k is determined by the required detection speed.

3. The coal mine dynamic face recognition attendance method based on deep learning of claim 2, wherein the face detection network classifies the faces as follows: the face detection network firstly uses a 1 × 1 convolution layer to reduce dimension on the generated feature mapping graph, and then generates a position sensitive score graph through a specific convolution layer; and the position sensitive score map and the candidate face region generated by the candidate region generation network are jointly sent to a position sensitive pooling layer to generate a category score map and a range frame prediction map, an average pooling operation is respectively utilized on the category score map and the range frame prediction map to aggregate the feature mapping maps into a vector, and finally the probability and the coordinate position of each candidate region including the face are output.

4. the coal mine dynamic face recognition attendance method based on deep learning of claim 1, 2 or 3, characterized in that the specific process of step 4 is as follows: the coordinates of the preprocessed pictures and the positions of the face regions are sent into a first convolution neural network, an average face shape is put into a face region coordinate range frame, face key point coordinates are initialized, the offset of the face key point coordinates is output through the first neural network, face key points are obtained, the face regions are geometrically normalized to a standard posture through similarity transformation, face alignment is achieved, and aligned face images are obtained.

5. The coal mine dynamic face recognition attendance method based on deep learning of claim 4, wherein the specific process of the step 5 is as follows: adopting a frame of a convolutional neural network and a loss function, and training the whole convolutional neural network by using the loss function in a supervision way; the convolutional neural network adopts a 20-layer residual error network;

The formula of the loss function is as follows:

Wherein: l is_ADenotes the A-softmax loss function, L_Crepresenting the central loss function, λ is used to balance the weight of the central loss function with the A-softmax loss function, N represents the number of training set samples, x_iRepresenting the characteristics of the ith training set sample, m is a hyper-parameter used to quantify the decision boundary, j represents the class,represents the y th_ithe center of the features of each of the categories,Is a weight vectorAnd x_iAngle between them, theta_j,irepresents W_jand a feature vector x_iThe included angle between the two parts is included,And W_jrespectively representing that the network full connection layer corresponds to the category y_iAnd the weight of j.

6. A coal mine dynamic face recognition attendance system based on deep learning is characterized by comprising an image acquisition module, a database, a video preprocessing module, a deep learning server and a detection recognition management module;

The image acquisition module: the face characteristic information acquisition system is arranged at a coal mine well entrance, is used for acquiring face information of coal mine well descending operators, generates corresponding face characteristic information and stores the face characteristic information into a database;

A database: the face recognition system is used for receiving and storing face feature information and identity information;

The video preprocessing module: acquiring a real-time monitoring video from a monitoring camera, converting the video into a frame of picture, detecting the fuzziness of each frame of picture, and sending the picture to a deep learning server when the fuzziness is higher than a set threshold value and the picture is not processed and is lower than the set threshold value;

The deep learning server: the system comprises a face detection module, a face alignment module and a face recognition module which are sequentially connected;

The method comprises the steps that a preprocessed picture is sent to a face detection module, a face detection algorithm based on deep learning is used for detecting a face area in the preprocessed picture, and if a face exists, coordinates of the face in the picture are output; then, the whole picture (preprocessed picture) and the detected face region coordinates are sent to a face alignment module together, the face key point feature information is detected, and the face is aligned; finally, the aligned face picture is sent to a face recognition module to extract the features of the face, and the features of the face are compared with the information in the face feature database to recognize the identity information of each face to be detected;

The detection, identification and management module: the system is used for managing the identity information of the detected human face and realizing the inquiry and recording of the attendance information of the personnel.