CN113723243B - Face recognition method of thermal infrared image of wearing mask and application - Google Patents
Face recognition method of thermal infrared image of wearing mask and application Download PDFInfo
- Publication number
- CN113723243B CN113723243B CN202110960380.3A CN202110960380A CN113723243B CN 113723243 B CN113723243 B CN 113723243B CN 202110960380 A CN202110960380 A CN 202110960380A CN 113723243 B CN113723243 B CN 113723243B
- Authority
- CN
- China
- Prior art keywords
- face
- neural network
- convolutional neural
- image
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 103
- 238000001514 detection method Methods 0.000 claims abstract description 63
- 238000012549 training Methods 0.000 claims description 104
- 230000006870 function Effects 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 11
- 238000013528 artificial neural network Methods 0.000 claims description 9
- 230000001815 facial effect Effects 0.000 claims 3
- 238000012360 testing method Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000005855 radiation Effects 0.000 description 3
- 239000011269 tar Substances 0.000 description 3
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000029058 respiratory gaseous exchange Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Chemical compound O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a thermal infrared image face recognition method for a mask and application thereof. The method comprises the following steps: acquiring a thermal infrared face image to be identified, and inputting the thermal infrared face image into a trained first convolutional neural network, wherein the first convolutional neural network is used for detecting whether the thermal infrared face image is a mask wearing image and a face detection frame; inputting the output data of the first convolutional neural network into a trained second convolutional neural network, wherein the second convolutional neural network is used for outputting face detection images without masks; and inputting the output data of the second convolutional neural network into a trained third convolutional neural network, wherein the third convolutional neural network is used for outputting a face recognition result. The invention can improve the accuracy of face recognition of the face mask.
Description
Technical Field
The invention belongs to the crossing field of biological feature recognition infrared technology and anti-terrorism technology, and particularly relates to a thermal infrared image face recognition method for a mask and application thereof.
Background
Face recognition by wearing a face mask, namely, for a face thermal infrared image of a person wearing a face mask, the label or name of the corresponding person can be known. The face recognition of the face mask is widely applied in various fields, and particularly has very high application value in the national security field of anti-terrorism.
The face recognition technology of visible light has been widely used in various fields. However, the face recognition technology of visible light cannot work under the condition of no external light source, and cannot recognize the face with the shielding object. When the face is masked with a mask or the like, the reflection-based visible light camera cannot ascertain the masked information. After wearing the mask, certain parts of the face are hidden, which makes visible light face recognition a very difficult task.
The thermal infrared image is a thermal radiation image, which is imaged according to the spatial difference of infrared radiation of the object. When the face is shielded by the mask, the heat of the face can penetrate the mask through the heat conduction and heat radiation modes to be acquired by the thermal infrared camera, and the technology can make up for the defect that the visible light camera completely loses the characteristics of the shielded part.
There is no mature solution for face recognition of face masks by using thermal infrared technology in the prior art.
Disclosure of Invention
Aiming at least one defect or improvement requirement of the prior art, the invention provides a thermal infrared image face recognition method and application of a face mask, which can improve the accuracy of face recognition of the face mask.
To achieve the above object, according to a first aspect of the present invention, there is provided a thermal infrared image face recognition method of a mask, comprising the steps of:
Acquiring a thermal infrared face image to be identified, and inputting the thermal infrared face image into a trained first convolutional neural network, wherein the first convolutional neural network is used for detecting whether the thermal infrared face image is a mask wearing image and a face detection frame;
inputting the output data of the first convolutional neural network into a trained second convolutional neural network, wherein the second convolutional neural network is used for outputting face detection images without masks;
And inputting the output data of the second convolutional neural network into a trained third convolutional neural network, wherein the third convolutional neural network is used for outputting a face recognition result.
Preferably, the training of the first convolutional neural network includes the steps of:
constructing a first training set and training labels thereof, wherein the first training set comprises a plurality of thermal infrared face images of different people wearing the face masks and a plurality of thermal infrared face images of different people not wearing the face masks, and the training labels of the thermal infrared face images comprise a face calibration frame and class labels for indicating whether the face masks are worn or not;
And setting a plurality of anchor blocks for the first convolutional neural network, and inputting the first training set and the training labels thereof into the first convolutional neural network for training.
Preferably, the training of the second convolutional neural network includes the steps of:
Constructing a second training set and training labels thereof, wherein the second training set comprises two training subsets, one training subset comprises a plurality of face detection images with face masks, and the other training subset comprises a plurality of face detection images without face masks, and the face detection images are marked with class labels which are expressed as the face detection images with face masks or the face detection images without face masks;
and inputting the second training set and the training label into the second convolutional neural network for training.
Preferably, the training of the third convolutional neural network includes the steps of:
Constructing a third training set and training labels thereof, wherein a plurality of face detection images without masks and a plurality of face detection images without masks are formed in the third training set, the face detection images are marked with category labels and identity labels, the category labels are used for representing whether the face detection images without masks are represented or the face detection images without masks are represented, and the category labels are used for distinguishing different people, and the identity labels of the face detection images without masks and the face detection images without masks belonging to the same person are the same;
and inputting the third training set and the training label into the third convolutional neural network for training.
Preferably, the second convolutional neural network comprises a first countermeasure network and a second countermeasure network, the first countermeasure network comprises a first generator and a first discriminator, the second countermeasure network comprises a second generator and a second discriminator, the class label of the face detection image of the mask is real_A, the class label of the face detection image of the mask is real_B, the mask image generated by the second discriminator is fake_A, and the mask image generated by the first discriminator is fake_B;
the first discriminator is used for judging whether the input image or the image generated by the second discriminator is real-A type or not, and the first generator is used for generating a fake-B type image according to the input or generated real-A type image;
the second discriminator is used for judging whether the input image or the image generated by the first discriminator is real_B class or not, and the second generator is used for generating a fake_A class image according to the input or generated real_B class.
Preferably, the loss function of the second convolutional neural network is:
Lossfull＝LossGan(G,Dx)+LossGan(F,DY)+λLosscycle
Wherein Loss full is a Loss function, the first generator is denoted as G, the first arbiter is denoted as D x,LossGan(G,Dx) represents the Loss functions of the first generator and the first arbiter, the second generator is denoted as F, the second arbiter is denoted as D y,LossGan(F,DY) represents the Loss functions of the second generator and the second arbiter, loss cycle is a counterloss, and λ is a weighting factor.
Preferably, the nine-grid loss function of the third convolutional neural network is:
Wherein i represents an ith image, N represents the total number of samples of the third training set, real_B_other represents different images of real_B class, other represents images belonging to different people, and fake_B_other represents different images of fake_B class;
loss1 is a Loss function for zooming in the difference between real_b class images and zooming out the difference between different class images;
Loss2 is a Loss function for zooming in the differences between real_b class images and fake_b class images, zooming out the differences between different class images;
loss3 is a Loss function for approximating the difference between images of the same fake B class and enlarging the difference between images of different classes.
According to a second aspect of the present invention there is provided a thermal infrared image face recognition system for a mask, comprising:
The first module is used for acquiring a thermal infrared face image to be recognized, inputting the thermal infrared face image into a trained first convolution neural network, and detecting whether the thermal infrared face image is a mask wearing image or not and a face detection frame;
The second module is used for inputting the output data of the first convolutional neural network to a trained second convolutional neural network, and the second convolutional neural network is used for outputting face detection images without masks;
And the third module is used for inputting the output data of the second convolutional neural network to a trained third convolutional neural network, and the third convolutional neural network is used for outputting a face recognition result.
According to a third aspect of the present invention there is provided an electronic device comprising a memory storing a computer program and a processor implementing the steps of any of the methods described above when the computer program is executed.
According to a fourth aspect of the present invention there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs any of the methods described above.
In general, the face recognition of the invention comprises three steps of face detection frame recognition of the face of the wearer, mask removal and face information recognition, and the accuracy of face recognition of the wearer can be improved, which is specifically shown in the following steps:
1) According to the invention, the thermal infrared images with and without the mask are input into the convolutional neural network for training to obtain the convolutional neural network meeting the requirements, so that whether the mask is worn or not can be rapidly and automatically detected, and the position of the face can be accurately framed.
2) According to the invention, the mask influence is removed by the thermal infrared technology, the characteristics of the part of the face shielded by wearing the mask can be recovered, and the method for removing the mask influence is completed by fully utilizing the characteristics of the face. Make up for the capability that the visible light can not utilize the information of the shielded part at all.
3) According to the invention, the information of the person belonging to the thermal infrared face image can be identified through the thermal infrared face identification technology after the mask is removed.
Drawings
Fig. 1 is a flowchart of a thermal infrared image face recognition method of a mask according to an embodiment of the present invention.
FIG. 2 is a schematic block diagram of anchor points of two scales in an embodiment of the present invention;
FIG. 3 is a diagram showing the detection result of the mask according to the embodiment of the invention;
FIG. 4 is a schematic diagram of a pair of mask-removed neural network training images in an embodiment of the invention;
FIG. 5 is a schematic diagram of the architecture of a second convolutional neural network for de-masking in an embodiment of the present invention;
FIG. 6 is a graph comparing the effects of the mask neural network before and after removal in accordance with an embodiment of the present invention;
FIG. 7 is a schematic diagram of a loss function of face recognition after mask removal according to an embodiment of the present invention;
Fig. 8 is a flowchart of training and testing a thermal infrared image face recognition method of a face mask according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. In addition, the technical features of the embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
The mask in the present application should not be construed as a mask in a narrow sense, but should be construed as any object that performs the function of shielding part or all of the face.
As shown in fig. 1, a thermal infrared image face recognition method for a mask according to an embodiment of the present invention includes steps S1 to S3.
S1, acquiring a thermal infrared face image to be recognized, inputting the thermal infrared face image into a trained first convolution neural network, wherein the first convolution neural network is used for detecting whether the thermal infrared face image is a mask wearing image and a face detection frame.
The thermal infrared face image to be identified may be an original image acquired by the thermal infrared camera, and may include a face and a background other than the face.
The first convolution neural network is used for judging whether the thermal infrared face image to be recognized is a mask wearing image, outputting a face detection frame containing a mask area if the thermal infrared face image to be recognized is the mask wearing image, and outputting the face detection frame if the thermal infrared face image to be recognized is the mask not wearing image.
Preferably, the training of the first convolutional neural network comprises the steps of:
(1) Constructing a first training set and training labels thereof, wherein the first training set comprises a plurality of thermal infrared face images of different people wearing the face masks and a plurality of thermal infrared face images of different people not wearing the face masks, and the training labels of the thermal infrared face images comprise a face calibration frame and class labels for indicating whether the face masks are worn or not;
(2) And setting a plurality of anchor blocks for the first convolutional neural network, and inputting the first training set and the training labels thereof into the first convolutional neural network for training.
In one embodiment, the training and testing of the first convolutional neural network specifically comprises the steps of:
(1) Forming a training set by using the thermal infrared face images of the N Zhang Dai masks and the L thermal infrared face images without the masks, acquiring M Zhang Regong external images as a test set, and respectively framing face frames for each thermal infrared image as calibration frames; the marks of the thermal infrared images of each mask are 1, and the marks of the thermal infrared images of each mask without mask are 0;
In order to guarantee a sufficient amount of thermal infrared images, it is necessary to guarantee sufficient experimental data. Specifically, a medium wave thermal infrared imager with the model number of TAURUS-110kM of IRCAM company in Germany can be adopted, and the testing environment of data is as follows: the face is 2 meters, 3 meters and 5 meters from the camera, videos with set time are recorded for each person, each video is cut according to set frame numbers, then a set number of photos are selected, 86 persons can be selected to shoot, 50 frames of video capturing forms are adopted, the influence of different scene backgrounds is included, and the accuracy of the follow-up use of the mask detection model is guaranteed through a large number of experiments. And then, the thermal infrared images intercepted by the video can be screened, so that the parameters are prevented from being learned by a computer in deep learning, the real parameters are influenced, for example, when a picture is cut, the fuzzy images are generally removed, 27 ten thousand Zhang Dere infrared images can be finally obtained, N=2.5 ten thousand Zhang Dai thermal infrared face images of the mask are obtained, L=2.5 ten thousand thermal infrared face images without the mask are obtained to form a training set together, and M=2 ten thousand Zhang Regong external images are obtained as a testing set.
The thermal infrared image of each mask worn is marked 1 and the thermal infrared image of each mask not worn is marked 0.
(2) The method comprises the steps of building a first convolutional neural network, setting the sizes of six anchor blocks according to the particularity of a face image, inputting a training set and training labels into the first convolutional neural network together for training, and thus obtaining a required training model of the convolutional neural network.
(2.1) Setting the sizes of six anchor blocks according to the specificity of the face image, wherein the sizes are specifically as follows:
In one embodiment, as shown in FIG. 2, the Fast Anchor algorithm (Fast Anchor) is implemented by designing two large-scale Anchor boxes (Anchor boxes), large-scale Anchor boxes (379,387), (251,292), (142,189), and small-scale Anchor boxes (190,193), (110, 116), (54,45).
(2.2) Parameters of the first convolutional neural network are as follows:
The number of times of the present invention (epoch) was set to 200, the batch (batch) was set to 8, the input image was set to 416×416, the momentum (movement) was set to 0.9, the decay (decay) was set to 0.0005, the learning rate (Ir) was set to 0.001, the color saturation (saturation) was set to 1.5, the exposure (exposure) was set to 1.5, the hue variation range (hue) was set to.1, and the training time was 83 hours.
(3) Inputting a thermal infrared image in the test set, and obtaining a face detection frame of the mask through a convolutional neural network; the invention can realize the processing of the single graph of 0.026s and has high precision. The test sample selects 2 ten thousand pictures, and the accuracy is 96.5%.
The mask detection result output through step S1 is shown in fig. 3.
S2, inputting output data of the first convolutional neural network into a trained second convolutional neural network, wherein the second convolutional neural network is used for outputting face detection images without masks.
And inputting the face detection frame image output by the first convolutional neural network into the second convolutional neural network.
The second convolutional neural network is used for removing the mask, and more information is recovered from the image, so that the identification of the subsequent third convolutional neural network is facilitated.
Preferably, the training of the second convolutional neural network comprises the steps of:
(1) Constructing a second training set and training labels thereof, wherein the second training set comprises two training subsets, one training subset comprises a plurality of face detection images with face masks, and the other training subset comprises a plurality of face detection images without face masks, and the face detection images are marked with class labels which are expressed as the face detection images with face masks or the face detection images without face masks;
(2) And inputting the second training set and the training label into a second convolutional neural network for training.
In one embodiment, the training and testing of the second convolutional neural network specifically comprises the steps of:
(1) The thermal infrared face images of which the face masks are detected in the mode of A=5 ten thousand and the thermal infrared face images of which the face masks are not detected in the mode of B=5 ten thousand are combined into a training set, and the thermal infrared images of which the face masks are detected in the mode of C=1 ten thousand are obtained to serve as a testing set. Selecting 0-59 number of volunteers as training set, and 60-85 number of volunteers as test set. The thermal infrared face image mark for detecting the wearing mask is set as real_A, and the thermal infrared face image mark for detecting the non-wearing mask is set as real_B. And (3) building a convolutional neural network for removing the mask, and inputting the training set and the training label into the convolutional neural network for training, so as to obtain a required training model of the convolutional neural network.
The training image pair for the second convolutional neural network for the de-mask process is shown in fig. 4.
(1.1) Second convolutional neural network composition of the mask:
As shown in fig. 5, the structure of the second convolutional neural network is as follows: the neural network for mask removal has two Gan networks, each Gan network has a generator and a discriminator, the first Gan network includes a generator 1 and a discriminator a, and the second Gan network includes a generator 2 and a discriminator B. The first countermeasure network comprises a first generator and a first discriminator, the second countermeasure network comprises a second generator and a second discriminator, the class label of the face detection image of the person wearing the face is real_A, the class label of the face detection image of the person not wearing the face is real_B, the mask image generated by the second discriminator is marked as fake_A, and the mask image generated by the first discriminator is marked as fake_B; the first discriminator is used for judging whether the input image or the image generated by the second discriminator is real-A type, and the first generator is used for generating a fake-B type image according to the input or generated real-A type image; the second discriminator is used for judging whether the input image or the image generated by the first discriminator is real_B class, and the second generator is used for generating a fake_A class image according to the input or generated real_B class. The two mirror image Gan networks form a ring network, and the two Gan networks are mutually opposed, share information and learn together.
As shown in fig. 5 (a), the first gan network inputs a real_a image, and the determiner a determines whether the image is real_a, and the generator 1 generates a fake_b image, and the determiner B determines whether the image is real_b, and then sends the fake_b image to the generator 2 to generate a fake_a image.
The second gan network, as shown in fig. 5 (B), first determines whether a real_b image is inputted by the determiner B, generates a fake_a image by the generator 2, determines whether the image is real_a by the determiner a, and then sends the fake_a image to the generator 1 to generate a fake_b image.
The Generator (Generator) in the Gan network uses a Resblock-component network, the downsampling part is convolved with stride, and the upsampling part uses deconvolution. The arbiter (Discriminator) adopts PATCHGANS structure in the Pix2Pix network, and the size is 70x70.
(1.2) Loss function of network:
The loss function during training is divided into three parts, such as equation (1), the first part being the loss function of generator 1 and of arbiter a, generator 1 of real_a to rake_b being denoted G and arbiter a being denoted D x. The second part is the loss function of generator 2 and of arbiter B, which generator 2 of real_b to fake_a is denoted F and arbiter B is denoted D Y.
Loss Gan(G,Dx), G represents the generators of real_a to real_b, and Dx represents the discriminators of real_a to real_b. Loss Gan(F,DY) F denotes the generators of real_b to real_a, dy denotes the discriminant of real_b to real_a.
The third part is the countering loss, λ is the weighting factor used to control the weight of the cyclic consistency loss in the total loss.
Lossfull＝LossGan(G,Dx)+LossGan(F,DY)+λLosscycle (1)
The number of times (epoch) of the present invention was set to 200. Batch (batch) was set to 6. The input image is set to 256 x 256. The learning rate (Lr) was set to 0.0002. B1 in the optimizer (Adam) is set to 0.5, b2 is set to 0.999, the cyclic loss coefficient (lambda_cycle) is set to 10.0, and the self-loss coefficient (lambda_id) is set to 5.0.
If a traditional countermeasure generation network is used, the images of each person wearing the face mask and not wearing the face mask need to be in one-to-one correspondence, the face position of each person needs to be ensured not to incline, and when a data set is manufactured, the images of each thermal infrared face can be different due to breathing in the process of breathing the face mask, and when water vapor exists in expiration, the images cannot be completely registered with the images of the person not wearing the face mask. However, in the actual collection data set, this cannot be achieved, and the conventional countermeasure generation network uses the deviation between pixels to complete the image migration, if the data set is a problem of face inclination, the effect is poor in the generalization test. However, the countermeasure generation network does not need to perform registration training on images of each person, so that the trained model generalization capability is good.
(2) And inputting thermal infrared images in face data of the number 60-85 people wearing the face mask in the test set, and obtaining the thermal infrared face image with the face mask removed through a convolutional neural network. The generated mask-removed fake_b data is output.
Removing the mask front-to-back effect pair is shown in fig. 6, for example.
And S3, inputting output data of the second convolutional neural network into a trained third convolutional neural network, wherein the third convolutional neural network is used for outputting a face recognition result.
The input of the third convolutional neural network is the data processed by the second convolutional neural network, and the output is the recognized face information.
In one embodiment, the training of the third convolutional neural network specifically comprises the steps of:
(1) Constructing a third training set and training labels thereof, wherein a plurality of face detection images without masks and a plurality of face detection images without masks are marked with category labels and identity labels, the category labels are used for representing whether the face detection images without masks are represented or the face detection images without masks are represented, the category labels are used for distinguishing different people, and the identity labels of the face detection images without masks and the face detection images without masks are identical;
(2) And inputting the third training set and the training label into a third convolutional neural network for training.
In one embodiment, the training and testing of the third convolutional neural network specifically comprises the steps of:
(1) D=2 ten thousand thermal infrared face images for detecting the unworn face mask and e=2 ten thousand thermal infrared face images for removing the unworn face mask are combined to form a training set, and f=4ten thousand Zhang Regong external images are obtained as a testing set. The images of the person who is not wearing the mask and the mask is removed are marked together. Building a convolutional neural network, and inputting a training set and a training label into the convolutional neural network together for training, so as to obtain a required training model of the convolutional neural network;
as shown in fig. 7, the nine-grid loss function during training is preferably:
In Loss1, i denotes the ith image, N denotes the N sets of samples, real_b denotes the image without mask, real_b_other denotes the different images without mask, and other denotes the images of different persons. Loss1 is used to pull up Euclidean distances between real_B classes and to pull up Euclidean distances from different classes. The euclidean distance may be replaced by other calculation methods that represent differences between pictures.
Real_b in Loss2 represents an image of an unworn mask, like_b represents a generated image of an unworn mask, and other represents an image of a different person. Loss2 is used to pull up the Euclidean distance between real_B and fake_B, and to pull up the Euclidean distance from different classes.
In Loss3, fake_b represents the generated images of the mask not being worn, fake_b_other represents the generated images of the different persons not being worn, and other represents the images of the different persons. Loss3 is used to pull up Euclidean distances between the like of the fake_B and to pull up Euclidean distances from the different like.
(2) And inputting the thermal infrared images in the test set, and obtaining the information of the people to whom the thermal infrared face images belong through a convolutional neural network. The performance of the model was checked using TAR (correct acceptance rate), FAR (false acceptance rate) and Accuracy (recognition rate) as evaluation indexes. Assume that there are currently total True sets of identical person pairs { two different thermal infrared face images of identical person } and total False sets of different person pairs { two thermal infrared face images of different person }, TAR and FAR are defined as follows:
wherein total TrueAccept is the number of the same person to which the model determines to belong; total FalseAccept is how many different pairs of people are determined by the model to belong to the same number of people; total FalseReject is the number of different people that the model determines to belong to. Different maximum similarity distances are chosen, so that different TARs and FARs can be obtained. The data set used contained 20000 co-person pairs, 20000 different person pairs. The area AUC of the ROC was 0.512. Maximum similarity distance=18, and recognition rate is 89.68%.
A thermal infrared image face recognition method of a mask according to another embodiment of the present invention is shown in fig. 8.
The thermal infrared image face recognition system of the embodiment of the invention comprises:
The first module is used for acquiring a thermal infrared face image to be recognized, inputting the thermal infrared face image into a trained first convolution neural network, and detecting whether the thermal infrared face image is a mask wearing image and a face detection frame or not;
The second module is used for inputting the output data of the first convolutional neural network to a trained second convolutional neural network, and the second convolutional neural network is used for outputting face detection images without masks;
And the third module is used for inputting the output data of the second convolutional neural network to a trained third convolutional neural network, and the third convolutional neural network is used for outputting the face recognition result.
The implementation principle and technical effect of the system are similar to those of the method, and are not repeated here.
The embodiment also provides an electronic device, which includes at least one processor and at least one memory, where the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the thermal infrared image face recognition method of the mask in the above embodiment, which are not repeated herein; in the present embodiment, the types of the processor and the memory are not particularly limited, for example: the processor may be a microprocessor, digital information processor, on-chip programmable logic system, or the like; the memory may be volatile memory, non-volatile memory, a combination thereof, or the like.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, the computer program being executed by a processor to implement the technical scheme of the thermal infrared image face recognition method embodiment of any of the above face masks. The implementation principle and technical effects are similar to those of the method, and are not repeated here.
It should be noted that, in any of the above embodiments, the methods are not necessarily sequentially executed in the sequence number, and it is meant that the methods may be executed in any other possible sequence, as long as it cannot be inferred from the execution logic that the methods are necessarily executed in a certain sequence.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (10)
1. The thermal infrared image face recognition method for the mask is characterized by comprising the following steps of:
Acquiring a thermal infrared face image to be identified, and inputting the thermal infrared face image into a trained first convolutional neural network, wherein the first convolutional neural network is used for detecting whether the thermal infrared face image is a mask wearing image and a face detection frame;
inputting the output data of the first convolutional neural network into a trained second convolutional neural network, wherein the second convolutional neural network is used for outputting face detection images without masks;
And inputting the output data of the second convolutional neural network into a trained third convolutional neural network, wherein the third convolutional neural network is used for outputting a face recognition result.
2. A method of facial recognition from thermal infrared images of a mask as set forth in claim 1, wherein the training of the first convolutional neural network comprises the steps of:
constructing a first training set and training labels thereof, wherein the first training set comprises a plurality of thermal infrared face images of different people wearing the face masks and a plurality of thermal infrared face images of different people not wearing the face masks, and the training labels of the thermal infrared face images comprise a face calibration frame and class labels for indicating whether the face masks are worn or not;
And setting a plurality of anchor blocks for the first convolutional neural network, and inputting the first training set and the training labels thereof into the first convolutional neural network for training.
3. A method of facial recognition from thermal infrared images of a mask as set forth in claim 1, wherein the training of the second convolutional neural network comprises the steps of:
Constructing a second training set and training labels thereof, wherein the second training set comprises two training subsets, one training subset comprises a plurality of face detection images with face masks, and the other training subset comprises a plurality of face detection images without face masks, and the face detection images are marked with class labels which are expressed as the face detection images with face masks or the face detection images without face masks;
and inputting the second training set and the training label into the second convolutional neural network for training.
4. A method of facial recognition from thermal infrared images of a mask as set forth in claim 3, wherein the training of said third convolutional neural network comprises the steps of:
Constructing a third training set and training labels thereof, wherein a plurality of face detection images without masks and a plurality of face detection images without masks are formed in the third training set, the face detection images are marked with category labels and identity labels, the category labels are used for representing whether the face detection images without masks are represented or the face detection images without masks are represented, and the category labels are used for distinguishing different people, and the identity labels of the face detection images without masks and the face detection images without masks belonging to the same person are the same;
and inputting the third training set and the training label into the third convolutional neural network for training.
5. A thermal infrared image face recognition method of a mask as defined in claim 4, wherein the second convolutional neural network comprises a first countermeasure network and a second countermeasure network, the first countermeasure network comprises a first generator and a first discriminator, the second countermeasure network comprises a second generator and a second discriminator, the class label of the mask-worn face detection image is real_a, the class label of the mask-not-worn face detection image is real_b, the mask-worn image generated by the second discriminator is fake_a, and the mask-not-worn image generated by the first discriminator is fake_b;
the first discriminator is used for judging whether the input image or the image generated by the second discriminator is real-A type or not, and the first generator is used for generating a fake-B type image according to the input or generated real-A type image;
the second discriminator is used for judging whether the input image or the image generated by the first discriminator is real_B class or not, and the second generator is used for generating a fake_A class image according to the input or generated real_B class.
6. A thermal infrared image face recognition method of a face mask according to claim 5, wherein the loss function of the second convolutional neural network is:
Lossfull＝LossGan(G,Dx)+LossGan(F,DY)+λLosscycle
Wherein Loss full is a Loss function, the first generator is denoted as G, the first arbiter is denoted as D x,LossGan(G,Dx) represents the Loss functions of the first generator and the first arbiter, the second generator is denoted as F, the second arbiter is denoted as D y,LossGan(F,DY) represents the Loss functions of the second generator and the second arbiter, loss cycle is a counterloss, and λ is a weighting factor.
7. A thermal infrared image face recognition method of a face mask according to claim 5, wherein the nine-grid loss function of the third convolutional neural network is:
Wherein i represents an ith image, N represents the total number of samples of the third training set, real_B_other represents different images of real_B class, other represents images belonging to different people, and fake_B_other represents different images of fake_B class;
loss1 is a Loss function for zooming in the difference between real_b class images and zooming out the difference between different class images;
Loss2 is a Loss function for zooming in the differences between real_b class images and fake_b class images, zooming out the differences between different class images;
loss3 is a Loss function for approximating the difference between images of the same fake B class and enlarging the difference between images of different classes.
8. A thermal infrared image face recognition system for a mask, comprising:
The first module is used for acquiring a thermal infrared face image to be recognized, inputting the thermal infrared face image into a trained first convolution neural network, and detecting whether the thermal infrared face image is a mask wearing image or not and a face detection frame;
The second module is used for inputting the output data of the first convolutional neural network to a trained second convolutional neural network, and the second convolutional neural network is used for outputting face detection images without masks;
And the third module is used for inputting the output data of the second convolutional neural network to a trained third convolutional neural network, and the third convolutional neural network is used for outputting a face recognition result.
9. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110960380.3A CN113723243B (en) | 2021-08-20 | 2021-08-20 | Face recognition method of thermal infrared image of wearing mask and application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110960380.3A CN113723243B (en) | 2021-08-20 | 2021-08-20 | Face recognition method of thermal infrared image of wearing mask and application |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113723243A CN113723243A (en) | 2021-11-30 |
| CN113723243B true CN113723243B (en) | 2024-05-17 |
Family
ID=78677109
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110960380.3A Active CN113723243B (en) | 2021-08-20 | 2021-08-20 | Face recognition method of thermal infrared image of wearing mask and application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113723243B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114821702A (en) * | 2022-03-15 | 2022-07-29 | 电子科技大学 | Thermal infrared face recognition method based on face shielding |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295521A (en) * | 2016-07-29 | 2017-01-04 | 厦门美图之家科技有限公司 | A kind of gender identification method based on multi output convolutional neural networks, device and the equipment of calculating |
| CN107609481A (en) * | 2017-08-14 | 2018-01-19 | 百度在线网络技术（北京）有限公司 | The method, apparatus and computer-readable storage medium of training data are generated for recognition of face |
| CN109086718A (en) * | 2018-08-02 | 2018-12-25 | 深圳市华付信息技术有限公司 | Biopsy method, device, computer equipment and storage medium |
| CN109508694A (en) * | 2018-12-10 | 2019-03-22 | 上海众源网络有限公司 | A kind of face identification method and identification device |
| CN109543640A (en) * | 2018-11-29 | 2019-03-29 | 中国科学院重庆绿色智能技术研究院 | A kind of biopsy method based on image conversion |
| CN110378234A (en) * | 2019-06-20 | 2019-10-25 | 合肥英威晟光电科技有限公司 | Convolutional neural networks thermal imagery face identification method and system based on TensorFlow building |
| CN110516576A (en) * | 2019-08-20 | 2019-11-29 | 西安电子科技大学 | Near-infrared living body faces recognition methods based on deep neural network |
| CN110633691A (en) * | 2019-09-25 | 2019-12-31 | 北京紫睛科技有限公司 | Binocular in-vivo detection method based on visible light and near-infrared camera |
| CN110991266A (en) * | 2019-11-13 | 2020-04-10 | 北京智芯原动科技有限公司 | Binocular face living body detection method and device |
| WO2020159437A1 (en) * | 2019-01-29 | 2020-08-06 | Agency For Science, Technology And Research | Method and system for face liveness detection |
| CN111695406A (en) * | 2020-04-23 | 2020-09-22 | 西安电子科技大学 | Face recognition anti-spoofing method, system and terminal based on infrared ray |
| CN111723655A (en) * | 2020-05-12 | 2020-09-29 | 五八有限公司 | Face image processing method, device, server, terminal, equipment and medium |
| CN111767877A (en) * | 2020-07-03 | 2020-10-13 | 北京视甄智能科技有限公司 | Living body detection method based on infrared features |
| CN111931594A (en) * | 2020-07-16 | 2020-11-13 | 广州广电卓识智能科技有限公司 | Face recognition living body detection method and device, computer equipment and storage medium |
| CN112633130A (en) * | 2020-12-18 | 2021-04-09 | 成都三零凯天通信实业有限公司 | Face mask removing method based on key point restoration image |
| CN112818722A (en) * | 2019-11-15 | 2021-05-18 | 上海大学 | Modular dynamically configurable living body face recognition system |
| CN113128481A (en) * | 2021-05-19 | 2021-07-16 | 济南博观智能科技有限公司 | Face living body detection method, device, equipment and storage medium |
| CN113221771A (en) * | 2021-05-18 | 2021-08-06 | 北京百度网讯科技有限公司 | Living body face recognition method, living body face recognition device, living body face recognition equipment, storage medium and program product |
-
2021
- 2021-08-20 CN CN202110960380.3A patent/CN113723243B/en active Active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295521A (en) * | 2016-07-29 | 2017-01-04 | 厦门美图之家科技有限公司 | A kind of gender identification method based on multi output convolutional neural networks, device and the equipment of calculating |
| CN107609481A (en) * | 2017-08-14 | 2018-01-19 | 百度在线网络技术（北京）有限公司 | The method, apparatus and computer-readable storage medium of training data are generated for recognition of face |
| CN109086718A (en) * | 2018-08-02 | 2018-12-25 | 深圳市华付信息技术有限公司 | Biopsy method, device, computer equipment and storage medium |
| CN109543640A (en) * | 2018-11-29 | 2019-03-29 | 中国科学院重庆绿色智能技术研究院 | A kind of biopsy method based on image conversion |
| CN109508694A (en) * | 2018-12-10 | 2019-03-22 | 上海众源网络有限公司 | A kind of face identification method and identification device |
| WO2020159437A1 (en) * | 2019-01-29 | 2020-08-06 | Agency For Science, Technology And Research | Method and system for face liveness detection |
| CN110378234A (en) * | 2019-06-20 | 2019-10-25 | 合肥英威晟光电科技有限公司 | Convolutional neural networks thermal imagery face identification method and system based on TensorFlow building |
| CN110516576A (en) * | 2019-08-20 | 2019-11-29 | 西安电子科技大学 | Near-infrared living body faces recognition methods based on deep neural network |
| CN110633691A (en) * | 2019-09-25 | 2019-12-31 | 北京紫睛科技有限公司 | Binocular in-vivo detection method based on visible light and near-infrared camera |
| CN110991266A (en) * | 2019-11-13 | 2020-04-10 | 北京智芯原动科技有限公司 | Binocular face living body detection method and device |
| CN112818722A (en) * | 2019-11-15 | 2021-05-18 | 上海大学 | Modular dynamically configurable living body face recognition system |
| CN111695406A (en) * | 2020-04-23 | 2020-09-22 | 西安电子科技大学 | Face recognition anti-spoofing method, system and terminal based on infrared ray |
| CN111723655A (en) * | 2020-05-12 | 2020-09-29 | 五八有限公司 | Face image processing method, device, server, terminal, equipment and medium |
| CN111767877A (en) * | 2020-07-03 | 2020-10-13 | 北京视甄智能科技有限公司 | Living body detection method based on infrared features |
| CN111931594A (en) * | 2020-07-16 | 2020-11-13 | 广州广电卓识智能科技有限公司 | Face recognition living body detection method and device, computer equipment and storage medium |
| CN112633130A (en) * | 2020-12-18 | 2021-04-09 | 成都三零凯天通信实业有限公司 | Face mask removing method based on key point restoration image |
| CN113221771A (en) * | 2021-05-18 | 2021-08-06 | 北京百度网讯科技有限公司 | Living body face recognition method, living body face recognition device, living body face recognition equipment, storage medium and program product |
| CN113128481A (en) * | 2021-05-19 | 2021-07-16 | 济南博观智能科技有限公司 | Face living body detection method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 高效防伪人脸识别系统;纪侨斌;徐树公;;工业控制计算机(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113723243A (en) | 2021-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| George et al. | On the effectiveness of vision transformers for zero-shot face anti-spoofing | |
| CN108537743B (en) | Face image enhancement method based on generation countermeasure network | |
| CN110210276A (en) | A kind of motion track acquisition methods and its equipment, storage medium, terminal | |
| Chen et al. | End-to-end learning of object motion estimation from retinal events for event-based object tracking | |
| CN110008846B (en) | Image processing method | |
| CN111860400B (en) | Face enhancement recognition method, device, equipment and storage medium | |
| CN111914761A (en) | Thermal infrared face recognition method and system | |
| CN111639580B (en) | Gait recognition method combining feature separation model and visual angle conversion model | |
| Khan et al. | Group-level emotion recognition using deep models with a four-stream hybrid network | |
| CN116343330A (en) | Abnormal behavior identification method for infrared-visible light image fusion | |
| Yuan et al. | MFFFLD: A multimodal-feature-fusion-based fingerprint liveness detection | |
| CN112183504B (en) | Video registration method and device based on non-contact palm vein image | |
| US20110182497A1 (en) | Cascade structure for classifying objects in an image | |
| CN115115552B (en) | Image correction model training method, image correction device and computer equipment | |
| CN113723243B (en) | Face recognition method of thermal infrared image of wearing mask and application | |
| Chu et al. | Semi-supervised 3d human pose estimation by jointly considering temporal and multiview information | |
| CN112464864A (en) | Face living body detection method based on tree-shaped neural network structure | |
| Lau et al. | Atdetect: Face detection and keypoint extraction at range and altitude | |
| Galiyawala et al. | Dsa-pr: discrete soft biometric attribute-based person retrieval in surveillance videos | |
| CN112183422A (en) | Human face living body detection method and device based on space-time characteristics, electronic equipment and storage medium | |
| JPH11283036A (en) | Object detector and object detection method | |
| CN110705345A (en) | Pedestrian re-identification method and system based on deep learning | |
| Rao | Implementation of Low Cost IoT Based Intruder Detection System by Face Recognition using Machine Learning | |
| CN113554007B (en) | Face frame calculation method and calculation system | |
| Dhamija et al. | Analysis of age invariant face recognition using quadratic support vector machine-principal component analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |