KR20220107536A - Method and system for robust face recognition system to wearing a mask - Google Patents

Abstract

The present invention relates to a face recognition method and system robust to wearing of a mask. According to an embodiment of the present invention, a face recognition system robust to wearing of a mask may include: a learning part for labelling collected image data and learning a deep learning network model using the labeled image data; an input part for inputting a user's face image; a detection part for detecting whether wearing the mask exists in the input face image; a pre-processing part for performing image pre-processing based on whether wearing the mask exists; and a determination part for comparing a result of performing image preprocessing with the user's face image pre-stored in a database to determine whether they are matched or not matched.

Description

Face recognition method and system robust to wearing a mask

The present invention relates to face recognition technology, and more particularly, to a face recognition method and system robust to wearing a mask using deep learning technology.

Face Recognition technology is one of the fields of biometrics. It refers to a technology in which a machine automatically identifies and authenticates a person using the unique feature information contained in each person's face. It has superior security compared to the authentication method, and has been widely used in various fields recently.

Conventional face recognition methods perform face recognition with traditional algorithms such as IFT, SURF, and BSIF, but recently, these algorithms have been replaced with deep learning methods to provide high recognition performance.

However, there is a problem in that the conventional face recognition method cannot distinguish the same person when a face is photographed in different environments despite the same person.

Also, the conventional face recognition method has a problem in that it does not perform well when recognition is attempted when a part of the face is covered, such as wearing a mask.

Therefore, it is necessary to include a model that can recognize faces using only the area around the eyes, excluding the areas covered by the mask.

1. Korean Patent Publication No. 2020-0084427 “Face Recognition System Extracting Feature Vector Using Deep Learning-based Facial Recognition Model” (Publication Date: July 13, 2020)

The present invention provides a face recognition method and system robust to wearing a mask that can recognize a face regardless of whether or not a mask is worn through a deep learning model trained with an image.

According to one aspect of the present invention, there is provided a face recognition method and system robust to wearing a mask.

A face recognition system robust to wearing a mask according to an embodiment of the present invention includes: a learning unit for labeling collected image data, and learning a deep learning network model using the labeled image data; an input unit for inputting a user's face image; a sensing unit for detecting whether a mask is worn on the input face image; a preprocessor for performing image preprocessing based on whether the mask is worn; and a determination unit that compares the image pre-processing result and the user's face image pre-stored in the database to determine whether they match or not.

According to an embodiment of the present invention, when the input user's face image does not wear a mask, the preprocessor may crop an image of an entire face region from the input user's face image.

According to an embodiment of the present invention, when the input image does not wear a mask, the determination unit may determine whether the image of the entire face area matches or does not match the user's face image pre-stored in the database. have.

According to an embodiment of the present invention, when the inputted user's face image wears a mask, the preprocessor may crop an image of a region around the eyes from the inputted user's face image.

According to an embodiment of the present invention, when the input image is wearing a mask, the determination unit may determine whether the image of the area around the eye matches or does not match the user's face image pre-stored in the database. .

According to an embodiment of the present invention, the face recognition method and system robust to wearing a mask has an advantage in that the face recognition system detects wearing of the mask and can perform face recognition without taking off or lowering the mask even when wearing the mask.

In addition, the face recognition method and system robust to wearing a mask has an advantage in that it is possible to increase the accuracy of face recognition with less data because a Siamese network capable of one-shot learning is used.

In addition, since face recognition is possible by using the database registered in the existing face recognition system as it is, there is an advantage in that the database construction time and cost are not incurred.

According to an embodiment of the present invention, the face recognition method robust to wearing a mask analyzes the correlation between the facial and eye movements, and more accurate measurement than the dementia screening method using only eye movements is possible.

1 to 3 are views for explaining a face recognition method robust to wearing a mask according to an embodiment of the present invention.
4 is a diagram for explaining a deep learning network model according to an embodiment of the present invention.
5 is a diagram for explaining a deep learning model of a face recognition method robust to wearing a mask according to an embodiment of the present invention.
6 is a view for explaining an ROC curve of a face recognition method robust to wearing a mask according to an embodiment of the present invention.
7 is a view for explaining a face recognition system robust to wearing a mask according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Also, as used in this specification and claims, the terms "a" and "a" and "a" are to be construed to mean "one or more" in general, unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do it with

1 is a view for explaining a face recognition method robust to wearing a mask according to an embodiment of the present invention.

Referring to FIG. 1 , a face recognition system 10 robust to wearing a mask may perform a face recognition method robust to wearing a mask by utilizing a deep learning network used for image processing. A face recognition method that is robust to wearing a mask is a face recognition method that is not affected by the location where the face is photographed, lighting, or wearing a mask.

In step S10 , the face recognition system 10 robust to wearing a mask may label the face image data collected for face recognition.

In step S20 , the face recognition system 10 robust to wearing a mask may learn a network model to be used for face recognition using the labeled image. In this case, a siamese network may be used as the network model.

The face recognition system 10 that is robust to wearing a mask generates a feature vector through convolution on an image input through a Siamese network model and an image registered in the face database DB, and then By calculating the distance, if the distance is less than the threshold for the image with the smallest distance, it is possible to calculate the result that the image is the same person. Such a Siamese network is capable of one-shot learning, so that it can learn efficiently on a small amount of data.

과

는 각각 input image의 특징벡터를 의미하고, margin을 뜻하는 m은 2.0으로 설정하였다. Dw는 두 특징벡터 간의 유클리디언 거리(Euclidean distance)이다. 만약

과

가 동일 인물(genuine pair)이라면 Y=0이고, 다른 인물(imposter pair)이면 Y=1이 할당된다. 따라서 이 함수에 의하면 동일 인물(genuine pair)의 거리(distance)를 더욱 낮게 예측할수록, 동일 인물(imposter pair)의 거리(distance)를 더욱 높게 예측할수록 L값은 작아진다. 이때, 본 발명에 따른 샴 네트워크 모델은 L값이 최소가 되는 방향으로 학습할 수 있다.The face recognition system 10 robust to wearing a mask may use the contrast loss function of Equation 1 for learning a siamese network after obtaining a distance between two input image feature vectors. W is the weight, Y is the label,

class

denotes a feature vector of the input image, respectively, and m, which denotes margin, was set to 2.0. Dw is the Euclidean distance between two feature vectors. what if

class

If is the same person (genuine pair), Y = 0, and if it is a different person (imposter pair), Y = 1 is assigned. Therefore, according to this function, the lower the distance between the same person (genuine pair) is predicted, and the higher the distance (distance) of the same person (imposter pair) is predicted, the smaller the L value. In this case, the Siamese network model according to the present invention may be trained in a direction in which the L value is minimized.

[Equation 1]

Therefore, the face recognition system 10 robust to wearing a mask learns a face recognition model wearing a mask by collecting black and white face data without a mask and face data wearing a mask through a Siamese network model. can do.

In step S30 , the face recognition system 10 robust to wearing a mask may input the user's face image into the network model.

In step S35 , the face recognition system 10 robust to wearing a mask may register the user's face image in the face database. In this case, the user's face image registered in the database may be an image including the entire face of the user who does not wear a mask.

In step S40, the face recognition system 10 robust to wearing a mask may detect whether the input image is wearing a mask.

In step S50 , the face recognition system 10 robust to wearing a mask may perform image preprocessing of a full face region when an input image is not wearing a mask.

The face recognition system 10 robust to wearing a mask may detect a landmark of the face by using the Adrian method for image preprocessing of the full face region.

As shown in FIG. 2 , in the face image 20 in which the mask is not worn, two areas of the entire face area 30 and the eye area 40 may be cropped. In this case, the face recognition system 10 robust to wearing a mask may generate a bounding box at a position to be cropped for pre-processing of data wearing a mask. In this case, the size of the entire face region in the input image may be resized to 120×120, and the size of the area around the eyes may be resized to 91×64.

Referring again to FIG. 1 , in step S60 , the face recognition system 10 robust to wearing a mask may recognize a full face when a full face image pre-processing is performed. To this end, the face recognition system 10 robust to wearing a mask may compare the input image of the learned network model with the user's face image registered in the face database to determine whether they match or not.

That is, the face recognition system 10 robust to wearing a mask may first check whether a mask is worn when a user's face is input, and then perform face recognition using a full face when not wearing a mask. .

In step S70 , the face recognition system 10 robust to wearing a mask may perform pre-processing of a periocular region that is not covered by the mask when the face photograph is worn with the mask.

The face recognition system 10 robust to wearing a mask may detect a landmark of the face using the Adrian method for preprocessing the periocular region.

3, the face recognition system 10 robust to wearing a mask crops the entire face area 60 and the area around the eyes 70 that are not covered by the mask in the face image 50 wearing the mask ( crop) can be done. In this case, the face recognition system 10 robust to wearing a mask may generate a bounding box at a position to be cropped for pre-processing of data wearing a mask.

Referring back to FIG. 1 , in step S80 , the face recognition system 10 robust to wearing a mask may perform periocular image preprocessing to recognize a periocular region in the image.

Thereafter, the face recognition system 10 robust to wearing a mask compares the input image of the learned network model with the user's face image registered in the face database to determine whether there is a match or a discrepancy.

Through this, the face recognition method robust to wearing a mask can recognize a face regardless of whether a mask is worn through a deep learning model trained with a face image. In addition, it can have the effect of reducing the inconvenience of having to take off the mask for face recognition and verifying identity in a safe way.

4 is a diagram for explaining a deep learning network model according to an embodiment of the present invention.

As shown in Figure 4, the face recognition system 10 robust to wearing a mask after the input image input to the deep learning network model is resized to 105 × 105, a plurality of convolution layers and It is reduced to a feature vector of 100×1 through a plurality of max pooling layers, and a distance calculation process may be performed.

5 is a diagram for explaining a deep learning model of a face recognition method robust to wearing a mask according to an embodiment of the present invention.

As shown in FIG. 5 , the face recognition system 10 robust to wearing a mask may learn a face image using a deep learning model. Face recognition is a system that matches the input image as many as the number of users registered in the database. To this end, the deep learning model uses a Siamese network model to perform 1:1 matching can be performed.

The Siamese network model may include a Siamese-face model and a Siamese-periocular model.

The Siamese-face model is a deep learning-based model trained on unmasked face images, and the Siamese-periocular model is trained on unmasked periocular images. It is a deep learning-based model.

A face recognition system 10 that is robust to wearing a mask tests unmasked images and masked images through a Siamese-face model, and in this case, the accuracy of the model is 93%, respectively, 63%.

A face recognition system that is robust to wearing a mask 10 tests unmasked images and masked images through a Siamese-periocular, in which case the accuracy of the model is 92%, respectively, 66%.

6 is a view for explaining an ROC curve of a face recognition method robust to wearing a mask according to an embodiment of the present invention.

Referring to FIG. 6 , the face recognition system 10 robust to wearing a mask tests the Siamese-fac model as masked face images, and the Siamese-periocular model is masked. Receiver operating characteristic curve (ROC curve) of the test result with periocular images.

In this case, the horizontal axis of the graph shown in FIG. 6 means false positive rate (FPR), and the vertical axis means true positive rate (TPR). In addition, the area under the curve (AUC) of the Siamese-face model is 0.6936 and that of the Siamese-periocular model is 0.7378. (True = genuine matching, False = imposter marching)

7 is a view for explaining a face recognition system robust to wearing a mask according to an embodiment of the present invention.

Referring to FIG. 7 , the face recognition system 10 robust to wearing a mask may include a learning unit 100 , an input unit 200 , a sensing unit 300 , a preprocessing unit 400 , and a determining unit 500 . .

The learning unit 100 may learn a deep learning network model to be used for face recognition by using the labeled image. The deep learning network model used by the learning unit 100 may be a Siamese network model.

The input unit 200 may acquire a user's face image. The face video image may be an image obtained by capturing a face viewing an image for 20 minutes using a camera at 30 frames per second.

According to an embodiment, the input unit 200 may input a face image through a Siamese network model.

The sensing unit 300 may determine whether to wear a mask when a face image is input from the input unit 200 .

The preprocessor 400 may preprocess the input face image based on whether or not the mask is worn. The preprocessor 400 may extract a region of interest.

The preprocessor 400 may adjust the size of the face image to a size required for deep learning using the extracted ROI. In this case, the preprocessor 400 may be set to include as much information as possible in the image by extracting the region of interest and adjusting the size.

The preprocessor 400 may extract the landmark of the face by using the Adrian method from the input image face image in order to extract the region of interest.

The preprocessor 400 may crop at least one of a full face area and a periocular area based on the extracted landmark.

According to an embodiment, when the acquired face image does not wear a mask, the preprocessor 400 may perform a preprocessing of cropping an image of a full face region.

According to an embodiment, when the acquired face image is an image wearing a mask, the preprocessor 400 may perform preprocessing of cropping a periocular image.

The determination unit 500 compares the image pre-processing result received from the pre-processing unit 400 with the image pre-stored in the database to perform at least one of full face area recognition and periocular recognition. can

The determination unit 500 may determine whether the input image matches or does not match by using information in a database of a user registered as a pre-learned model according to whether or not the mask is worn.

According to an embodiment, when the input image does not wear a mask, the determination unit may determine whether the image of the entire face area matches or does not match the user's face image pre-stored in the database.

According to an embodiment, when the input image is wearing a mask, the determination unit may determine whether the image of the area around the eye matches or does not match the user's face image pre-stored in the database.

According to an embodiment, when the input image does not wear a mask, the determination unit 500 may be configured to perform a relationship between a masked image and a user's face image pre-stored in a database through a Siamese-face model. Concordance and inconsistency can be determined.

According to an embodiment, when the input image is wearing a mask, the determination unit 500 matches unmasked images and a user's face image pre-stored in the database through a Siamese-periocular model. and discrepancy.

The above-described face recognition method robust to wearing a mask may be implemented as a computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). can The computer program recorded in the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

In the above, even though all the components constituting the embodiment of the present invention are described as being combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more.

Although acts are shown in a particular order in the drawings, it should not be understood that the acts must be performed in the specific order or sequential order shown, or that all depicted acts must be performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is

Up to now, the present invention has been looked at focusing on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

10: Face recognition system that is robust to wearing a mask
100: study department
200: input unit
300: detection unit
400: preprocessor
500: judgment unit

Claims (11)

In a face recognition system that is robust to wearing a mask,
a learning unit for labeling the collected image data and learning a deep learning network model using the labeled image data;
an input unit for inputting a user's face image;
a sensing unit for detecting whether a mask is worn on the input face image;
a preprocessor for performing image preprocessing based on whether the mask is worn; and
Comprising a determination unit that compares the image preprocessing result and the user's face image pre-stored in the database to determine whether there is a match or a mismatch
A face recognition system that is robust to wearing a mask.
According to claim 1,
The preprocessor
If the input user face image does not wear a mask,
Cropping the image of the entire face area from the input user face image
A face recognition system that is robust to wearing a mask.
3. The method of claim 2,
the judging unit
If the input image does not wear a mask,
Determining whether the image of the entire face area matches or does not match with the user's face image pre-stored in the database
A face recognition system that is robust to wearing a mask.
According to claim 1,
The preprocessor
When the input user face image wears a mask,
Cropping the image of the eye area from the input user's face image
A face recognition system that is robust to wearing a mask.
5. The method of claim 4,
the judging unit
If the input image is wearing a mask,
Determining whether the image of the area around the eye matches or does not match the user's face image pre-stored in the database
A face recognition system that is robust to wearing a mask.
In the face recognition method robust to wearing a mask,
labeling the collected image data;
Learning a deep learning network model using the labeled image data;
inputting a user's face image;
detecting whether a mask is worn on the input face image;
performing image pre-processing based on whether the mask is worn; and
Comparing the image pre-processing result and the user's face image pre-stored in the database to determine whether there is a match or inconsistency
A face recognition method that is robust to wearing a mask.
7. The method of claim 6,
The step of performing image pre-processing based on whether the mask is worn
If the input user face image does not wear a mask,
Cropping an image of an entire face region from the input user face image
A face recognition method that is robust to wearing a mask.
8. The method of claim 7,
The step of determining whether the match and the inconsistency are
If the input image does not wear a mask,
Comprising the step of determining whether the image of the entire face area matches or does not match with the user's face image pre-stored in the database
A face recognition method that is robust to wearing a mask.
7. The method of claim 6,
The step of performing image pre-processing based on whether the mask is worn
When the input user face image wears a mask,
Cropping an image of an eye area from the input user's face image
A face recognition method that is robust to wearing a mask.
10. The method of claim 9,
The step of determining whether the match and the inconsistency are
If the input image is wearing a mask,
and determining whether the image of the area around the eye matches or does not match the user's face image pre-stored in the database.
A face recognition method that is robust to wearing a mask.
A computer program recorded in a computer-readable recording medium executing any one of the face recognition methods robust to the mask wearing of claims 6 to 10.

Images