KR102380426B1 - Method and apparatus for verifying face - Google Patents

Abstract

A face authentication method and apparatus are disclosed. A face authentication method according to an embodiment includes detecting a face region from an input image, determining whether an input face displayed in the face region is a partial face, and in response to determining that the input face is a partial face, a reference image and generating a synthesized image by synthesizing the face image of the face region; and determining whether authentication succeeds based on the synthesized image.

Description

Face authentication method and device {METHOD AND APPARATUS FOR VERIFYING FACE}

The description below relates to face authentication technology.

Face authentication technology, which is one of biometric authentication technologies, is an authentication technology that determines whether a user is a legitimate user based on a face displayed in a still image or video. The face authentication technology has the advantage of being able to confirm the authentication target in a contactless manner. Recently, because of the convenience and efficiency of the face authentication technology, the face authentication technology has been widely used in various application fields such as security systems, mobile authentication, and multimedia data search.

A face authentication method according to an embodiment may include detecting a face region in an input image; determining whether the input face displayed in the face region is a partial face; generating a synthesized image by synthesizing a reference image and a face image of the face region in response to determining that the input face is a partial face; and determining whether authentication succeeds based on the synthesized image.

In the face authentication method according to an embodiment, the determining whether the input face is a partial face may include determining whether the input face is a partial face based on a position of the face region detected in the input image. .

In the face authentication method according to an embodiment, the determining whether the input face is a partial face may include determining whether the input face is a partial face when the detected face region includes a part of a boundary of the input image. can decide

In the face authentication method according to an embodiment, the generating of the synthesized image includes generating the synthesized image by synthesizing image information of the reference image and image information of the face image based on the detected face region. The image information of the reference image may be image information of a region corresponding to the rest of the face region excluding the detected face region in the face region corresponding to the entire face.

In the face authentication method according to an embodiment, determining whether the input face is a partial face may include: detecting landmarks of the face in the face region; and determining that the input face is a partial face when landmarks corresponding to at least one of predefined face region regions are not detected in the detected face region.

In the face authentication method according to an embodiment, the generating of the synthesized image comprises synthesizing image information of a corresponding region of the reference image corresponding to a facial region in which the landmarks are not detected and image information of the face image. Thus, the composite image can be generated.

In the face authentication method according to an embodiment, determining whether authentication succeeds based on the face image may include: extracting a feature of the input face based on a difference image between the face image and the reference image; and determining whether authentication succeeds based on a comparison result between the extracted features and registered features corresponding to all faces among the registered features.

A face authentication method according to an embodiment may include: determining whether an occlusion area exists in an input face displayed in the face image; and in response to determining that the occlusion region exists, generating a synthesized image by synthesizing image information of a corresponding region corresponding to the occlusion region in the reference image and image information of the face image. there is.

A face authentication apparatus according to an embodiment includes a processor, wherein the processor detects a face region from an input image, determines whether an input face displayed in the face region is a partial face, and wherein the input face is a partial face In response to the determination, a synthesized image may be generated by synthesizing the reference image and the face image of the face region, and whether authentication succeeds may be determined based on the synthesized image.

According to an embodiment, an apparatus including a neural network model includes: a common feature extractor that receives a face image and outputs an intermediate feature; a first feature extractor connected to the common feature extractor and configured to receive the intermediate feature and extract a feature to be used for face authentication when the face image includes full face information of the user; and a second feature extractor connected to the common feature extractor and configured to receive the intermediate feature when the face image includes partial face information of the user and extract a feature to be used for face authentication.

A computing device according to an embodiment includes: a camera for acquiring a face image of a user; and a processor for performing face authentication by comparing the acquired face image with a registered face image, wherein the computing device is configured to display the face on a display screen of the computing device that the user can see while the face authentication is performed. The acquired face image may not be displayed.

1A and 1B are diagrams for explaining face authentication according to an embodiment.
2A to 2D are diagrams illustrating examples of images in which a partial face is captured.
3 to 5 are flowcharts illustrating a face authentication method according to an embodiment.
6 is a view for explaining an example of a face registration process and a face authentication process according to an embodiment.
7A to 7D are diagrams for explaining examples of extracting registered features in a face registration process according to an embodiment.
8A and 8B are diagrams for explaining an example of generating a composite image in a face authentication process according to an embodiment.
9 is a diagram illustrating an example of synthesized images according to a partial face type according to an exemplary embodiment.
10 is a flowchart illustrating a face authentication method according to another embodiment.
11 is a diagram for describing an example of generating a synthesized image when an occlusion region exists in an input image, according to an exemplary embodiment.
12 is a diagram illustrating a configuration of a face authentication apparatus according to an embodiment.
13 is a diagram illustrating a configuration of a computing device according to an embodiment.
14 is a diagram illustrating a configuration of a learning apparatus according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component. In addition, when it is said that a certain element is "connected" to another element, it may be directly connected or connected to the other element, but it should be understood that another element may exist in between.

Expressions in the singular are generally to be construed to mean "one or more" unless otherwise specified. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1A is a diagram for describing face authentication according to an exemplary embodiment.

Face authentication is an authentication method for determining whether a corresponding user is a valid user based on face information of a user who has attempted authentication, and may be used to authenticate a valid user in user login, payment service, or access control. Referring to FIG. 1A , a face authentication device, which is a device for performing face authentication, may be included in the computing device 120 to operate. Computing device 120 may be, for example, a smartphone, a wearable device, a tablet computer, a netbook, a laptop, a desktop, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, or a vehicle ignition. It may be a device.

The computing device 120 may perform an authentication process for the user 110 who wants to access the computing device 120 through face authentication. For example, when the user 110 attempts user authentication with the computing device 120 to unlock the computing device 120 , the computing device 120 uses an image acquisition device such as the camera 130 . to obtain a face image of the user 110 , and analyze the obtained face image to determine whether to unlock the computing device 100 . In an embodiment, a face image captured by the user 110 through the camera 130 for face authentication may be provided through the screen of the computing device 120 . The corresponding face image functions as a preview image. The user 110 may adjust the shape of the face to be input for face authentication by referring to the preview image.

In an embodiment, the computing device 120 may detect the facial region 140 from the acquired facial image and extract features from the facial region 140 using a feature extractor. The computing device 120 may determine whether user authentication succeeds based on a comparison result between the extracted feature and the registered feature for a valid user. The feature extractor is a model that outputs feature information based on input information, and may be, for example, a neural network model trained in advance by training data. The neural network model is a model that models the characteristics of human biological nerve cells by mathematical expressions. When it is determined that user authentication is successful, the user 110 may successfully unlock the lock mode of the computing device 120 . Conversely, when it is determined that user authentication has failed, the computing device 120 continues to operate in the lock mode. A valid user may register his or her face in the computing device 120 in advance through the face registration process, and the computing device 120 provides information for identifying the valid user to a storage device or cloud storage medium. can be saved For example, a face image of a valid user or a feature extracted from the face image may be stored as registration information of the valid user.

In order to attempt face authentication, the user 110 photographs his or her face through the camera 130 . In this case, when a face is photographed out of the field of view (FoV) of the camera 130 , a partial face may be photographed instead of a full face. As shown in FIG. 1B , the user 110 may attempt face authentication while holding the computing device 120 at an angle. In this case, the view of the camera 130 does not include the entire face, so a partial face is captured can be Unlike in FIG. 1A , when the preview image captured by the camera 130 does not appear on the screen 150 that can be viewed by the user in the face authentication process, there are relatively many cases in which a partial face is captured in the face authentication process. . The user 110 tends to want to photograph a full face rather than a partial face for face authentication. However, if a preview image is not provided to the user 110 , the user 110 determines whether the entire face is photographed or a partial face. This is because it is impossible to know whether this is being photographed, and accordingly, it is difficult for the user 110 to adjust so that the entire face is photographed.

2A to 2D show examples of images in which a partial face is captured. In each of the images of FIGS. 2A to 2D , a partial face in which a partial facial part is omitted is photographed. Since a partial face may contain fewer features for identifying a user than a full face, in general, when face authentication is performed based on a partial face, the accuracy of face authentication may be lower than when face authentication is performed based on a full face. there is. However, since it is not always expected that the entire face is input for face authentication, it may be required to increase the accuracy of face authentication even if a partial face is input. In particular, as in the embodiment of FIG. 1B , when a preview image is not provided in the face authentication process, a partial face is often photographed for face authentication. Increasing the accuracy may be desired. Embodiments to be described below provide a solution that enables face authentication even when a preview image for photographing a face is not provided, and more accurately performs face authentication even when face authentication is performed based on a partial face.

3 to 5 are flowcharts illustrating a face authentication method according to an embodiment. In cases where certain embodiments may be implemented otherwise, certain steps of functions or operations may be performed differently from the flowchart. For example, two consecutive steps in FIGS. 3 to 5 may be performed substantially simultaneously, or the order of the steps may be reversed depending on a related function or operation.

Referring to FIG. 3 , in step 310 , the face authentication apparatus receives an input image. The input image is an image input to the face authentication device and is an image to be subjected to face authentication. The input image may be acquired by, for example, an image acquisition device such as a digital still camera or a video camera. Although not shown in the drawing, according to an embodiment, the face authentication apparatus may perform pre-processing on the received input image. The preprocessing process may include one or more processes of processing the input image in a form more suitable for face authentication. For example, the preprocessing process includes a process of removing noise included in the input image, a process of increasing the contrast of the input image, a deblurring process of removing blur included in the input image, It may include a process of removing a background region, a warping process of correcting distortion included in the input image, and a process of binarizing the input image.

In step 320, the face authentication apparatus detects a face region from the input image. For example, the face authentication apparatus may detect a face region from an input image using a Haar-based cascade adaboost classifier. However, the scope of the embodiment is not limited thereto, and the face authentication apparatus may detect a face region from an input image using various face region detection techniques. As an example, the face image of the face region extracted from the input image may have the form of a patch image.

In operation 330 , the face authentication apparatus may selectively normalize the face image of the face region. In an embodiment, the face authentication apparatus may detect facial landmarks in the detected face region, and normalize the input image based on the detected landmarks. Normalization is, for example, an image cropping process of extracting a face image from an input image, a process of matching the positions of landmarks in the face image to a predefined reference position, and a process of adjusting the size of the face image and the like.

The face authentication apparatus uses, for example, a landmark detection technique based on an Active Contour Model (ACM), an Active Shape Model (ASM), an Active Appearance model (AAM), a Supervised Descent Method (SDM), or a neural network model, such as a facial region can detect facial landmarks. The landmarks of the face are feature points for a major facial region, and are, for example, feature points for identifying eyebrows, eyes, nose, lips, chin, ears, or contours of a face. The face authentication apparatus may match the positions of the landmarks to the reference positions by performing affine transformation on the face region based on the positions of the detected landmarks. Here, the affine transformation serves to correspond the vector space indicated by the positions of the landmarks to another vector space.

When the normalization process of step 330 is performed, the face image in steps 340 to 370 below may be replaced with a face image on which the normalization process has been performed.

In step 340, the face authentication apparatus determines whether the input face displayed in the face image is a partial face. In an embodiment, the face authentication apparatus may determine whether the input face is a partial face based on a detection result of the face region (eg, a position of the detected face region). The face authentication apparatus may determine that the input face is a partial face, for example, when the detected face region includes a portion of the boundary of the input image or when the detected face region deviates from the input image. When the detected face region does not include a boundary of the input image or the detected face region does not deviate from the input image and is included in the input image, the face authentication apparatus may determine that the input face is the entire face.

In an embodiment, the face authentication device determines the type of the partial face based on whether the detected face region includes any of the boundaries of the input image or whether the detected face region deviates from the input image. can decide For example, the face authentication apparatus determines the partial face type as the first type when the portion deviating from the input image is the upper side, and distinguishes the partial face type from the first type when the portion deviating from the input image is the lower side It can be determined as the second type to be According to the type of the partial face, the synthesis form of the synthesized image to be described below may vary.

In another embodiment, the determination of whether the face is a partial face may be based on a detection result of landmarks. For example, the face authentication apparatus detects facial landmarks in the detected face region, and when landmarks corresponding to one or more of predefined facial part regions are not detected, It may be determined that the displayed input face is a partial face. Conversely, if landmarks for predefined face region regions are all detected from the face region displayed in the face image, the face authentication apparatus may determine that the input face is a full face rather than a partial face. When it is determined that the input face is a partial face, the face authentication apparatus may also determine the type of the partial face based on the detected landmarks. For example, the face authentication apparatus determines the partial face type as the first type when landmarks corresponding to the eyebrows and eyes are not detected, and when landmarks corresponding to the chin and lips are not detected, the partial face The type of may be determined as the second type.

In the above embodiments, it is assumed that the types defined for the partial face are the first type and the second type. However, depending on the embodiment, only one partial face type or three or more types may exist.

When it is determined in step 340 that the input face is a partial face, the face authentication apparatus generates a composite image by synthesizing the reference image and the face image in response to the determination in step 350 . In an embodiment, the face authentication apparatus generates a synthesized image by synthesizing image information of a reference image and image information of a face image based on the detected face region. Here, the image information of the reference image used to generate the synthesized image is information that the partial face does not include. In other words, in the face region corresponding to the entire face, the face authentication apparatus obtains image information corresponding to the remaining face region excluding the face region detected in step 320 from the reference image. In another embodiment, the face authentication apparatus may generate a synthesized image by synthesizing image information of a face image and image information of a corresponding region corresponding to a facial region in which landmarks are not detected in the reference image. Accordingly, the composite image includes image information of each of the face image and the reference image, and image information about a face region that is photographed out of the field of view of the camera and does not appear in the input image may be filled with image information of the reference image. In the above embodiments, the image information indicates a pixel value of a pixel included in an image.

In an embodiment, a region of a reference image to be used for synthesis may be determined according to a face region that does not appear in the input face. The face authentication apparatus may determine the type of the partial face based on the detection result of the landmarks, and may determine the area of the reference image to be used for synthesis according to the determined type. For example, when the type of the partial face is determined as the first type, the face authentication apparatus synthesizes image information of the face image and image information of the first corresponding region corresponding to the determination of the first type in the reference image to generate a synthesized image. can create As another example, when the partial face type is determined as the second type, the face authentication apparatus generates a composite image by synthesizing image information of the face image and image information of the second corresponding region corresponding to the second type determination in the reference image. can do. Here, the first correspondence area and the second correspondence area may be different from each other.

The reference image is a predefined image used to generate a composite image. In the face registration process, the same reference image is used to generate a composite image. In an embodiment, the reference image may be determined based on training images used to train the feature extractor. For example, the reference image may be an average image or an average value image of training images. The average image may be generated by averaging pixel values at positions corresponding to each other among the training images. The average value image may be generated by averaging pixel values of all pixels included in the training images and allocating the average pixel value to all pixels. In the average image, the shape may appear because pixel values may vary depending on the position of each pixel, but in the average image, since all pixels have the same pixel value, the shape does not appear. However, the example of the reference image is not limited to the above embodiment, and the shape of the reference image may vary. For example, the reference image may be a monochromatic color image regardless of the training image.

In step 360, the face authentication apparatus determines whether authentication is successful based on the synthesized image. Step 360 is described below with reference to FIG. 4 . Referring to FIG. 4 , in step 410 , the face authentication apparatus generates a difference image between a synthesized image and a reference image. The difference image may include pixel value difference information at positions corresponding to each other between the synthesized image and the reference image. For example, the face authentication apparatus may generate a difference image by subtracting a pixel value of a corresponding position of the reference image from each pixel value of the synthesized image.

In step 420, the face authentication apparatus extracts features for the input face using a feature extractor. Image information of the difference image may be input to the feature extractor, and the feature extractor may output a feature vector corresponding to image information of the difference image. In one example, the feature extractor may be a neural network model including output layers corresponding to each of the types of partial faces. In one example, the feature vector output from the output layer that is the top layer of the feature extractor or the feature vector determined based on the value extracted from the hidden layer of the feature extractor and the value extracted from the output layer is the feature for the input face can be extracted as The feature extractor may be learned through a learning process, which will be described in detail with reference to FIG. 14 .

In step 430 , the face authentication apparatus determines whether authentication succeeds based on a comparison result between the feature extracted by the feature extractor and the registered feature. The registration characteristic represents a characteristic of a valid user registered in the face registration process. In the face registration process, registered features corresponding to the entire face and registered features corresponding to predefined types of partial faces may be registered. In an embodiment, the face authentication apparatus may compare a registered feature corresponding to a partial face type among a plurality of pre-registered registered features and an extracted feature, and determine whether or not authentication of the input face is successful based on the comparison result. there is.

In an example, the face authentication apparatus may determine a degree of similarity between the extracted feature and the registered feature, and determine whether the user appearing in the input face is a valid user based on the similarity. The greater the difference between the extracted feature and the registered feature, the smaller the similarity, and the smaller the difference, the greater the similarity. The face authentication apparatus may determine that authentication is successful when the similarity is greater than a threshold value, and determine that authentication has failed when the similarity is less than or equal to the threshold.

Referring back to FIG. 3 , if it is determined in step 340 that the input face is not a partial face, in step 370 , the face authentication apparatus determines whether authentication is successful based on the face image. Non-partial face means that the input face is a full face. Step 370 is described below with reference to FIG. 5 . Referring to FIG. 5 , in step 510 , the face authentication apparatus generates a difference image between a face image and a reference image. The difference image may include pixel value difference information at positions corresponding to each other between the face image and the reference image.

In step 520, the face authentication apparatus extracts features for the input face using a feature extractor. Image information of the difference image generated in step 510 may be input to the feature extractor, and the feature extractor may output a feature vector corresponding to image information of the difference image. A feature of the input face may be extracted from the difference image in a manner similar to that in step 420 of FIG. 4 .

In step 530, the face authentication apparatus determines whether authentication is successful based on a comparison result between the extracted feature and the registered feature. The face authentication apparatus may compare the extracted features with the registered features corresponding to the entire face among the registered features, and determine whether or not authentication of the input face is successful based on the comparison result. In a similar manner to step 430 of FIG. 4 , a degree of similarity between the extracted feature and the registered feature may be determined, and whether authentication succeeds may be determined based on the degree of similarity.

6 is a view for explaining an example of a face registration process and a face authentication process according to an embodiment.

Referring to FIG. 6 , in the face registration process 610 , a feature is extracted from a registered image 615 , and the extracted feature is stored as a registered feature. In an embodiment, the registration image 615 is obtained by the user taking a picture of his/her own face using a camera for face registration. In the face registration process 610 , a registered image 615 in which the entire face is captured may be used.

According to a process similar to steps 310 to 330 of FIG. 3 , a normalized registered image 625 corresponding to the entire face may be obtained from the registered image 615 . Except that the 'input image' is changed to the 'registered image 615' in the process of steps 310 to 330, the processing process of steps 310 to 330 may be applied here as it is. .

In addition, one or more synthesized images 630 are generated based on the normalized registration image 625 and the reference image 620 . According to an embodiment, only one synthesized image 630 may be generated, or synthesized images 630 corresponding to the types of each partial face may be generated. The synthesized image 630 may be a synthesized image in which a partial region of the registered image 625 is replaced with image information of a reference image according to each type of partial face. In one example, the reference image 620 may be an average image or an average value of training images used to learn the feature extractor. Features corresponding to the normalized registered image 625 and synthesized image 630 (each synthesized image when there are multiple synthesized images) are extracted 635 using a feature extractor, and each extracted feature is stored as registered features. (640) becomes. In this case, the feature corresponding to the registered image 625 and the feature corresponding to the synthesized image 630 may be stored separately from each other.

In the face authentication process 650 , face authentication is performed based on a comparison result between the feature extracted from the input image 660 and the feature registered in the face registration process 610 . In an embodiment, when the face authentication device receives the input image 660 that is the target of face authentication, the face authentication device detects a face region from the input image 660 and determines whether the input face displayed in the face region is a partial face. can decide In FIG. 6 , since the mouth area is omitted from the face area shown in the input image, it is determined that the input face is a partial face. In this case, the face authentication apparatus generates the synthesized image 670 based on the detected face image of the face region and the reference image 620 used in the face registration process 610 . The process of generating the synthesized image 670 from the input image 660 may refer to steps 310 to 350 of FIG. 3 .

The face authentication apparatus extracts ( 680 ) features of the input face from the synthesized image 670 using the feature extractor used in the face registration process ( 610 ). Thereafter, the face authentication apparatus may compare the extracted feature with the registered feature, and determine whether authentication is successful based on the comparison result ( 685 ). In an embodiment, the registered feature to be compared with the extracted feature may be determined based on the shape of the input face displayed in the input image 660 . For example, when the mouth region is omitted as in the input image 660 , in the face registration process 610 , the first synthesized image (the mouth region is image information of the reference image 620 ) among the synthesized images 630 . A registered feature extracted based on the synthesized image replaced by ) is compared with a feature extracted from the input image 660 .

In another embodiment, when only one synthesized image 630 is generated in the face registration process 610 and a feature extracted from the synthesized image 630 is registered as a feature corresponding to a partial face, the input image 660 is If it is determined that the input face shown in is a partial face, a synthesized image 670 for face authentication may be generated in the same synthesized form as the above synthesized image 630 . For example, if the composite image 630 of the face registration process 610 is generated by replacing the lower part of the face (eg, the part including the mouth region) with image information of the reference image 620 , , the synthesized image 670 generated in the face authentication process 650 also has a synthesized form in which the lower part of the face is replaced with image information of the reference image 620 .

7A to 7D are diagrams for explaining examples of extracting registered features in a face registration process according to an embodiment.

Referring to FIG. 7A , registered features are extracted from a registered image 615 using a feature extractor 740 to register a face appearing in a registered image 615 in the face registration process. In this case, the normalized image 705 corresponding to the entire face from the registered image 615 , and synthesized images 710 and 715 based on the registered image 615 (or image 705 ) and the reference image 620 . , 720, 725, 730) may be generated. Each of the composite images 710 , 715 , 720 , 725 , and 730 may correspond to each type of partial face.

A difference image between the image 705 and the reference image 620 may be input to the feature extractor 740 , and the feature extractor 740 may output a first feature corresponding to the image 705 . In addition, difference images between the respective synthesized images 710 , 715 , 720 , 725 , and 730 and the reference image 620 are individually or simultaneously input to the feature extractor 740 , and the feature extractor 740 selects each synthesized image. Features corresponding to (710, 715, 720, 725, 730) may be output. For example, when a difference image between the synthesized image 710 and the reference image 620 is input to the feature extractor 740 , the second feature corresponding to the partial face type of the synthesized image 710 is obtained from the feature extractor 740 . can be output. The first feature is a feature corresponding to the entire face, and the second to sixth features are features corresponding to each type of the partial face.

In the case of the synthesized images 720 , 725 , and 730 , the left region of the face is synthesized with image information of the reference image 620 . In the present embodiment, features are not extracted from the synthesized images having a symmetrical composition form and the composition form of the above synthetic images 720 , 725 , and 730 . For example, in the synthesized image 720 , the left region of the face is synthesized with image information of the reference image 620 , and a synthesized image is generated in which the right region of the face is synthesized with image information of the reference image 620 , and the synthesis It does not extract features from the image. This is because the input face can be transformed into a partial face in which the left region of the face does not appear through image processing even if the input face, which is the target of face authentication, is a partial face in which the right region of the face does not appear. For example, the above modified partial face may be obtained by inverting the input face left and right. This is due to the fact that the shape of the face is almost symmetrical. By extracting features from the image of the left and right inverted partial faces, and comparing the extracted features with the registered features extracted from the synthesized image 720, feature comparison for partial faces of the same type becomes possible. Of course, according to an embodiment, it is also possible to further generate additional synthesized images having a synthesized form of the synthesized images 720 , 725 , and 730 and a symmetrical synthesized form, and to extract a feature for face registration from the generated synthesized images. possible.

According to another embodiment, as shown in FIG. 7B , the feature extractor may exist separately from each other according to each type of a full face and a partial face. For example, based on the first feature extractor 750 that extracts features corresponding to the entire face based on the image 705 and the synthesized images 710, 715, 720, 725, 730 for each type of partial face Feature extractors 752 , 754 , 756 , 758 , 760 for extracting corresponding features may exist individually.

According to another embodiment, the feature extractor may include a common feature extractor 762 and individual feature extractors 764 , 766 , 768 , 770 , 772 , and 774 as shown in FIG. 7C . The common feature extractor 762 and the feature extractors 764 , 766 , 768 , 770 , 772 , and 774 are connected to each other. The intermediate feature output from the common feature extractor 762 is transmitted to the feature extractors 764, 766, 768, 770, 772, 774, and from each of the feature extractors 764, 766, 768, 770, 772, 774 , features corresponding to each type of a full face or a partial face are output. For example, when a difference image between the image 705 and the reference image 620 is input to the common feature extractor 762 , an intermediate feature corresponding to the corresponding difference image is output from the common feature extractor 762 , and the output intermediate The features are passed to a first feature extractor 764 . The first feature extractor 764 outputs a first feature corresponding to the image 705 based on the intermediate feature received from the common feature extractor 762 .

As another example, when a difference image between the synthesized image 710 and the reference image 620 is input to the common feature extractor 762 , an intermediate feature corresponding to the corresponding difference image is output from the common feature extractor 762 , and the output intermediate The features are passed to a second feature extractor 766 . The second feature extractor 766 outputs a second feature corresponding to the synthesized image 710 based on the intermediate feature received from the common feature extractor 762 . In one embodiment, the common feature extractor 762 and the individual feature extractors 764, 766, 768, 770, 772, and 774 may be implemented as one neural network model. In this case, the common feature extractor 762 includes at least one of a convolutional layer and a fully connected layer, and each of the feature extractors 764 , 766 , 768 , 770 , 772 , and 774 , respectively. may include a fully connected layer.

The number of types of partial faces in FIGS. 7A to 7C is not limited to those shown in the drawings, and according to embodiments, there may be one or more types of partial faces. An embodiment in which there is one type of partial face is shown in FIG. 7D .

Referring to FIG. 7D , in the face registration process, features for registering a face appearing in a registration image 615 are extracted. A synthesized image 710 is generated from the registered image 615 based on the normalized image 705 corresponding to the entire face, the registered image 615 (or the image 705 ), and the reference image 620 . In the present embodiment, the lower part of the face in the synthesized image 710 is synthesized with image information of the reference image 620 . A difference image between the image 705 and the reference image 620 is input to the common feature extractor 780 , and an intermediate feature corresponding to the image 705 is output from the common feature extractor 780 . The output intermediate features are input to a first feature extractor 782 , and the first feature extractor 782 outputs a first feature corresponding to the entire face based on the input intermediate features. Also, a difference image between the synthesized image 710 and the reference image 620 is input to the common feature extractor 780 , and an intermediate feature corresponding to the synthesized image 710 is output from the common feature extractor 780 . The output intermediate feature is input to a second feature extractor 784 , and the second feature extractor 784 outputs a second feature corresponding to the partial face based on the input intermediate feature. Similar to the embodiment of FIG. 7C , the common feature extractor 780 and the individual feature extractors 782 and 784 may be implemented as one neural network model.

The structure of the feature extractor described in FIGS. 7A to 7D may also be used in the face authentication process. Here, the structure of the feature extractor may be implemented as one or more neural network models, and the feature extractor may be included in a computing device for performing face authentication.

If the feature extractor 740 shown in FIG. 7A is used to extract registered features in the face registration process, the face authentication process may be performed based on the feature extractor 740 . Alternatively, if the feature extractors 750, 752, 754, 756, 758, and 760 shown in FIG. 7B are used in the face registration process, the corresponding feature extractors 750, 752, 754, 756, 758, 760) can be used in the same way. When the input face displayed in the face image includes full face information, the image information for the face image is input to the first feature extractor 750 , and face authentication is performed based on the features extracted from the first feature extractor 750 . is carried out When the input face displayed in the face image includes partial face information, a feature extractor to be used among the second to sixth feature extractors 752 , 754 , 756 , 758 , 760 may be selected according to the type of the partial face. . Thereafter, image information of the face image is transmitted to the selected feature extractor, and face authentication is performed based on the features extracted from the feature extractor.

If the common feature extractor 780 and the first and second feature extractors 782 and 784 shown in FIG. 7D are used to extract the registered feature, the corresponding common feature extractor 780 and the first and second feature extractors are used. A face authentication process may be performed based on (782, 784). The common feature extractor 780 may receive a face image and output an intermediate feature, and may be connected to the first and second feature extractors 782 and 784 . The first feature extractor 782 extracts features corresponding to the entire face, and the second feature extractor 784 extracts features corresponding to the partial face. When the face image includes the entire face information of the user, the first feature extractor 782 may receive intermediate features from the common feature extractor 780 and extract features to be used for face authentication. When the face image includes partial face information of the user, the second feature extractor 784 may receive intermediate features from the common feature extractor 780 and extract features to be used for face authentication. The common feature extractor 780 may include, for example, one or more of a convolutional layer and a fully connected layer, and each of the first and second feature extractors 782 and 784 may include a fully connected layer. The second feature extractor 784 for extracting the feature corresponding to the partial face may be composed of a plurality of feature extractors 766, 768, 770, 772, and 774 as shown in the example of FIG. 7C . Here, each of the feature extractors 766 , 768 , 770 , 772 , and 774 may extract a feature to be used for face authentication with respect to different types of partial face images.

8A and 8B are diagrams for explaining an example of generating a composite image in a face authentication process according to an embodiment.

Referring to FIG. 8A , when an input image 810 is input to the face authentication apparatus, the face authentication apparatus detects a face region 815 from the input image 810 .

In an embodiment, the face authentication apparatus determines whether the face region 815 includes a portion of a boundary of the input image 810 or whether the face region 815 deviates from the input image 810 based on whether the input image 810 is outside the input image. It may be determined whether the input face shown in 810 is a partial face. In the present embodiment, the face region 815 includes the boundary of the input image 810 , and some position values among the position values for defining the detected face region 815 exist outside the input image 810 . Therefore, the input face is determined as a partial face. In another embodiment, the face authentication apparatus may detect landmarks 820 of the face within the face region 815 and determine whether the input face is a partial face based on the detection result of the landmarks 820 . In the present embodiment, since the landmark corresponding to the mouth region is not detected in the face region 815 , the face authentication apparatus may determine that the input face is a partial face.

The face authentication apparatus may normalize the input image 810 based on the positions of the detected landmarks 820 . The normalization process may include extracting a face region from the input image 810 and matching the positions of the landmarks 820 with predefined reference positions.

When it is determined that the input face displayed in the face image 840 is a partial face, the face authentication apparatus performs a process 830 of generating a synthesized image 860 using the reference image 850 . Here, the reference image 850 may be an average image of training images used to learn the feature extractor. In an embodiment, the face authentication apparatus determines the type of the partial face displayed in the face image 840 representing the detected face area based on the detected location of the face area 815 or the detection result of the landmarks 820 . can An area to be synthesized with the face image 840 in the reference image 850 may be determined according to the determined partial face type. The face authentication apparatus synthesizes the image information of the face image 840 with the image information of the corresponding region 855 corresponding to the face region 845 that does not appear in the face image 840 in the reference image 850 . (860). In this case, the boundary at which the synthesis is made does not necessarily have to be a straight line, and the shape of the boundary may be a curve or a polygon according to an embodiment. The synthesized image 860 includes an area 865 representing image information of the face image 840 and an area 870 representing image information of the reference image 850 .

After the synthesized image 860 is generated, a difference image between the synthesized image 860 and the reference image 850 is input to a feature extractor for feature extraction. When a difference image between the reference image 850 and the face image 840 showing a partial face, not the synthesized image 860 , is used for feature extraction, the region corresponding to the face region 845 in the difference image is the reference image The image information of 850 may act as noise, which may decrease the accuracy of face authentication. However, as in the present embodiment, when a difference image between the synthesized image 860 and the reference image 850 is used, image information of a region corresponding to the face region 845 in the difference image is combined with the synthesized image 860 and the reference image. It is canceled by the difference process between the images 850 so that no noise is generated. Accordingly, the face authentication apparatus can maintain high accuracy of face authentication even when a partial face is input.

In an embodiment, the reference image 850 may be an average image of training images as shown in FIG. 8A , or an average value image of training images or an image in which all pixels have the same pixel value as shown in FIG. 8B . . Referring to FIG. 8B , an example in which a synthesized image 890 is generated based on a reference image 880 that is an average value image is illustrated. The reference image 880 may be an image generated by allocating an average value obtained by averaging pixel values of all pixels of the training images to all pixels. The face authentication apparatus synthesizes the image information of the face image 840 with the image information of the corresponding region 885 corresponding to the face region 845 appearing in the face image 840 in the reference image 880, thereby creating a synthesized image ( 890) is created.

9 is a diagram illustrating an example of synthesized images according to a partial face type according to an exemplary embodiment.

Referring to FIG. 9 , a synthesized image may be generated in a different form according to the type of a partial face displayed in the input image. As in the input images 912 , 914 , 916 , 918 , and 920 , the shape of the partial face may vary. For example, the mouth region may not appear on the input face as in the input image 912 , or the face region above the eyebrows may not appear on the input face as in the input image 914 . The face authentication apparatus distinguishes the partial face type, and synthesizes the reference image 910 and the input images 912 , 914 , 916 , 918 , 920 based on the partial face type to obtain synthesized images 922 , 922 , 924, 926, 928, 930). A region of the reference image 910 to be reflected in each of the composite images 922 , 924 , 926 , 928 , and 930 may be determined according to the type of the partial face.

However, the synthesis form of the synthesized image corresponding to the partial face in face authentication is not limited to the embodiment of FIG. 9 , and the synthesis form of the synthesized image may be one or plural. For example, if it is assumed that the synthesis form of the synthesized image exists only by synthesizing the lower part of the face with image information of the reference image as in the synthesized image 922 , when it is determined that the input face displayed in the input image is a partial face, the face The authentication device generates a synthesized image in a synthesized form in which the lower part of the input face displayed in the input image is synthesized with image information of the reference image regardless of the shape of the partial face.

10 is a flowchart illustrating a face authentication method according to another embodiment.

In cases where certain embodiments may be implemented otherwise, certain steps of functions or operations may be performed differently from the flowchart. For example, two consecutive steps in FIG. 10 may be performed substantially simultaneously, or the order of the steps may be reversed according to a related function or operation.

According to an embodiment, the face authentication apparatus may further consider an occlusion region appearing in the input image in the face authentication process. The occlusion area is an area covered by the input face due to, for example, glasses, sunglasses, a mask, or a hair band. Such an occluded area prevents extracting of the user's unique features from the input face shown in the input image, and is a factor that reduces the accuracy of face authentication. The face authentication apparatus may determine whether an occluded area exists in the input image, and if there is an occluded area, replace image information of the occluded area with image information of the reference image and perform face authentication. A face authentication method for the present embodiment will be described below.

Referring to FIG. 10 , in step 340 , the face authentication apparatus determines whether an input face displayed in a face image is a partial face. The contents from step 310 to step 340 of FIG. 3 are included here as they are. When it is determined that the input face is a partial face, in step 1010 , the face authentication apparatus determines whether an occlusion area exists in the input face displayed in the face image.

When it is determined that the occluded area exists, in step 1020, the face authentication apparatus performs image information of the first corresponding area corresponding to the partial face type in the reference image, image information of the second corresponding area corresponding to the occluded area, and the face A synthesized image is generated by synthesizing the image information of the image. In one example, the face authentication apparatus combines image information of a first corresponding region of a reference image with a face image and replaces image information of an occluded region detected in the face image with image information of a second corresponding region of the reference image. A composite image can be generated by According to an embodiment, the first correspondence area and the second correspondence area may overlap each other.

When it is determined in step 1010 that there is no occluded area, in step 1030, the face authentication apparatus synthesizes image information of the first corresponding area corresponding to the type of partial face in the reference image and image information of the face image. to create a composite image. This process is the same as the process of generating the synthesized image described in the embodiments such as step 350 of FIG. 3 and FIGS. 8A, 8B, and 9 .

If it is determined in step 340 that the input face is not a partial face, in step 1040 the face authentication apparatus determines whether an occlusion area exists in the input face displayed in the face image. When it is determined that the occluded region exists, the face authentication apparatus generates a synthesized image by synthesizing the image information of the face image and image information of the second corresponding region corresponding to the occluded region in the reference image in step 1050 . The face authentication apparatus may generate a composite image by replacing image information of an occluded region in the face image with image information of a second corresponding region of the reference image.

When the synthesized image is generated in step 1020 , 1030 , or 1050 , in operation 1060 , the face authentication apparatus determines whether authentication is successful based on the synthesized image. This process is the same as that of step 360 of FIG. 3 .

If it is determined in step 1040 that the occluded area does not exist, in step 1070, the face authentication apparatus determines whether authentication succeeds based on the face image. In this case, the entire face appears without the presence of an occluded region in the input image, and step 1070 is the same as step 370 of FIG. 3 .

When the face authentication method is implemented as described above, it is possible to prevent a decrease in the recognition rate according to the presence or absence of an occluded area. In the present embodiment, the occlusion area is detected in the registration image even in the face registration process, and when there is an occlusion area, a composite image is generated by replacing the occlusion area with image information of the reference image, and then the registered features are extracted from the synthesized image. do. According to an embodiment, even if an occlusion region does not actually exist in the registered image, it is assumed that various types of occlusion regions exist in the registered image, and synthesized images corresponding to the types of each occlusion region are generated, and then synthesized images are obtained from the synthesized images. It is also possible to extract registered features.

11 is a diagram for describing an example of generating a synthesized image when an occlusion region exists in an input image, according to an exemplary embodiment.

Referring to FIG. 11 , when the input face shown in the face image 1110 is a partial face, the face authentication apparatus first corresponds to a reference image 1120 corresponding to a face region 1114 that is not shown in the face image 1110 . Image information of the area 1124 may be added to the face image 1110 . Meanwhile, an occluded area of the glasses 1112 exists in the face image 1110 . When the face authentication apparatus detects the glasses 1112 in the face image 1110, the face authentication apparatus may replace it with image information of the second corresponding region 1122 of the reference image 1120 corresponding to the region in which the glasses 1112 appear. . The synthesized image 1130 generated through this process includes image information 1135 of the face image 1110 , image information 1145 of a first corresponding region of the reference image 1120 , and a second correspondence of the reference image 1120 . Consists of image information 1140 of the area.

12 is a diagram illustrating a configuration of a face authentication apparatus according to an embodiment.

Referring to FIG. 12 , the face authentication apparatus 1200 receives an input image for face authentication. The face authentication apparatus 1200 may determine whether authentication succeeds based on a comparison result between the feature extracted from the input image and the registered feature previously stored in the database 1230 . A valid user may pre-register features or images of his/her face through the face registration process, and the registered information may be stored in the database 1230 .

The face authentication apparatus 1200 may perform one or more operations described or illustrated herein in relation to the face authentication method, and may provide a result of the face authentication to the user. The face authentication device 1200 may output the face authentication result in the form of voice, vibration, text, picture, or video, but the scope of the embodiment is not limited thereto, and the face authentication device 1200 may be provided in various forms. to print the authentication result.

The face authentication apparatus 1200 may include one or more processors 1210 and a memory 1220 . The memory 1220 is connected to the processor 1210 and may store instructions executable by the processor 1210 , data to be calculated by the processor 1210 , or data processed by the processor 1210 . Memory 1220 may include non-transitory computer-readable media, such as high-speed random access memory and/or non-volatile computer-readable storage media (eg, one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices). may include

The processor 1210 may execute instructions for executing one or more operations described with reference to FIGS. 1 to 11 . For example, the processor 1210 detects a face region from the input image, and determines whether the input face displayed in the detected face region is a partial face or a full face. For example, the processor 1210 may determine that the input face is a partial face when the detected face region includes a portion of the boundary of the input image or when the detected face region deviates from the input image. When the detected face region does not include the boundary of the input image or the detected face region does not deviate from the input image and is included in the input image, the processor 1210 may determine that the input face is the entire face. As another example, when landmarks corresponding to one or more of the predefined face region regions are not detected, the processor 1210 may determine that the input face is a partial face. Conversely, when landmarks corresponding to all of the predefined face region regions are detected, the processor 1210 may determine that the input face is the entire face.

When it is determined that the input face is a partial face, the processor 1210 generates a composite image by synthesizing the reference image and the face image. In an example, the processor 1210 may generate a synthesized image by synthesizing image information of a corresponding region of a reference image corresponding to a facial region in which the above landmarks are not detected and image information of the face image. In another embodiment, the processor 1210 determines whether an occlusion region exists in the input face displayed in the face image, and when it is determined that the occlusion region exists, image information of a corresponding region corresponding to the occlusion region in the reference image A synthesized image can be generated by synthesizing the image information of the and face image. The processor 1210 extracts a feature of the input face from the synthesized image using the feature extractor, compares the extracted feature with the registered feature, and determines whether authentication succeeds based on the comparison result. In response to determining that the input face is not a partial face (ie, determining that the input face is a full face), the processor 1210 extracts features for the input face from the face image using a feature extractor, and registers the extracted features and The features are compared, and based on the comparison result, it is determined whether or not authentication succeeds.

13 is a diagram illustrating a configuration of a computing device according to an embodiment.

Referring to FIG. 13 , the computing device 1300 may perform a face authentication process by acquiring an image showing the user's face and comparing features extracted from the acquired image with previously stored registered features. The computing device 1300 may correspond to the computing device 120 of FIG. 1 , and may functionally include the functions of the face authentication device 1200 of FIG. 12 .

The computing device 1300 may include a processor 1310 , a memory 1320 , a camera 1330 , a storage device 1340 , an input device 1350 , an output device 1360 , and a network interface 1370 . The processor 1310 , the memory 1320 , the camera 1330 , the storage device 1340 , the input device 1350 , the output device 1360 , and the network interface 1370 may communicate with each other via the communication bus 1380 . there is.

The camera 1330 may capture still images, video images, or both. The camera 1330 may acquire a face image of the user by capturing a face region input by the user to attempt face authentication.

The processor 1310 executes functions and instructions for execution in the computing device 1300 . For example, the processor 1310 may process instructions stored in the memory 1320 or the storage device 1340 . The processor 1310 may perform one or more operations described above with reference to FIGS. 1 to 12 . For example, the processor 1310 may perform face authentication by comparing a face image acquired by the camera 1330 with a registered face image. The processor 1310 may perform face authentication even if the acquired face image does not include the entire face information of the user. When the acquired face image includes partial face information of the user, the processor 1310 generates a composite image between the face image and a predefined reference image based on the type of partial face information, and based on the generated composite image to perform face authentication.

The memory 1320 stores information for a face authentication process. Memory 1320 may include a computer-readable storage medium or a computer-readable storage device. For example, the memory 1320 may include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), or other types of nonvolatile memory known in the art. The memory 1320 may store instructions for execution by the processor 1310 , and may store related information while the software or application is executed by the computing device 1300 .

The storage device 1340 includes a computer-readable storage medium or a computer-readable storage device. The storage device 1340 may store a database including registered features or registered images. According to an embodiment, the storage device 1340 may store a larger amount of information than the memory 1320 and may store the information for a long period of time. For example, the storage device 1340 may include a magnetic hard disk, an optical disk, a flash memory, an electrically programmable memory (EPROM), a floppy disk, or other form of non-volatile memory known in the art.

The input device 1350 may receive an input from the user through tactile, video, audio, or touch input. For example, input device 1350 may include a keyboard, mouse, touch screen, microphone, or any other device capable of detecting input from a user and communicating the detected input to computing device 1300 .

The output device 1360 may provide an output of the computing device 1300 to the user through a visual, auditory, or tactile channel. For example, the output device 1360 may visualize information related to face authentication and provide it to the user. The output device 1360 may include, for example, a liquid crystal display, a light emitting diode (LED) display, a touch screen, a speaker, a vibration generating device, or any other device capable of providing output to a user. In an embodiment, the computing device 1300 may display a face image (preview image) acquired by the camera 1330 on a display screen that a user can see while face authentication is performed, and display the face image. may not be displayed.

The network interface 1370 may communicate with an external device through a wired or wireless network. For example, the network interface 1370 may include an Ethernet (Ethernet) card, an optical transceiver, a radio frequency transceiver, or any other network interface card capable of sending and receiving information. The network interface 1370 may wirelessly communicate with an external device using a communication method such as Bluetooth, Wi-Fi, 3G, or 4G.

14 is a diagram illustrating a configuration of a learning apparatus according to an embodiment.

Referring to FIG. 14 , a learning apparatus 1400 is an apparatus for learning a feature extractor 1430 used in face authentication. In one example, the feature extractor 1430 is a neural network model that outputs a result value corresponding to an input value, and may be trained based on a training image and a reference image. Here, the reference image is the same as the reference image described in the face registration process and face authentication process above.

The learning apparatus 1400 may include one or more processors 1410 and a memory 1420 . The memory 1420 is connected to the processor 1410 and may store instructions executable by the processor 1410 , data to be calculated by the processor 1410 , or data processed by the processor 1410 .

When the learning image is input to the learning apparatus 1400 , the processor 1410 may detect a face region from the learning image and determine whether a face displayed in the detected face region is a partial face or a full face. When it is determined that the face shown in the training image is a partial face, the processor 1410 generates a composite image by synthesizing the image information of the reference image with the image information of the training image based on the type of the partial face, and a feature extractor 1430 is used to extract features from the synthesized image. Contents described in the face authentication process described in this specification may be directly applied to these processes. In one example, the processor 1410 calculates a loss due to a difference between the feature extracted by the feature extractor 1430 and the intended feature, and parameters configuring the feature extractor 1430 such that the calculated loss is reduced. By adjusting the values, the feature extractor 1430 can be trained.

The learning apparatus 1400 may repeatedly perform the above process for each of a large number of learning images to gradually adjust the parameters of the feature extractor 1430 in a desirable direction.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Claims (33)

detecting a face region in the input image;
determining whether the input face displayed in the face region is a partial face;
generating a synthesized image by synthesizing a reference image and a face image of the face region in response to determining that the input face is a partial face; and
Determining whether authentication is successful based on the composite image
including,
The reference image is
An image determined based on learning images used to train a feature extractor used to extract features from the synthesized image,
Face authentication method. According to claim 1,
Determining whether the input face is a partial face comprises:
and determining whether the input face is a partial face based on a position of the face region detected in the input image. 3. The method of claim 2,
Determining whether the input face is a partial face comprises:
and determining that the input face is a partial face when the detected face region includes a part of a boundary of the input image. 3. The method of claim 2,
The step of generating the composite image includes:
generating the synthesized image by synthesizing image information of the reference image and image information of the face image based on the detected face region;
The image information of the reference image is image information of a region corresponding to the rest of the face region excluding the detected face region in the face region corresponding to the entire face. According to claim 1,
Determining whether the input face is a partial face comprises:
detecting landmarks of a face in the face region; and
determining that the input face is a partial face when, in the detected face region, landmarks corresponding to at least one of predefined facial part regions are not detected;
A face authentication method comprising a. 6. The method of claim 5,
The step of generating the composite image includes:
and generating the synthesized image by synthesizing image information of a corresponding region of the reference image corresponding to a facial region in which the landmarks are not detected and image information of the face image. 7. The method of claim 6,
The method of claim 1, wherein the location and shape of the corresponding area are determined based on the area of the face in which the landmarks are not detected. According to claim 1,
The step of determining whether the authentication succeeds,
extracting features for the input face using a feature extractor; and
Determining whether authentication succeeds based on a comparison result between the extracted feature and a registered feature corresponding to the partial face type among the registered features
A face authentication method comprising a. 9. The method of claim 8,
The type of the partial face is,
The face authentication method is determined based on a detection result of landmarks corresponding to at least one of the predefined facial region regions. 9. The method of claim 8,
In the feature extractor,
A face authentication method in which image information of a difference image between the synthesized image and the reference image is input. 9. The method of claim 8,
The feature extractor is
a neural network model including output layers corresponding to each of the types of partial faces. 9. The method of claim 8,
The registration features are:
and a registration feature corresponding to each of the predefined partial face types. delete According to claim 1,
in response to determining that the input face is not a partial face, determining whether authentication is successful based on the face image
A face authentication method further comprising a. 15. The method of claim 14,
The step of determining whether authentication succeeds based on the face image,
extracting a feature of the input face based on a difference image between the face image and the reference image; and
determining whether authentication succeeds or not based on a comparison result between the extracted features and registered features corresponding to the entire face among the registered features
Including face authentication methods. According to claim 1,
determining whether an occlusion region exists in the input face displayed in the face image; and
generating a synthesized image by synthesizing image information of a corresponding region corresponding to the occlusion region in the reference image and image information of the face image in response to determining that the occlusion region exists;
A face authentication method further comprising a. According to claim 1,
detecting landmarks of a face in the face region; and
adjusting the face region displayed in the input image based on the positions of the detected landmarks
A face authentication method further comprising a. A computer-readable storage medium storing instructions for executing the method of any one of claims 1 to 12 and 14 to 17. including a processor;
The processor is
Detects a face region in the input image,
determining whether the input face displayed in the face region is a partial face;
In response to determining that the input face is a partial face, a reference image and a face image of the face region are synthesized to generate a synthesized image;
Determining whether or not authentication succeeds based on the synthesized image,
The reference image is
An image determined based on learning images used to train a feature extractor used to extract features from the synthesized image,
face authentication device. 20. The method of claim 19,
The processor is
and determining that the input face is a partial face when the detected face region includes a part of a boundary of the input image. 20. The method of claim 19,
The processor is
generating the synthesized image by synthesizing image information of the reference image and image information of the face image based on the detected face region;
The image information of the reference image is image information of a region corresponding to the rest of the face region excluding the detected face region from the face region corresponding to the entire face. 20. The method of claim 19,
The processor is
detecting landmarks of the face in the face region;
and determining that the input face is a partial face when landmarks corresponding to at least one of predefined face region regions are not detected in the detected face region. 23. The method of claim 22,
The processor is
and generating the synthesized image by synthesizing image information of a corresponding region of the reference image corresponding to a facial region in which the landmarks are not detected and image information of the face image. 20. The method of claim 19,
The processor is
extracting features for the input face using a feature extractor,
and determining whether or not authentication succeeds based on a comparison result between a registered feature corresponding to the type of the partial face among registered features and the extracted feature. 20. The method of claim 19,
The processor is
determining whether an occlusion area exists in the input face displayed in the face image;
and generating a synthesized image by synthesizing image information of a corresponding region corresponding to the occlusion region and image information of the face image in the reference image in response to determining that the occlusion region exists. a common feature extractor that receives a face image and outputs intermediate features;
a first feature extractor connected to the common feature extractor and configured to receive intermediate features corresponding to the entire face from the common feature extractor and extract features to be used for face authentication when the face image includes full face information of the user; and
It is connected to the common feature extractor, and when the face image includes partial face information of a user, an intermediate feature corresponding to a synthesized image generated by synthesizing a reference image and a face image of the face region is input from the common feature extractor A second feature extractor that receives and extracts features to be used for face authentication
including,
The reference image is
An image determined based on learning images used to train the second feature extractor used to extract features from the synthesized image,
A device with a neural network model. 27. The method of claim 26,
The common feature extractor is a device including a neural network model, including a convolutional layer 27. The method of claim 26,
The common feature extractor includes a neural network model including a fully-connected layer. 27. The method of claim 26,
The apparatus including a neural network model, wherein the first feature extractor and the second feature extractor include a fully connected layer. 27. The method of claim 26,
The apparatus comprises a plurality of second feature extractors;
The apparatus including a neural network model, wherein each of the second feature extractors extracts features to be used for face authentication with respect to different types of partial face images. A computing device comprising:
a camera for acquiring a user's face image; and
a display that a user can see while acquiring the face image;
The user who wants to access the computing device in a locked state to generate a composite image by combining image information of the acquired face image and image information corresponding to parts missing from the acquired face image in a predefined reference image A processor that performs face authentication based on the synthesized image for
including,
the computing device in the locked state does not display the acquired face image to the user on the display while the face authentication is performed using a neural network for face authentication;
In response to determining that authentication between the composite image and the registered face image of a valid user in the face authentication is successful, the computing device releases the lock state to allow the user to access the computing device,
The synthesized image includes image information of the acquired face image for a detected partial face among all faces and includes image information of the reference image for the missing parts,
computing device. 32. The method of claim 31,
The processor is
and performing the face authentication even if the acquired face image does not include the entire face information of the user. delete

Images