JP2023025914A - Face authentication device, face authentication method, and computer program - Google Patents

Abstract

To reduce authentication errors when wearing a mask or the like on which another person's face is printed.SOLUTION: A face authentication device includes: face image input means for acquiring an input face image; registered face image holding means for holding a plurality of registered face images in advance; face coexistence determination means for determining whether face areas of different persons coexist in the input face image; and control means for performing a predetermined notification when the face coexistence determination means determines that the face areas of different persons coexist in the input face image.SELECTED DRAWING: Figure 4

Description

The present invention relates to a face authentication device, a face authentication method, a computer program, and the like.

Conventionally, there is a technology for face recognition of a person who wears a face shield such as a mask. For example, in Japanese Patent Laid-Open No. 2004-100000, an object that shields the face, such as a mask, is compared with a standard image representing the face to estimate the area where the object that shields the face exists, and the area with the object is excluded. Face authentication is performed using the face image obtained. Also, Non-Patent Document 1 describes a general face detection technique for identifying a face area.

Japanese Patent No. 6630999

J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, "Retina Face: Single-Shot Multi-Level Face Localization in the Wild," in Conference on Computer Vision and Pattern Recognition (CVPR) 2020, pp. 5203-5212.

However, in Japanese Patent Application Laid-Open No. 2002-200010, the area where the mask or the like exists is estimated by comparison with an image representing a standard face. Therefore, when the face of another person is printed on the mask, the mask area cannot be estimated. Also, since there are face areas of different persons, there are cases where erroneous authentication is performed.
SUMMARY OF THE INVENTION It is an object of the present invention to provide a face authentication device that solves the problems described above and can reduce authentication errors when a person wears a mask or the like printed with another person's face or the like.

A face authentication device according to one aspect of the present invention includes:
face image input means for acquiring an input face image;
registered face image holding means for pre-holding a plurality of registered face images;
face mixture determination means for determining whether face regions of different persons are mixed in the input face image;
and control means for performing a predetermined notification when the face mixture determination means determines that face areas of different persons are mixed in the input face image.

According to the present invention, it is possible to realize a face authentication device capable of reducing authentication errors when a person wears a mask or the like printed with another person's face or the like.

(A) is a system configuration diagram of a face authentication device according to an embodiment of the present invention, (B) is a diagram showing an example of application to smartphone unlocking, and (C) is a diagram showing an example of application to gate authentication. is. 2 is a functional block diagram of the face authentication device of Example 1. FIG. 5 is a flow chart of unlock processing in the first embodiment. FIG. 4 is a flowchart of face authentication processing in step S301 of FIG. 3. FIG. FIG. 5 is a flowchart of face image acquisition processing in step S401 of FIG. 4 ; FIG. FIG. 5 is a flowchart of mixed face determination processing in step S402 of FIG. 4; FIG. FIG. 11 is a table exemplifying processing results of mixed face determination processing; FIG. (A) is a diagram showing an example of a plurality of persons and a plurality of mask patterns, and (B) is a diagram for explaining case No. 1 in FIG. 7 . (A) is a diagram for explaining the No. 2 case in FIG. 7, (B) is a diagram for explaining the No. 3 case in FIG. 7, (C) is a diagram for explaining the No. 4 case in FIG. 7, and (D) is a diagram. 7 is a diagram for explaining the case of No. 7 in No. 7, and (E) is a diagram for explaining the case of No. 6 in FIG. 7. FIG. FIG. 11 is a functional block diagram of a face authentication device of Example 2; 10 is a flowchart of face authentication processing of Example 2. FIG. FIG. 12 is a flowchart showing details of step S110 in FIG. 11; FIG. FIG. 11 is a functional block diagram of a face authentication device of Example 3; 14 is a flow chart showing face image acquisition processing according to the third embodiment. FIG. 15 is a flowchart of a suspicious person recording process in step S142 of FIG. 14; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the present invention will be described below using examples with reference to the accompanying drawings. In each drawing, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.
Also, in the embodiment, an example in which a camera and a computer are combined as a face authentication device will be described. However, face authentication devices include digital still cameras, digital movie cameras, smart phones, tablet computers, general-purpose computers, vehicle-mounted cameras, drone cameras, robots, gate authentication systems, and the like.

[Computer configuration]
FIG. 1A is a system configuration diagram of a face authentication device according to an embodiment of the present invention.
In FIG. 1A, 101 is a CPU (Central Processing Unit) that controls the entire computer device 100, and 102 is a ROM (Read Only Memory) that stores programs and parameters that do not need to be rewritten. A RAM (Random Access Memory) 103 temporarily stores programs and data supplied from an external device or the like.

The external storage device 104 is a storage device such as a hard disk, SSD, or memory card connected to the computer device 100 . The external storage device 104 may include optical discs such as DVDs and CDs that are detachable from the computer device 100, magnetic or optical cards, IC cards, memory cards, and the like.
An input device interface 105 is an interface with an input device 109 such as a pointing device or a keyboard for inputting data by user's operation, and with the camera 110 .

The output device interface 106 is an interface with a monitor display 111 for displaying data held by the computer device 100 and supplied data, and a speaker 112 for producing sound. A communication interface 107 is a communication interface for connecting to a network line 113 such as the Internet to transmit and receive signals. A system bus 108 is a transmission line that communicably connects the units 101 to 107 . Each process in the flowcharts described later is configured to be performed by the CPU 101 executing a program stored in a computer-readable storage medium such as the ROM 102 .

An application example of the face authentication device of this embodiment will be described with reference to FIGS. 1(B) and 1(C).
FIG. 1B is a diagram showing an example of application to smartphone unlocking. When the user of the smartphone in FIG. 1B presses the power button 116, the camera 114 captures an image of the user's face. After that, the unlocking process shown in FIG. 3, which will be described later, is executed, and if the authentication is correct, the lock is unlocked and the home screen is displayed on the monitor display unit 115 . On the other hand, if the authentication fails, a message such as authentication failure is displayed.

FIG. 1C is a diagram showing an example applied to a gate authentication system. In FIG. 1C, when user 119 comes to the front of the door, camera 117 takes an image of the user. After that, the unlocking process shown in FIG. 3, which will be described later, is executed, and if the authentication is correct, the door 120 is unlocked and the user can open the door. On the other hand, when the authentication fails, a display such as authentication failure is displayed on the monitor display unit 118 .

[Configuration of face authentication device]
FIG. 2 is a functional block diagram of the face authentication device of Example 1. FIG. Some or all of the functional blocks shown in FIG. 2 are implemented by causing the CPU 101 as a computer to execute a computer program stored in the ROM 102 as a storage medium. However, some or all of them may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used.

Moreover, each functional block shown in FIG. 2 may not be built in the same housing, and may be configured by a combination of separate devices connected to each other via signal paths. For example, some of the functional blocks in FIG. 2 may be arranged in an external server or the like.
A photographing processing unit 200 generates a face image using an imaging unit 201 and a face detection unit 202 .
The imaging unit 201 acquires an image from a camera 110 including a CMOS image sensor or the like.
The face detection unit 202 identifies a face area for each frame of the image obtained by the imaging unit 201 . A known face detection technique, such as the method described in Non-Patent Document 1, may be used to identify the face area. Alternatively, other known methods may be used.

The face authentication unit 210 performs authentication using the face image obtained by the imaging processing unit 200 . In this embodiment, face authentication is performed by matching with a registered person only when it is determined that the person is not wearing a mask on which another person's face is printed. Otherwise, an authentication error occurs. The face authentication unit 210 further includes functional blocks indicated at 211-217. In the following description, the mask may be any shield that shields at least part of the face, including stickers, eye patches, glasses, hats, and the like.

A face image input unit 211 receives an input of a face image obtained by the imaging processing unit 200 . For example, when the photographing processing unit 200 temporarily stores a face image in the RAM 103, the face image input unit 211 receives information such as the memory address of the image storage destination in the RAM 103 and the size. Alternatively, when the imaging processing unit 200 saves the image in the external storage device 104 , it receives the file address of the save destination of the image in the external storage device 104 . After that, the face image input unit 211 reads and acquires the image based on the information such as the memory address, size, file address, and the like. Here, the face image input unit 211 functions as face image input means for acquiring an input face image.

The registered face image holding unit 212 performs a registered face image holding step of holding the face image of the registered person in advance. For example, in the example of the smartphone shown in FIG. 1B, the face image of the smartphone user is registered in advance. Further, for example, in the example of gate authentication shown in FIG. 1C, face images of persons permitted to enter the room are registered in advance. These registered face images are held in the RAM 103 and the external storage device 104 . At that time, it may be stored in association with an ID or the like for identifying a person. Here, the registered face image holding unit 212 functions as a registered face image holding unit that holds a plurality of registered face images in advance.

The face matching unit 213 determines whether or not the two face images match. That is, the face matching unit 213 functions as face matching means for matching the input face image with the registered face image and determining a match.
Then, the face matching unit 213 extracts, for example, information that can identify an individual from the face image, and converts it into a feature amount vector (also called a feature amount). The degree of similarity between two face images is determined by calculating the degree of similarity of the feature vector obtained from each, and determining that the two face images match if the degree of similarity is equal to or greater than a predetermined value. If the degree of similarity is less than a predetermined value, it is determined that the two face images do not match. Cosine similarity, the reciprocal of the distance between vectors, or the like may be used as the similarity of the feature vectors.

In addition, Convolutional Neural Networks (CNN), which is a type of neural network, may be used in extracting the feature amount from the face image. CNN is trained in advance so that information for identifying individuals can be extracted from face images. Learning means tuning a large number of parameters required for CNN processing. For learning, a known method may be used. For example, a loss function using softmax may be defined, and parameters of the CNN may be adjusted by back propagation so that the value of the loss function becomes small.

Also, in the learning of the CNN, it is possible to learn in advance using partially masked images so that it can be determined that the person is the same person even if a part of the face image is masked. The methods are not limited to these.
The registered face search unit 214 searches the registered face image storage unit 212 for a face image that matches the face image obtained from the face image input unit 211, the masked face generation unit 215, or the like. Specifically, the face collation unit 213 is used to extract the feature amount of each face image, and all the face images in the registered face image holding unit 212 are collated with the given face image to identify a matching person. get.

For efficient processing, a feature amount may be extracted in advance from the face image held by the registered face image holding unit 212 and stored in the registered face image holding unit 212 . In this case, the feature amount is extracted from the given face image, the degree of similarity with the feature amount of the registered face image holding unit 212 is calculated, and a person having a degree of similarity equal to or higher than a predetermined value is acquired as a matching person. You can Note that the registered face search method is not limited to these.

The masked face generation unit 215 generates an image in which a part of the face is masked from the face image obtained from the face image input unit 211 or the like. The area to be shielded is a shielded face, which is an image in which a part of the face is shielded by applying a predetermined shielding pattern for several face areas such as the mouth, eyes, head, one eye, nose, etc. Generate an image. Here, the masked face generation unit 215 functions as a masked face generation unit that generates a plurality of types of masked face images in which part of the face of the input face image is masked.

Specifically, the coordinates of organ points such as the eyes, the corners of the eyes, the nose, and the corners of the mouth are obtained, and a masking image is generated based on the coordinates of the organ points. For example, in the case of a shielding image that hides the mouth, the coordinates of the three points of the right and left corners of the mouth and the tip of the nose, and a black-painted image to be superimposed are prepared in advance as a shielding template. An affine transformation matrix is obtained from the coordinates of the three points obtained by the organ detection and the three coordinates of the shielding template held in advance, and the template image is affine-transformed and superimposed on the face image. Alternatively, without using a template, a large number of organ points such as the outline of the face may be obtained, and these organ points may be connected to hide the mouth and the like.

The portion to be shielded may be painted with a single color such as black or white, or a texture prepared in advance may be applied.
Alternatively, an edge or the like on the face image may be detected, and the masked area may be determined using the edge as a boundary. This makes it possible to capture the contours of masks, eyepatches, glasses, hats, etc., determine shielding regions along the contours, and effectively generate an image in which only the regions with ornaments are shielded.

For the organ detection described above, a known method such as a method of estimating that an organ point exists at a pixel position with the highest probability of existence using a CNN for estimating the existence probability of an organ point from a face image on a pixel-by-pixel basis is used. You can use it. Note that the method for generating a masked face image partly masked by a masked image is not limited to these.
The face mixture determination unit 216 determines whether or not the face image to be authenticated obtained from the face image input unit 211 or the like includes face areas of a plurality of persons. Here, the mixed-face determining unit 216 functions as mixed-face determination means for determining whether face regions of different persons are mixed in the input face image. Details of the mixed face determination process will be described later with reference to FIG.

Note that the mixed face determination process may be realized by a method other than the flowchart shown in FIG. For example, a CNN or the like that has been learned so as to be able to classify a face image photographed while wearing a mask on which another person's face is printed, such as a print mask, and a face image of a real face, is prepared in advance.
Then, a face image obtained from the face image input unit 211 or the like may be input to the CNN to obtain a determination result as to whether or not face areas of a plurality of persons are mixed. It should be noted that the face mixture determination method is not limited to these.

The face authentication control unit 217 determines whether or not to perform face authentication according to the determination result of the mixed face determination unit 216 . Specifically, face authentication is not performed when the face mixture determination unit 216 determines that faces of different persons are mixed. Conversely, when the face mixture determination unit 216 determines that faces of different persons are not mixed, face authentication is performed. Details will be described with reference to the flowchart of FIG. 4, which will be described later.

The error face processing unit 220 performs error processing when face authentication is not executed by the face authentication control unit 217 of the face authentication unit 210 .
The error notification unit 221 notifies the user and surroundings of an authentication error via the monitor display 111 and the speaker 112 . Note that the error notification unit 221 may notify a predetermined circuit in the face recognition device. A predetermined circuit in the face recognition device locks part of the operation of the face recognition device or transmits a notification to an external security system when receiving the notification from the error notification unit 221. may be performed. Alternatively, the error may be written to a log file. It should be noted that it is not essential to perform an action upon notification, and no action may be performed.
[Unlock process]

FIG. 3 is a flowchart of unlocking processing according to the first embodiment.
3 to 6, 11, 12, 14 and 15 are executed by the CPU 101 as a computer executing a computer program stored in the ROM 102 or the like. Here, the CPU 101 functions as control means for controlling the entire face authentication apparatus based on a program.

The unlocking process is started when the user inputs an unlocking request to the face authentication device. For example, in the case of the smart phone shown in FIG. 1B, when the power button 116 is pressed, the processing in FIG. 3 is activated. In the case of the gate authentication shown in FIG. 1C, the process of FIG. 3 is activated when the camera detects that the user has come to the gate. Alternatively, this process may be started by detecting that the user presses a button or the like (not shown).

In step S301, face authentication processing is executed to determine whether the person photographed by the camera is a registered person. The details of the face authentication processing in step S301 will be described later using the flowchart of FIG.
In step S302, when the result of the face authentication processing in step S301 is authentication success, the process proceeds to step S303, otherwise the process proceeds to step S301.

In step S303, unlock processing is performed. For example, the smartphone shown in FIG. 1B is unlocked and the home screen is displayed on the monitor display unit 115 . In the case of the gate authentication shown in FIG. 1C, the door 120 is unlocked for a predetermined period of time, allowing the user to enter the room.
[Face authentication processing]

FIG. 4 is a flow chart of face authentication processing in step S301 of FIG.
In step S401 (face image input step), a process of acquiring a face image from the camera is performed. Details will be described later with reference to the flowchart of FIG.
In step S402, it is determined whether or not the face image acquired in step S401 includes faces of different persons (face mixture determination step). Details will be described later with reference to the flowchart of FIG.

In step S403, processing is performed by the face authentication control unit 217, and when it is determined in step S402 that faces of different persons are mixed, the process proceeds to step S405. Otherwise, the process moves to step S404.
In step S404, the registered face search unit 214 is used to search for a face image that matches the face image acquired in step S401.

That is, the CPU 101 as a control means retains a registered face image that matches the input face image when the face mixture determination means determines that face areas of different persons are not mixed in the input face image. Search by means. Then, when only one person is retrieved, it is determined that the authentication is successful. Otherwise, it is determined that authentication has failed.
In step S405, the error notification unit 221 notifies the user of the error.

In this manner, the CPU 101 as a control unit issues a predetermined notification indicating that the authentication has failed when the mixed face determination unit determines that the input face image includes face areas of different persons. control so that Here, in steps S403 to S405, if it is determined that the face areas of different persons do not coexist, a registered face image that matches the input face image is searched, and if it is determined that the face areas of different persons coexist, , a control process is performed to provide a predetermined notification.
[Face Image Acquisition Processing]

FIG. 5 is a flow chart of the facial image acquisition processing in step S401 of FIG.
The processing in FIG. 5 is mainly processed in the photographing processing section 200 .
In step S<b>501 , the imaging unit 201 obtains the current frame image from the camera 110 .
In step S502, the face detection unit 202 acquires a face area from the frame image obtained in step S501. When multiple face regions are acquired, the face region near the center of the frame is adopted. Alternatively, the number of acquired face regions may be set to zero, and processing may be performed to determine that face acquisition has failed.

In step S503, it is determined whether or not the face has been successfully acquired. When one face area is acquired from the frame image in step S502, the process moves to step S504. Otherwise, the process moves to step S505.
In step S504, the image of the face area is cut out from the frame image to generate the face image.

In step S505, it is determined whether or not the determination in step S503 has failed continuously. Specifically, when the number of failures in face acquisition in step S503 exceeds a predetermined number, it is determined that failures have occurred consecutively. Alternatively, when the duration of consecutive failures exceeds a predetermined time, it is determined that consecutive failures have occurred. If it is determined that there have been consecutive failures, the process moves to step S506. Otherwise, return to step S501.

At step S506, the authentication process is interrupted. Specifically, the unlocking process in the flowchart of FIG. 3 is interrupted. Then, a message such as "face cannot be detected correctly" is displayed on the monitor display 111 to the user. Alternatively, the speaker 112 is used to transmit by voice.
[Face Mixing Judgment Processing]

FIG. 6 is a flow chart of the mixed face determination process in step S402 of FIG.
This process is called from step S402 in the flowchart of FIG. 4, and the face mixture determination unit 216 determines whether or not the face image obtained in step S401 includes faces of different persons.
In step S<b>601 , a masked face image is generated using the masked face generation unit 215 . Specifically, organ detection processing is performed on a given face image, and several predetermined shielding patterns are applied. For example, a pattern that hides the mouth, a pattern that hides both eyes, and the like are applied to generate a plurality of types of masked face images.

In step S602, the registered face search unit 214 is searched for a registered face image of a matching registered person for each shielded face image obtained in step S601. If there are duplicate registered images acquired, the duplicate registered images are ignored. For example, when the same three registered face images are obtained from three different masked face images, only one registered face image is adopted and the other two are not adopted.

In step S603, a collation result list used in later-described processing is initialized to be empty.
Step S604 is a loop for sequentially processing the registered face images obtained in step S602, and the registered face images are assigned numbers starting from 0 in order. In order to refer to this using i as a variable, i is initialized to 0 first. Further, when i is less than the number of registered face images, the process proceeds to step S605, and when i is equal to or greater than the number of registered face images, the loop is exited and the process proceeds to step S609.

Step S605 is a loop for sequentially processing the masked face images obtained in step S601, and the masked face images are assigned numbers starting from 0 in order. In order to refer to this using j as a variable, j is initialized to 0 first. Further, when j is less than the number of masked face images, the process proceeds to step S606, and when j is equal to or greater than the number of masked face images, the loop is exited and the process proceeds to step S608.

In step S606, the face matching unit 213 is used to match the registered face image i with the masked face image j. Then, a set of "registered face image i, masked face image j, and matching result" is recorded in the matching result list. It should be noted that when the registered face image is searched for each masked face image in step S602, the registered face image i and the masked face image j are also collated. Therefore, the result of matching in step S602 may be cached, and the result of matching between the registered face image i and the masked face image j may be obtained from the cache.

Step S607 is the end of the masked face image loop, adds 1 to j, and returns to step S605.
Step S608 is the end of the registered face image loop, increments i by 1, and returns to step S604.
In step S609, it is determined whether or not the collation result list is empty. The collation result list becomes empty when, for example, no registered face image is obtained in step S602. When the matching result list is empty, the process moves to step S612. Otherwise, the process moves to step S610.

In step S610, it is determined whether or not all matching results in the matching result list match. If all the collation results match, the process moves to step S612. Otherwise, the process moves to step S611.
In step S611, the processing result of this flowchart is set to "face mixed".
In step S612, the processing result of this flowchart is set to "no mixed face".
[Example of mixed face determination processing]

An example of mixed face determination processing will be described with reference to FIGS. 7 to 9. FIG.
FIG. 7 is a table exemplifying the processing result of the mixed face determination processing.
Column 701 indicates whether the person in the input face image is wearing a print mask. Wearing indicates wearing a mask, and non-wearing indicates not wearing a mask.

A column 702 indicates whether or not the face image of the person displayed on the print mask is registered in the registered face image holding unit 212 . “Yes” indicates that the face image of the person displayed on the print mask is registered in the registered face image holding unit 212 , and “No” indicates that it is not registered in the registered face image holding unit 212 .

A column 703 indicates whether or not the face image of the person appearing in the input image is registered in the registered face image holding unit 212 . “Yes” indicates that the face image of the person appearing in the input image is registered in the registered face image holding unit 212 , and “No” indicates that it is not registered in the registered face image holding unit 212 .
A column 704 indicates the state of the matching result list obtained in step S609 of the mixed face determination process under the conditions shown in columns 701 to 703. FIG. "Match and non-match exist" indicates that both matching and non-matching matching results are present in the matching result list. "Match all" indicates that there are only matching matching results in the matching result list. "Empty" indicates that the matching result list is empty.

A column 705 indicates the final result of the mixed face determination process. "Mixed face" indicates that step S611 was reached as a result of the mixed face process, and "no mixed face" indicates that "no mixed face" was determined in step S612.
Each case shown in FIG. 7 will be described in detail below.

FIG. 8A is a diagram showing an example of multiple people and multiple mask patterns. That is, FIG. 8A shows a person 801, a person 802, a print mask 803, and a print mask 804 that appear in the following description. A print mask 803 is a mask on which the mouth of the person 802 is printed. A print mask 804 is a mask printed with the mouth of a person who is neither the person 801 nor the person 802 .

The case No. 1 in FIG. 7 will be described in detail. Case No. 1 is a case in which an unregistered person who is not registered in the registered face image holding unit 212 wears a print mask printed with the mouth of a person registered in the registered face image holding unit 212 . In other words, it can be said that this is a case of impersonating a registered person.

FIG. 8B is a diagram for explaining the case No. 1 in FIG.
In FIG. 8B, an input face image 805 is a face image of a person 801 wearing a print mask 803, and it is assumed that this is input to the mixed face determination processing flow of FIG.
A masked face image is generated (step S601) for the input face image 805, and masked face images 806 and 807, for example, are obtained. Here, for the sake of explanation, it is assumed that there are two shielding patterns. Then, registered face images are searched using these shielded face images (step S602).

In this example, the person 801 is not registered, and only the person 802 is registered in the registered face image holding unit 212 . Therefore, a registered face image 808 is obtained by the registered face search unit 214 based on the pattern of the print mask from the masked face image 807 . On the other hand, in the masked face image 806, since the design of the print mask portion is masked, the corresponding person cannot be retrieved and the registered face image cannot be obtained. Next, the registered face image 808, the masked face image 806, and the masked face image 807 are collated to obtain a match/mismatch determination result (steps S604 to S608).

Here, as shown in FIG. 8B, the masked face image 806 is determined to be "mismatched by face matching", and the masked face image 807 is determined to be "matched by face matching". Then, it is determined whether or not these determination results are the same (step S610). In this case, the determination results are different, so the determination result of "face mixed" is obtained (step S611). That is, the mixed face determination means searches for registered face images that match the plurality of types of masked face images, respectively, in the plurality of registered face images held in the registered face image holding means. If different, it is determined that the face areas of different persons are mixed.

Next, FIG. 9A is a diagram for explaining the case of No. 2 in FIG.
Case No. 2 is a case in which a registered person registered in the registered face image holding unit 212 wears a print mask 804 printed with the mouth of a person not registered in the registered face image holding unit 212 .
In FIG. 9A, an input face image 901a is a face image of a person 801 wearing a print mask 804, and it is assumed that this is input to the mixed face determination processing flow of FIG.

A masked face image is generated (step S601) for the input face image 901a, and masked face images 902a and 903a are obtained. Then, registered face images are searched using these shielded face images (step S602). In this example, since the person 801 is registered, a registered face image 904a is obtained by the registered face search unit 214 from the features of the person's eyes using the masked face image 902a. Then, the registered face image 904a, the masked face image 902a, and the masked face image 903a are collated to obtain a match/mismatch determination result (steps S604 to S608).

Here, as shown in FIG. 9A, it is determined that "face matching matches" for the masked face image 902a, and "face matching does not match" for the masked face image 903a. Then, it is determined whether or not these determination results are the same (step S610). In this case, the determination results are different, so that the processing result of "face mixed" is obtained (step S611).

Next, FIG. 9B is a diagram for explaining case No. 3 in FIG.
Case No. 3 is a case in which a registered person registered in the registered face image holding unit 212 wears a print mask 803 printed with the mouth of the person registered in the registered face image holding unit 212 .
In FIG. 9B, an input face image 901b is a face image of a person 801 wearing a print mask 803, and it is assumed that this is input to the mixed face determination processing flow of FIG.

A masked face image is generated (step S601) for the input face image 901b, and a masked face image 902b and a masked face image 903b are obtained. Then, registered face images are searched using these shielded face images (step S602). In this example, since the person 801 and the person 802 are registered, a registered face image 904b can be obtained from the features of the person's eyes using the masked face image 902b.

In addition, a registered face image 905b is obtained from the mask features of the mouth with the masked face image 903b. Then, the registered face image and the masked face image are collated, and the judgment result of face collation matching/face collation non-coincidence is obtained as shown in FIG. 9B (steps S604 to S608). Since these determination results are not the same, a processing result of "face mixed" is obtained (step S611).

Next, FIG. 9(C) is a diagram for explaining the case of No. 4 in FIG.
Case No. 4 is a case in which a person who is not registered in the registered face image holding unit 212 wears a print mask printed with the mouth of the person who is not registered in the registered face image holding unit 212 .
In FIG. 9C, an input face image 901c is a face image of a person 801 wearing a print mask 804, and it is assumed that this is input to the mixed face determination processing flow of FIG. A masked face image is generated (step S601) for the input face image 901c, and masked face images 902c and 903c are obtained.

Then, registered face images are searched using these shielded face images (step S602). In this example, since the person 801 is not registered, no registered face image is obtained. As a result, no matching result between the registered face image and the masked face image is obtained (steps S604 to S608). Therefore, since the collation result list is empty (step S609), the processing result is obtained as "no mixed face" (step S612).

Next, FIG. 9(D) is a diagram for explaining the case of No. 5 in FIG.
Case No. 5 is a case where the registered person registered in the registered face image holding unit 212 has a bare face without wearing a mask.
In FIG. 9D, it is assumed that an input face image 901d is a face image of a person 801 with a real face, and that this is input to face mixture determination processing. A masked face image is generated (step S601) for the input face image 901d, and a masked face image 902d and a masked face image 903d are obtained. Then, registered face images are searched using these shielded face images (step S602).

In this example, since the person 801 is registered, a registered face image 904d is obtained from the features of the person's eyes based on the masked face image 902d. Also, a registered face image 904d is obtained from the features of the person's mouth based on the masked face image 903d.
Then, the registered face image and the masked face image are collated, and the face collation matching determination result is obtained as shown in FIG. 9C (steps S604 to S608). Since all determination results are the same, a processing result of "no mixed faces" is obtained (step S612).

Next, FIG. 9E is a diagram for explaining the case of No. 6 in FIG.
The case No. 6 in FIG. 7 will be described in detail. Case No. 6 is a case in which a person who is not registered in the registered face image holding unit 212 has a bare face without a mask.
It is assumed that the input face image 901e is the real face image of the person 801 and is input to the mixed face determination processing flow of FIG. A masked face image is generated (step S601) for the input face image 901e, and a masked face image 902e and a masked face image 903e are obtained. Then, registered face images are searched using these shielded face images (step S602).

In this example, the person 801 is not registered, so a registered face image cannot be obtained. As a result, no matching result between the registered face image and the masked face image is obtained (steps S604 to S608). Therefore, since the collation result list is empty (step S609), the processing result "no mixed faces" is obtained (step S612).

As described above, in this embodiment, face authentication processing is not executed for a person wearing a mask, eyepatch, or the like printed with a face image of a person other than the person himself/herself. Therefore, it is possible to suppress face authentication errors due to wearing a mask on which a part of another person's face is printed.

In the first embodiment, an error occurs when a user whose face is to be authenticated wears a print mask or the like, and it is necessary to remove the print mask. In this embodiment, face authentication is enabled without removing a mask or the like. Specifically, when it is determined that faces are mixed, the user is instructed to change the facial expression, and face authentication is performed using the real face region in which the facial expression has changed.

FIG. 10 is a functional block diagram of the face authentication device of Example 2. FIG. The difference from FIG. 2 is that functional blocks 222 to 224 are added to the error face processing section 220 in FIG. Descriptions of functional blocks having the same reference numerals as those in FIG. 2 will be omitted or simplified.
As in FIG. 2, part or all of the functional blocks shown in FIG. 10 are realized by causing the CPU 101 as a computer to execute a computer program stored in the ROM 102 as a storage medium. However, some or all of them may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like may be used.

Also, each functional block shown in FIG. 10 may not be built in the same housing, and may be configured by separate devices connected to each other via signal paths. Also, for example, some of the functional blocks in FIG. 10 may be arranged in an external server or the like.
The instruction notification unit 222 is a functional block for instructing the user to make a predetermined facial expression change, and instructs the user via the monitor display 111 and the speaker 112 . Instructions include, for example, instructions to smile, instructions to blink, and instructions to open the mouth. Alternatively, a combination of these may be used.

The facial expression change measuring unit 223 measures the facial expression change for each of the predetermined masking candidate regions. Specifically, a plurality of consecutive frames of face images are obtained, and relative positional change amounts of organ point coordinates are measured or estimated.
For example, the relative coordinates of each organ point are obtained for each frame with reference to the nose or the like. Measure the amount of change in movement of relative coordinates of organ points between frames. By obtaining the amount of change in movement of the organ point to be shielded for each of the shielding candidate areas and taking the sum of the obtained amounts, the amount of change in the shielding candidate area is output.

Alternatively, a CNN or the like may be learned so as to estimate an area with facial expression change and the amount of facial expression change from a plurality of frames of face images, and the area with facial expression change may be identified using it. Note that the method of facial expression change measurement is not limited to these.
The low change masking unit 224 uses the masking candidate regions and the amount of change from the facial expression change measuring unit 223 to generate an image in which the region with the small amount of change is masked. Specifically, a masking candidate area showing a change amount less than a predetermined threshold is obtained, and the pixels included in the masking candidate area are blacked out to hide a part of the input face image that has little facial expression change. Generate an image.

FIG. 11 is a flow chart of face authentication processing according to the second embodiment.
Since steps S401 to S405 in FIG. 11 are the same as the processing in FIG. 4, description thereof is omitted.
In step S110, extraction processing of a natural face area is performed.

FIG. 12 is a flow chart showing details of step S110 in FIG.
In step S121, the control unit uses the instruction notification unit 222 to instruct the user to change the facial expression. That is, when the face mixture determination means determines that the input face image includes face areas of different persons, an authentication error notification is sent to the user in step S405, and a change in facial expression is instructed in step S121. give notice.

In step S122, multiple frames of face images are obtained. Specifically, by performing the face image acquisition process shown in FIG. 5 for several consecutive frames, face images of a plurality of frames are acquired.
In step S123, the facial expression change measuring unit 223 is used to measure the facial expression change for each masking candidate region.

In step S124, it is determined whether or not the user has indicated a change in facial expression in response to the instruction in step S121. Specifically, it is determined that there is no facial expression change when the facial expression change is less than a predetermined threshold value in all of the masking candidate areas in step S123. If there is no facial expression change, the process returns to step S121; otherwise, the process proceeds to step S125.

In step S125, the low change shielding unit 224 is used to shield the areas with little facial expression change, thereby generating an excluded face image. That is, after instructing the user to change the facial expression by a predetermined notification, the control means measures the facial expression change in the input facial image in step S123, and generates a facial image excluding areas with little facial expression variation in steps S124 and S125. do. It should be noted that when excluding areas with little change in facial expression, the area with little change in facial expression may be shielded as in the present embodiment, or may be simply excluded from the face recognition area.

As described above, when it is determined that faces are mixed, the user is instructed to change the facial expression, and the bare face area in which the facial expression has changed is measured, thereby distinguishing the bare face area with higher accuracy. As a result, highly accurate face authentication can be performed, so face authentication can be performed without removing a mask or the like.

In Example 1 and the like, when a suspicious person such as an impersonator wears a printed mask, face recognition can be prevented, but the suspicious person cannot be recorded.
In the third embodiment, when it is determined that the face is mixed, and the user cancels the face authentication without succeeding in the face authentication, the user is determined to be a suspicious person, and a face image that can be searched for the suspicious person. to record the image of the suspicious person, and further share the suspicious person information with other face recognition devices.

FIG. 13 is a functional block diagram of a face authentication device according to the third embodiment; The difference from FIG. 10 is that a suspicious person recording unit 230 is added in FIG. 13, and description of functional blocks with the same reference numerals as in FIG. 10 will be omitted or simplified.
As in FIGS. 2 and 10, some of the functional blocks shown in FIG. 13 are implemented by causing the CPU 101 as a computer to execute a computer program stored in the ROM 102 as a storage medium. However, some or all of them may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like may be used.

Also, each functional block shown in FIG. 13 may not be built in the same housing, and may be configured by separate devices connected to each other via signal paths. Also, for example, some of the functional blocks in FIG. 13 may be arranged in an external server or the like.
A suspicious person recording unit 230 is for recording a face image or video of a suspicious person when the user who requested face authentication is a suspicious person, and has functional blocks 231 to 234 .
The suspicious person determination unit 231 determines whether or not the person determined to have a mixed face in the mixed face determination in step S402 is a suspicious person. Alternatively, it is determined whether or not the person determined to have failed face recognition in step S404 is a suspicious person. Here, the suspicious person determination unit 231 functions as suspicious person determination means for determining whether or not a person is a suspicious person.

Specifically, when face authentication is stopped (interrupted), if face authentication fails for a predetermined number of times or more, it is determined that the person is a suspicious person. Abort (interrupt) of face authentication is automatically determined to be aborted (interrupted) when face images are not continuously obtained from captured images. Alternatively, when the user presses the power button 116 to turn off the power after the face authentication fails, it may be determined that the face authentication is stopped (suspended).

It should be noted that the method for detecting cancellation (interruption) of face authentication is not limited to these. The number of face authentication failures is the sum of the number of times when it is determined that the face is mixed in the face authentication processing flow shown in FIG. to count. Alternatively, only one may be counted. Note that the suspicious person determination method is not limited to these.

The suspicious person face holding unit 232 stores and holds the face image of the person judged to be a suspicious person. For example, when it is determined that a mixed face is present and the face authentication is interrupted, the masked face image for which matching is not matched in the mixed face determination is retained. Alternatively, when the mixed face determination is passed and the face authentication fails, the face image used in the failed face authentication is stored.

Further, the suspicious person information stored in the suspicious person face holding unit 232 can be accessed from other face recognition devices, or can be shared by being uploaded to a predetermined shared server. Alternatively, the suspicious person face holding unit 232 may be provided in a predetermined shared server. Similarly, the registered image holding unit 212 may also be provided in a predetermined shared server.
Here, the suspicious person face retaining unit 232 functions as a suspicious person face retaining means for registering the masked face image of the person when the suspicious person determining means determines that the person to be authenticated is a suspicious person. ing.

The suspicious person image recording unit 233 (suspicious person image recording means) records the image in which the suspicious person appears and is stored in the suspicious person face holding unit 232 . Specifically, an image obtained from the imaging unit 201 is recorded, and a section in which a suspicious person appears is specified by a suspicious person recording process (flowchart in FIG. 15), which will be described later, and the image of that portion is recorded. Store for a long time. The video of the section where the suspicious person does not appear is deleted after a predetermined period of time, and only the video in which the suspicious person appears is stored for a long period of time. Alternatively, the images for a predetermined period before and after the appearance of the suspicious person may be stored together for a long period of time as a record.

In addition, in order to facilitate later retrieval of the section in which the suspicious person appears, predetermined additional information indicating that the video of the suspicious person is recorded in the section in which the suspicious person appears is recorded and retained together with the video. is desirable. The additional information includes, for example, information indicating that the person is suspicious, a combination of date and time, ID of the suspicious person, and the like. Here, the suspicious person video recording section 233 functions as a suspicious person face holding means for registering the masked face image of the person when the suspicious person determination means determines that the person to be authenticated is a suspicious person. there is Note that the method of recording the suspicious person video is not limited to these.

The suspicious person display control unit 234 displays a suspicious person face image (for example, a masked face image) registered and held by the suspicious person face holding unit 232 and a video showing a suspicious person registered and held by the suspicious person video recording unit 233. and the above-described additional information are displayed on display means such as the monitor display 111 or the like. That is, the suspicious person display control unit 234 functions as a suspicious person display control unit for displaying, on the display unit, the masked face image registered in the suspicious person face holding unit or the image stored in the suspicious person video recording unit. ing.
Specifically, in the case of the example of the smartphone in FIG. 1B, after unlocking, the suspicious person display application is automatically or manually activated and displayed on the monitor display 111 .

In the example of gate authentication in FIG. 1B, when another computer device accesses the face authentication device via the network line 113, the face authentication device can be displayed on the monitor display of the other computer device. Send a signal. In addition, the suspicious person's face holding unit 232 holds the face image of the unmasked face before being masked, and when the face image of the suspicious person is displayed, the unmasked image of the unmasked face is also stored. You may enable it to display. It should be noted that the suspicious person display method is not limited to these.

FIG. 14 is a flow chart showing face image acquisition processing according to the third embodiment.
Since steps S501 to S506 are the same as those in FIG. 5, the description thereof is omitted.
In S141, the suspicious person determination unit 231 is used to determine whether face authentication has failed a predetermined number of times or more. Specifically, for example, when it is determined that faces are mixed in the face authentication processing flow of FIG.

If the number of times is equal to or greater than the predetermined threshold value, the process proceeds to step S142. Otherwise, the processing flow of FIG. 14 ends.
In step S142, a suspicious person recording process is performed. That is, the suspicious person determining means determines whether or not the person is a suspicious person when authentication of the input face image fails a predetermined number of times or more.

FIG. 15 is a flowchart of the suspicious person recording process in step S142 of FIG.
In step S151, it is determined whether or not the mixed face determination was determined to be unsuccessful due to mixed faces immediately before. If it is determined to be a failure due to mixed faces, the process proceeds to step S152. Otherwise, the process proceeds to step S153.

In step S152, a masked face image (a face image partly masked by a masked image) for which matching is not matched in face mixture determination is acquired. Specifically, for example, in the case of FIG. 8B, a masked face image indicated by 806 is acquired.
In step S153, the face image of the subject whose face has been authenticated is obtained. Specifically, the face image of the subject whose face has been authenticated in step S404 is acquired. However, as a premise of step S153, it is determined in step S141 of FIG. 14 that the number of authentication failures is equal to or greater than the predetermined number of times.

When the first and third embodiments are combined, the face image acquired in step S153 is an unregistered real face because authentication has failed, although faces are not mixed in face mixture determination. An image will be acquired. On the other hand, when combined with the second embodiment, a bare face may be obtained in step S153, but a masked face image may be obtained by the flow of the extraction processing of the bare face region in FIG.

In step S154, the masked face image obtained in step S152 or the bare face image obtained in step S153 is stored in the suspicious person face holding unit 232 and registered. That is, in step S154, the suspicious person face holding unit 232 registers the masked face image of the person when the suspicious person determining means determines that the person to be authenticated is a suspicious person.
In step S155, based on the face image of the suspicious person registered in step S154, the suspicious person video recording unit 233 performs processing for saving the video in which the suspicious person appears.

In the third embodiment and the like, before step S301 of the unlocking process, it is determined whether or not the user who requested face authentication is a person existing in the suspicious person face holding unit 232, and If it exists in the part 232, the unlocking process may be interrupted. Alternatively, in that case, the image of the suspicious person may be recorded. As a result, when a suspicious person appears again, defensive measures against the suspicious person (for example, reporting to a security company or the police) can be taken.

Also, in the case of gate authentication, surveillance cameras may be installed in addition to the camera 117 near the door. Therefore, when a person existing in the suspicious person's face holding unit 232 is detected by another surveillance camera, the image of the suspicious person may be recorded. This makes it possible to track and monitor suspicious persons over a wide area.

When displaying the video of the suspicious person, if the suspicious person display control unit 234 displays a face image of a print mask or the like painted in black, for example, the image of the suspicious person can be displayed without being confused by the pattern. The characteristics of a person can be displayed in an easy-to-understand manner.
As described above, according to the third embodiment, it is possible to acquire and register a masked face image of a suspicious person whose face is partly masked by a print mask or the like, a real face, or the like as a suspicious person face image. Therefore, the image of the suspicious person can be retrieved and saved without being affected by the pattern of the print mask or the like. Furthermore, by sharing the information about the suspicious person with other face recognition devices, etc., it is possible to improve the recognition accuracy and efficiency of the suspicious person.

The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention. They are not excluded from the scope of the invention.
The present invention can be embodied as, for example, a system, device, method, program, storage medium, or the like. Specifically, it may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of one device.

It should be noted that a computer program that implements the functions of the above-described embodiments for part or all of the control in this embodiment may be supplied to a face authentication device or the like via a network or various storage media. Then, the computer (or CPU, MPU, etc.) in the face authentication device or the like may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

200 photographing processing unit 201 imaging unit 202 face detection unit 210 face authentication unit 211 face image input unit 212 registered face image holding unit 213 face collation unit 214 registered face search unit 215 masked face generation unit 216 mixed face determination unit 217 face authentication control unit 220 error face processing unit 221 error notification unit

Claims (13)

face image input means for acquiring an input face image;
registered face image holding means for pre-holding a plurality of registered face images;
face mixture determination means for determining whether face regions of different persons are mixed in the input face image;
and a control unit for performing a predetermined notification when the mixed face determination unit determines that the input face image includes face areas of different persons. When the face mixture determining means determines that the input face image does not include face regions of different persons, the control means determines the registered face image that matches the input face image as the registered face image. 2. The face authentication device according to claim 1, wherein the search is performed from holding means. 3. The face authentication device according to claim 2, further comprising face collation means for collating said input face image and said registered face image to determine a match. a masked face generating means for generating a plurality of types of masked face images in which a part of the face of the input face image is masked; It is characterized by searching whether a face image exists in each of the plurality of registered face images held in the registered face image holding means, and judging that face areas of different persons are mixed when each search result is different. The face authentication device according to any one of claims 1 to 3. 5. The face authentication device according to any one of claims 1 to 4, wherein said predetermined notification is to notify that authentication of said input face image has failed. 6. The face authentication device according to any one of claims 1 to 5, wherein the predetermined notification includes a notification instructing the user to change the facial expression. 7. The face according to claim 6, wherein after instructing the user to change the facial expression by the predetermined notification, the facial expression change in the input facial image is measured, and a facial image is generated excluding an area with little facial expression change. Authenticator. 8. The apparatus according to any one of claims 1 to 7, further comprising suspicious person determination means for determining whether or not said person is a suspicious person when authentication of said input face image fails more than a predetermined number of times. A facial recognition device as described. 9. The face recognition system according to claim 8, further comprising suspicious person face holding means for registering a masked face image of said person when said person is determined to be said suspicious person by said suspicious person determination means. Device. 10. The face authentication device according to claim 9, further comprising suspicious person image recording means for storing an image including the suspicious person based on the masked face image registered in the suspicious person face holding means. 11. The face authentication device according to claim 10, further comprising suspicious person display control means for displaying, on a display means, the masked face image registered in the suspicious person face holding means or the video stored by the suspicious person video recording means. . a face image input step of acquiring an input face image;
a registered face image holding step of pre-holding a plurality of registered face images;
a mixed face determination step of determining whether face regions of different persons are mixed in the input face image;
When it is determined that the face regions of different persons are not mixed in the face mixture determination step, the registered face image matching the input face image is searched, and when it is determined that the face regions of different persons are mixed, , and a control step of performing a predetermined notification. A computer program for causing a computer to function as each means of the face authentication device according to any one of claims 1 to 11.

Images