JP2024080541A - Method, program, and computer system for face authentication - Google Patents

Abstract

To provide a method, program and computer system for face authentication, which do not use the step of instructing a user to make a facial expression, and offer high accuracy, less burden on authentication targets and highly real-time performance.SOLUTION: A computer-implemented face authentication method is provided, comprising a first authentication step of computing a degree of similarity based on feature point extraction, a second authentication step of computing a degree of similarity based on expression parameter extraction, and a step of making an authentication determination based on both the degree of similarity computed in the first authentication step and the degree of similarity computed in the second authentication step.SELECTED DRAWING: Figure 2

Description

The present application relates generally to a method, program, and device for authentication based on a human face image, and more particularly to a method, program, and computer system for face authentication that is highly accurate and reduces the burden on the person to be authenticated.
Regarding.

Authentication methods that identify a person based on a human facial image are becoming common. For example, it is widely used to authenticate users when signing in to a mobile information terminal, and for security at building entrances. Facial recognition has the advantage of placing a small burden on users as it does not require them to remember passwords or carry special devices such as tokens, making impersonation relatively difficult, and relatively little psychological resistance on the part of users.

Generally, facial recognition technology extracts features from an image of a user's face and compares them with previously saved features to determine whether the user is the person in question, or to which of multiple people the user corresponds. Here, the issue is how to determine the threshold for determining whether the face matches. If the threshold is set too high, the probability of a false positive, in which a different person is recognized as the person in question, increases, while if the threshold is set too low, the probability of a false negative, in which the person in question is not recognized as the person in question, increases.

One method to address these issues is face recognition technology that uses the user's facial expressions. For example, a technology that can quantify the degree of a person's facial expression (e.g., Patent Document 1) has been disclosed that improves the accuracy of face recognition by having the user smile (e.g., Patent Document 2). However, such a method has issues such as placing a burden on the person to be authenticated (particularly, people may be reluctant to make facial expressions in a public environment, such as when authenticating at the entrance to a building), and requiring a long time for authentication (particularly, it is difficult to apply it to cases where passing people need to be authenticated in real time).

The present invention has been made in consideration of these problems, and aims to provide a face recognition method, program, and computer system that is highly accurate (high recognition rate and low false recognition rate), places a small burden on the user, and can be applied to real-time authentication.

Published Patent Application JP2014-206903 Published Patent Application JP2016-009453

To provide a face authentication method, program, and computer system that is highly accurate, places a small burden on the person being authenticated, and is highly real-time.

The present invention solves the above problems by providing a face recognition method executed by a computer, which includes a first authentication step of calculating a similarity with registered face information based on feature point extraction, a second authentication step of calculating a similarity with registered facial information based on facial expression parameter extraction, and a step of determining authentication based on both the first similarity calculated in the first authentication step and the second similarity calculated in the second authentication step, and does not include a step of instructing a user to make a facial expression.

The present invention also solves the above problem by providing the face recognition method described in paragraph 0008, in which, in the step of determining whether or not authentication has been successful, it is determined that the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold.

The present invention also solves the above problem by providing the face recognition method described in paragraph 0008, in which in the second authentication step, a similarity determination is made using representative values of facial expression parameters extracted from a video image of the user's face within a predetermined time period.

The present invention also solves the above problem by providing the face recognition method described in paragraph 0008, in which the second authentication step uses facial expression parameters of anger, sadness, surprise, and happiness, and does not use facial expression parameters of disgust, fear, and neutral.

The present invention also solves the above problem by providing a face recognition program that causes a computer to execute a first set of instructions for calculating a similarity based on feature point extraction, a second set of authentication instructions for calculating a similarity based on facial expression parameter extraction, and a set of instructions for making an authentication decision based on both the first similarity calculated by the first set of authentication instructions and the second similarity calculated by the second set of authentication instructions, and does not include a set of instructions that instruct a user to make a facial expression.

The present invention also solves the above problem by providing the face recognition program described in paragraph 0012, in which the set of instructions for determining whether or not authentication has been successful includes a set of instructions for determining that authentication has been successful only if the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold.

The present invention also solves the above problem by providing the face recognition program described in paragraph 0012, in which the second group of authentication commands includes a group of commands for performing a similarity determination using representative values of facial expression parameters extracted from a video image of the user's face within a predetermined time period.

The present invention also solves the above problem by providing the face recognition program described in paragraph 0012, in which the second group of authentication commands uses facial expression parameters of anger, sadness, surprise, and happiness, but does not use facial expression parameters of disgust, fear, and neutral.

The present invention also solves the above problem by providing a computer system configured to execute the program described in any one of paragraphs 0012 to 0015.

A method, program, and computer system for face authentication that is highly accurate, places a small burden on the person being authenticated, and is highly real-time is provided.

1 is an embodiment of an information system in which the method according to the present invention is implemented; 2 illustrates a functional structure of an embodiment of an authentication device according to the present invention. 11 is a flowchart of a process performed by a first authentication unit according to an embodiment of the present invention. FIG. 1 is a conceptual diagram showing the degree of facial expression of a user in normal times. 13 is a flowchart of a process performed by a second authentication unit in the embodiment of the present invention. FIG. 13 is a diagram illustrating another embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the drawings. Figure 1 shows an embodiment of an information system in which the method according to the present invention is executed. This embodiment is intended for security at the gates of buildings, but the present invention is not limited to this and can also be applied to, for example, user authentication of mobile information terminals, and identification of suspicious people from an unspecified number of passersby on the street.

The camera (100) is a means for capturing an image of the user's face, and it is desirable that the camera (100) can capture moving images at a predetermined frame rate. The authentication device (101) is a computer device for processing the image of the user's face captured by the camera (100) and performing facial authentication. The authentication device is not limited to a computer device installed within a facility managed by an organization such as a company, but may be a computer device with at least a part of its functions installed outside the organization and managed by a cloud service or the like. The user (102) is a user who is the subject of facial authentication. The output device (103) is a device for outputting the result of authentication, and may be a display device (display, monitor) for human visual confirmation, a device that generates a signal such as an audio alarm, an interface device for operating the gate opening and closing mechanism, or a relay program for sending the authentication result to another program.

Figure 2 shows the functional structure of an authentication device (101) according to one embodiment of the present invention. The face image acquisition unit (201) is a means for acquiring an image to be the subject of face authentication from images acquired by the camera (100). In the present invention, since it is necessary to read facial expression parameters from the image, it is preferable that the image is not a still image but a moving image of about a few seconds in length. It is preferable that the program is configured to determine which image includes the entire face of the user (103) and is in focus, and automatically select it. The program may be configured to give instructions to the user (103) by message, voice, etc., so that the face of the user (103) can be photographed appropriately.

The first authentication unit (202) is a means for calculating the similarity between the face of the user (103) and the registered face information based on the feature points of the user's face image. The processing details will be described later. The second authentication unit (203) is a means for calculating the similarity between the face of the user (103) and the registered face information based on the expression parameters. The processing details will be described later. The authentication judgment unit (204) is a means for returning the result of the final judgment (whether or not the authentication was successful, whether the user's face matches the registered one, which of the multiple registered face information is closest to the user's face, etc.) based on the similarity calculated by the first authentication unit (202) and the similarity calculated by the second authentication unit (203). The face information database (205) is a means for storing data of a person to be compared with the face image of one or more users (102), and it is desirable that the face information data is set up in advance by photographing and processing the face image of the user (102). It is preferable that the facial information database (205) stores in advance the facial image feature values used in the first authentication unit (202) and the facial expression parameters used in the second authentication unit (203) obtained by the same acquisition method as when authentication is performed. It is preferable that each record in the facial information database (205) includes an identifier (key) that uniquely identifies each person to be compared. In addition, each record (or another storage means accessed by the record identifier) may include attribute information of the person to be compared (identification number, name, age, gender, etc.). When the present invention is applied to face recognition processing when signing in to a smartphone or a personal computer, the facial information database (205) only needs to include face information of one person or a very small number of people, not multiple people.

The above functions may be realized by cooperative processing between computer hardware and a dedicated program (software), but each of the above functions does not have to correspond to one program. For example, the first authentication unit (202) and the second authentication unit (203) may be implemented by one program. As mentioned above, these functions are not limited to computers managed by a company, and some of them may be realized by computers operated or managed outside the organization by a cloud service, for example. They may also be realized by hardware installed in a camera, smartphone, etc.

Figure 3 shows a flowchart of the process performed by the first authentication unit (202) in an embodiment of the present invention. This embodiment assumes a case where it is determined whether there is facial information in the facial information database (205) that is sufficiently similar to the face of the user (103) (the similarity falls within a predetermined threshold range) (for example, security at the gate of a building), but even in a case where the target facial information has already been determined and it is determined whether the user (103) is the person in question (for example, security when signing in to a smartphone), the similarity based on the facial features is returned as a result.

First, the first authentication unit (202) acquires a face image of the user (102) to be subjected to face authentication from the face image acquisition unit (201) (S301). This image may be a still image, or may be a short (several seconds) moving image. It is desirable that the face image acquisition unit (201) provides an image that is selected so that the face of the user (102) is clearly shown, and that has been subjected to appropriate pre-processing such as brightness normalization and edge enhancement. Next, the first authentication unit (202) extracts features from the face image (S302). Known features such as Haar-like features may be used. Next, the first authentication unit (202) reads out the features of the face information registered in the face information database (205), compares them with the features of the face image of the user (102), and calculates a similarity (matching score) (S303). The calculation of the similarity may use a known method in the field of face authentication technology. Alternatively, the similarity may be output by having machine learning software or a neural network learn from training data. In addition, the feature point detection step (S302) may also be performed by a neural network (typically the convolutional layer of a CNN (convolutional neural network)). In this case, attribute information such as age and gender contained in the face information database (205) may be used as input parameters for determining the degree of similarity (the inventor's experiments have shown that doing so can further improve the accuracy of face recognition). Finally, the first authentication unit (202) returns the identifier of the face information in the face information database (205) with the highest degree of similarity and the degree of similarity (S304).

As shown in FIG. 4, the inventor's experiments have revealed that even when a user is not consciously making a facial expression, the facial expression parameters of the facial image show values unique to that user. Even when it is difficult to make a definitive similarity judgment using facial feature point information alone, it is possible to identify people with high accuracy by additionally using differences in facial expression parameters. In conventional face recognition, there is a trade-off in that a smaller similarity threshold in similarity judgment increases the possibility of judging a target person as a different person (false negative), and a larger similarity threshold in similarity judgment increases the possibility of judging a different person as the target person (false positive). However, by additionally performing authentication based on facial expression parameters that are unconsciously visible on the user's face even when not making a facial expression, the possibility of both false negatives and false positives can be reduced.

5 shows a flowchart of the process performed by the second authentication unit (203) in the embodiment of the present invention. The second authentication unit (203) calculates the similarity based on the facial expression parameters of the user. Here, the facial expression is a facial expression in a normal (unconscious) state in which the user is not intentionally making a facial expression (for example, by being instructed to "smile"). First, the second authentication unit (203) acquires a facial image of the user (102) to be subjected to facial authentication from the facial image acquisition unit (201) (S501). This image is preferably a short (several seconds) moving image. Next, the second authentication unit (203) extracts facial expression parameters from the facial image (S502). The extraction of the facial expression parameters may be performed by a method such as that disclosed in Patent Document 1, for example, but it is preferable to calculate a representative value such as an average value from multiple values of the facial expression parameters in multiple frames in the moving image and use the calculated representative value in the authentication process. Next, the second authentication unit (203) reads out facial expression parameters from a record in the face information database (205) that corresponds to the identifier output by the first authentication unit (202) (S503). Finally, the second authentication unit (203) compares the facial expression parameters obtained from the face image of the user (102) with the facial expression parameters of the above record in the face information database (205) and calculates the degree of similarity (S504).

The facial expression parameters may consist of angry, sad, surprised, happy, disgust, fear, and neutral, each of which may be expressed as a normalized numerical value. The similarity between the facial expression parameters extracted from the user's face image and the facial expression parameters in the face information database (204) may be calculated by calculating the Euclidean distance between the facial expression parameters and normalizing it, or may be calculated based on another formula. The similarity may be calculated by learning using machine learning or deep learning. In this case, attribute information such as age and gender included in the face information database (205) may be used as an input parameter for determining the similarity (the inventor's experiments have revealed that doing so can further improve the accuracy of face recognition).

The inventor's experiments have revealed that the display parameters of anger, sadness, surprise, and happiness vary relatively widely among individuals and tend to have values unique to each individual, whereas the display parameters of disgust, fear, and neutral do not vary significantly among individuals. Therefore, the second authentication unit (203) may calculate the similarity using only the facial expression parameters of anger, sadness, surprise, and happiness, or may calculate the similarity by relatively increasing the weighting of the facial expression parameters of anger, sadness, surprise, and happiness. The processing of the first authentication processing unit (202) and the processing of the second authentication processing unit (203) may be performed sequentially, or may be performed simultaneously in parallel to speed up the authentication processing.

The authentication judgment unit (204) performs a final authentication judgment (typically, whether the face image of the user (102) is the same person as the face information registered in the face information database (204)) based on both the similarity calculated by the first authentication processing unit (202) and the similarity calculated by the second authentication processing unit (203). For example, a threshold A (a stricter judgment threshold) and a threshold B (a looser judgment threshold) may be set, and when the similarity calculated by the first authentication processing unit (202) is a value between the thresholds A and B (i.e., when it is highly likely that they are the same person but not certain), it may be determined whether they are the same person based on whether the similarity calculated by the second authentication processing unit (203) exceeds a threshold C (a predetermined value different from the thresholds A and B). Also, it may be determined that the authentication is successful (the user matches the person registered in the face information database (204)) only when both the first authentication processing unit (202) and the second authentication processing unit (203) are successful (the similarity between the user's face and the items in the face information database (204) is equal to or greater than a predetermined threshold). As an alternative method, a final similarity may be calculated from the similarity determined by the first authentication processing unit (202) and the similarity determined by the second authentication processing unit (203), and a final authentication decision may be made based on the calculated final similarity. The final similarity may be calculated based on a weighted average of the similarity determined by the first authentication processing unit (202) and the similarity determined by the second authentication processing unit (203), or a nonlinear calculation formula may be used. The final authentication decision may be made by having machine learning or deep learning learn using the similarity calculated by the first authentication processing unit (202) and the similarity calculated by the second authentication processing unit (203) as inputs.

FIG. 6 shows another embodiment of the present invention. FIG. 6a) shows an embodiment in which the face authentication method of the present invention is applied to the authentication of a user (602) when signing in (logging in) to a portable information device such as a smartphone (601) or a personal computer. In this embodiment, the authentication device (101) is the smartphone (601) itself (part of the processing may be performed by another computer device via a network), and the output device (103) may be a program that conveys the authentication result to a program that performs the login processing of the smartphone. In this case, the element corresponding to the face information database (205) may be a file stored in the portable information device, and may be configured to store only the face information of one user (the authorized user of the portable information device). In this case, the result of the first authentication unit (202) may be only the similarity between the user (602) who is signing in and the face information (feature points) in the file, and the second authentication unit (203) may compare it with the facial expression parameters in the face information database (205) (file) without obtaining an identifier from the first authentication unit (202).

FIG. 6b) is an example in which the face recognition method according to the present invention is applied to multiple moving subjects (603). The camera (604) may be a fixed camera or may be mounted on a moving object such as a drone. The authentication device (101) and the output device (103) in this embodiment may be the same as those described in FIG. 1. In this embodiment, it is preferable that the image acquisition unit (201) is configured to execute software for performing face recognition, extract a human face portion from a captured video image, perform necessary preprocessing on it, and pass it to the first authentication processing unit (202) and the second authentication processing unit (203). In addition, the output of the output device (103) may be attribute information (identification number, name, etc.) included in a record in the face information database (205) that has a similarity to the multiple subjects (603) equal to or greater than a predetermined threshold. In any of the embodiments in FIG. 6, the hardware (camera, computer, etc.) used may be the same as that used in face recognition in the prior art, and the present invention can be implemented by simply changing the software.

(Technically significant effects of the present invention)
The face recognition method has the inherent advantages of being less burdensome to users, being relatively difficult to impersonate, and having relatively little psychological resistance on the part of users, but the present invention provides further advantages by improving the authentication system without increasing the psychological resistance of users or sacrificing real-time performance. In addition, the hardware devices such as cameras used in conventional face recognition can be reused, and the authentication accuracy can be improved simply by changing the software, making it cost-effective.

Claims (9)

A first authentication step of calculating a similarity between the extracted feature points and registered face information;
a second authentication step of calculating a similarity between the extracted facial expression parameters and registered facial expression parameters based on the extracted facial expression parameters;
performing authentication based on both the first similarity calculated in the first authentication step and the second similarity calculated in the second authentication step;
does not include a step of instructing a user to make a facial expression,
A computer implemented method for facial recognition. In the step of determining whether or not the authentication is successful, the authentication is determined to be successful only when the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold.
The face authentication method according to claim 1 . In the second authentication step, a similarity determination is performed using a representative value of a facial expression parameter extracted from a video image of the face of the user within a predetermined time period.
The face authentication method according to claim 1 . In the second authentication step, facial expression parameters of anger, sadness, surprise, and happiness are used, and facial expression parameters of disgust, fear, and neutral are not used.
The face authentication method according to claim 1 . A first set of instructions for calculating a similarity based on feature point extraction;
A second set of authentication commands for calculating a similarity based on extracted facial expression parameters;
a group of instructions for making an authentication decision based on both the first similarity calculated by the group of instructions for the first authentication and the second similarity calculated by the group of instructions for the second authentication;
Does not contain commands that instruct the user to make facial expressions;
Face recognition program. the set of instructions for determining whether or not authentication is successful includes a set of instructions for determining that the authentication is successful only if the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold.
The face recognition program according to claim 5 . the second authentication command group includes a command group for performing a similarity determination using a representative value of a facial expression parameter extracted from a video image of the face of the user within a predetermined time period;
The face recognition program according to claim 5 . the second set of authentication instructions use facial expression parameters of anger, sadness, surprise, and happiness, and do not use facial expression parameters of disgust, fear, and neutral;
The face recognition program according to claim 5 . A computer system configured to execute the program according to any one of claims 5 to 8.

Images