JP6679291B2 - Applicant authentication device, authentication method, and security authentication system using the method - Google Patents

Description

  The present invention relates to an applicant authentication device, an authentication method, and a security authentication system using the method, and more particularly, to an authentication technique for correctly determining the identity of the applicant.

  2. Description of the Related Art Conventionally, in situations such as cardless payment and cashless payment in hardware transactions, it is necessary to carry out troublesome procedures such as presentation of an identification card for identity verification, signing, and stamping. In addition to the gold merchandise transaction, it is also necessary to verify the identity of the person in various situations, such as immigration examinations during travel, entrance procedures to specific facilities, and identity verification for applications and examinations. Furthermore, what is needed in these situations is a password for certifying the person himself, and the management of the password is currently entrusted to him.

  As a means for simplifying these troublesome procedures, various technologies related to biometric authentication have been proposed recently. In particular, with the progress of image analysis technology, it has been proposed to use face recognition technology to simplify the settlement procedure, examination procedure, and approval procedure. However, although the image analysis technology has advanced, the accuracy of face recognition has not been completed, and it is also true that crimes due to forgery and impersonation are endless. In particular, the face authentication technology that has been conventionally proposed is one that identifies a person by extracting a person's face from an image captured by a camera or the like, analyzing the characteristic elements of the person's face, and collating it with information registered in advance. Many. However, with such a technique of capturing a person as a one-dimensional still image, it is difficult to detect forgery or camouflage using a facial photograph of the person, and it is difficult to prevent a crime due to impersonation.

  Therefore, as disclosed in Patent Document 1, by changing the facial expression of a person (for example, a smile or a wink), the face before the change (normal face) and the face after the change (a smile or a wink face) are the same. Techniques for determining sex have also been proposed. According to this technique, since the person must change the facial expression on the spot, it is possible to see a spoofed photograph using the photograph of the person or the person who cannot change the facial expression.

  As a technique for determining the identity of a person by changing the facial expression, for example, a method described in Patent Document 2 has been proposed.

  However, even if the facial expression is changed, it is possible to pretend by preparing a changed face image in advance if the authentication mechanism is understood.

  In order to prevent such a loophole, a person is photographed from a three-dimensional angle using a wide-angle camera and a reflector as described in Patent Document 3, and the background shown in the image is made different. Therefore, a technique for identifying forgery or forgery has been proposed. Here, it is considered that the impersonation is eliminated by using not only the face image but also the background image thereof, but there is a drawback that the system configuration for authentication is complicated and large.

  On the other hand, there is also a method disclosed in Patent Document 4 as a technique for performing authentication by combining a person and a background without inviting a complicated system configuration. The technique disclosed here is characterized in that when the face of a person to be authenticated is photographed by a camera, the background is changed. Specifically, by taking two pictures with different backgrounds at the same place and determining whether or not the same person appears in these two pictures, it was confirmed that the person was taken at the same location. Is to prove that.

  According to this method, without using large-scale equipment such as a three-dimensionally arranged camera or reflecting mirror, you can take two pictures by changing the background behind the person while standing in front of the camera. Since it is good, the system configuration can be simplified. However, forgery and disguise is easy once you know which background will be used.

JP 2012-109976 A JP, 2003-233816, A JP 2007-11456 A JP, 2007-26330, A

  As described above, it is hard to say that any of the conventionally proposed authentication techniques has a high degree of perfection. It can be said that this is because the basic technical idea is static determination that treats the authentication target as a one-dimensional still image captured at a specific timing. Therefore, no matter how the person and the background are combined, or how many images are added to the authentication target, if the person and the background are known, it is relatively easy to forge or camouflage. This leads to the undesired result.

  Furthermore, the conventional authentication technology is designed based on the basic concept of comparing and collating the captured image with the information stored in the database, so that the comparison information or collation information required for authentication is stored in the database in advance. You need to register. For this reason, the amount of information to be registered increases as the accuracy of authentication is improved, and a large-capacity database is required, or the database capacity must be scaled out many times. There is a trouble in the system configuration that it does not happen.

  Further, conventionally, the purpose of using a background image as information for authentication is to confirm the correctness of the place where the person was photographed in many cases. Therefore, if the place where the authentication is performed is specified, it is not so difficult to obtain the background image. Furthermore, it is also possible to easily combine the background image and the person image with a known image combining technique. This is also the main factor that cannot prevent impersonation. In order to solve this, a technique of changing the background has been proposed as described in Patent Document 4, but the relationship between the background image before the change and the background image after the change is considered. Since it does not exist, it cannot be an effective means for solving the above-mentioned drawbacks caused by the static determination.

  In addition, in Patent Document 4, even if the background is changed, the facial expressions of the person in the image are the same, so whether the person actually in the shooting location is the person himself or is he really alive? There is also a drawback that it cannot be accurately determined. This drawback may cause a fatal misjudgment in the case of confirming the identity of a person at a remote place only by the transmitted face image like the Internet communication. In particular, in a situation where confirmation of survival is required, there is a major drawback that the accuracy of judgment is significantly reduced.

  An object of the present invention is to solve the above-mentioned drawbacks and to provide a new authentication technique capable of greatly improving the accuracy of identity verification.

  It is another object of the present invention to provide an authentication technique that improves the reliability of authentication while reducing the burden on the person to be authenticated during authentication.

  Still another object of the present invention is to provide a security authentication system that can obtain a safe and reliable authentication result without increasing the size and complexity of the device configuration.

  In order to achieve the above object, the applicant authentication apparatus according to the present invention as set forth in claim 1 changes the expression of the applicant and the background of the background at a first timing and the expression of the applicant. An image information receiving unit that receives a changed facial expression and a second image of the background taken at a second timing, and extracts facial expression information and background information from the first image and the second image. And an element extraction unit that determines the identity of the applicant based on the facial expression information extracted from the first image and the facial expression information extracted from the second image, and the element extraction unit extracted from the first image. The background determination unit that determines the identity of the shooting location based on the background information extracted from the second image and the background information extracted from the second image, and the applicant's identity verification in response to the determination results of the facial expression determination unit and the background determination unit. The background determination unit includes an authentication determination unit that performs Characterized by a determination requirement time information from a first timing reaches the second timing.

  The applicant authentication apparatus according to claim 2 is characterized in that the background is a still image.

  The applicant authentication apparatus according to claim 3 is characterized in that the background determination unit determines the identity of the shooting location by comparing the background information in the first image with the background information in the second image.

  The applicant authentication apparatus according to claim 4 is characterized in that the background is a moving image.

  In the applicant authentication apparatus according to claim 5, the background determination unit collates the background information extracted from the second image with the moving image information that appears with the lapse of time from the first timing to the second timing. It is characterized by determining the identity of the shooting location.

  The applicant authentication apparatus according to claim 6 confirms that both the facial expression information and background information extracted from the second image and the facial expression information and background information extracted from the first image have changed. This is characterized by determining the identity of the shooting locations.

  The applicant authentication apparatus according to claim 7 is characterized in that the background includes a still image and a moving image.

  The applicant authentication apparatus according to claim 8 is characterized in that the background determination unit determines the identity of the shooting location by combining the background determination process of the still image and the background determination process of the moving image.

  The applicant authentication apparatus according to claim 9 is characterized in that, when the background determination section cannot confirm the identity of the shooting location, the determination by the facial expression determination section is ignored and the authentication is not suitable.

  The applicant authentication apparatus according to claim 10 collects an image of a facial expression of the applicant and environmental information at the time of the imaging at a first timing, then changes the facial expression of the applicant, and displays the changed facial expression. The information collecting unit that collects the captured image and the environmental information at the time of the capturing at the second timing, and the relationship between the image captured at the first timing and the image captured at the second timing is the same for the applicant. The facial expression determination unit that determines the sex, and the environmental information collected at the first timing and the environmental information collected at the second timing are compared as dynamic information along the time axis to determine the identity of the applicant's place. It is characterized in that it has an environment judging section for judging, and an authentication judging section for authenticating the applicant based on the judgment results of the facial expression judging section and the environment judging section.

  The applicant authentication apparatus according to claim 11 uses, as the environmental information, information that can be detected by a sensor, such as sound information, vibration information, or atmospheric pressure information obtained from the environment in which the applicant is photographed. Characterize.

  According to the present invention as set forth in claim 12, in the authentication method for confirming the identity of the applicant based on an image obtained by photographing the expression of the applicant and the background thereof at a predetermined place, the expression of the applicant and the background thereof are A first step of receiving the first image information photographed at the timing 1 and the second image information obtained by changing the facial expression of the applicant and the changed facial expression and the background thereof; A second step of extracting information about a facial expression and information about a background from the image information and the second image information, the facial expression information extracted from the first image information and the facial expression information extracted from the second image information. And a facial expression determination step of determining the identity of the applicant by calculating the degree of similarity (or based on the mutual relevance thereof), and the background information extracted from the first image information and the second image information. The background information A background determination step of determining the identity of the shooting location in association with the passage of time from the timing to the second timing, and authenticating the applicant based on the determination results in the facial expression determination step and the background determination step. The characteristic applicant authentication method is disclosed.

  In the applicant authentication method according to claim 13, the applicant is the person himself / herself when both the confirmation of the identity of the shooting location in the background determination step and the confirmation of the identity of the applicant in the facial expression determination step are established. It is characterized by authenticating that there is.

  The applicant authentication method according to claim 14 follows a first step of collecting information regarding an expression of the applicant photographed at a first timing and information regarding an environment at the time of the photographing, and the first timing. A second step of collecting information about the facial expression of the applicant taken at the second timing and information about the environment at the second timing, and information about the facial expression collected at the first step and the second step. The third step of determining the identity of the applicant based on the above, and the fourth step of determining the identity of the location of the applicant by collating the information about the environment collected at the first timing and the second timing. It is characterized by including a step and a fifth step of making an applicant authentication determination in response to the determination results obtained in the third and fourth steps.

  The authentication method according to a fifteenth aspect is characterized in that the time from the first timing to the second timing is arbitrarily changed.

  The authentication method according to a sixteenth aspect is characterized in that, as the environmental information, information that can be detected by a sensor from a place where the image is taken is used.

  The security authentication system according to claim 17 collects an image regarding the facial expression of the applicant and information regarding the environment of the place where the applicant is at the first timing, and at the second timing subsequent to the first timing, An information collecting unit that collects images related to facial expressions and environmental information related to the location, information received by the information collecting unit, image information related to facial expressions and environmental information related to the location are extracted from the received information, and the extracted information is extracted. Based on the first determination that determines whether the images collected at the first timing and the second timing are images of the same person based on the information about the two facial expressions, and the extracted information about the two environments Based on the judgment result by the information judgment unit and the information judgment unit that executes the second judgment for judging whether or not the applicant exists in the same place, the applicant is the same person. If it is determined that the applicant's place is the same place, the authentication determination unit for instructing the authentication permission and the security service is executed to the applicant in response to the instruction from the authentication determination unit. And a service execution unit.

  The authentication system according to claim 18 is characterized in that the time from the first timing to the second timing can be changed.

  In the present invention described above, the word "applicant" means a settler in a money transaction, a person who undergoes immigration or entrance examination, a person who applies for an examination or application, or an approval procedure for an application or request. It includes all those who need proof of identity, such as those who need

  The word "expression" is intended to mean not only facial expressions, but also changes such as poses, gestures, and body movements that accompany movements that are expressed in each part of the body that can be identified as image information. Is. For example, examples of facial expressions include smiles, winks, opening and closing of mouths, etc., and facial expressions include raising and lowering of hands and twisting of the neck.

  Furthermore, the word “time lapse” does not mean, for example, two pieces of information A and B collected at different timings at the same place as static information independent of each other, but is related to each other connected by the same time axis. It refers to capturing as dynamic information. In other words, instead of individually recognizing the information A as the information A at a certain timing and the information B at another timing, the information A changes (or does not change) to the information B over time. By considering the information as dynamic information, it means that the information A and B have a temporal connection. For details, refer to the following description.

  In addition, the word "environment" is detected by using a sensor (including a sound collection microphone) from the background image that can be taken at the same time when the facial expression of the applicant is taken, or from around the place where the facial expression of the applicant is taken. It includes possible information, for example, information on sound, vibration, atmospheric pressure, temperature, humidity, position and the like.

  A major feature of the present invention is that the temporal relevance (time elapsed information) of the environmental information acquired at the first and second timings is a determination requirement. Therefore, by arbitrarily setting / changing the elapsed time, it is possible to secretly change the information used in the determination process. In other words, two pieces of environment information connected by time are used as a hidden password, and the password itself is changed by changing the time. Normally, the password is used by the person to be authenticated (applicant), but in the present invention, the new idea of having this on the authenticating side greatly reduces the burden on the person to be authenticated for password management. There is.

  The authentication device according to claim 1 determines the identity of the applicant based on the change in facial expression by using the “change in facial expression” and the “background image” of the applicant, and determines the identity between places by using the background image. It is characterized by In particular, the applicant's existence at the same place and being alive by making the judgment of the time elapse of the background image, that is, whether there is temporal continuity in the background image before and after the facial expression changes. You can check if you are (moving). Specifically, by comparing and collating two background images before and after the facial expression change, it is possible to determine whether or not the applicant exists at the same temporally continuous place. Therefore, it is possible to obtain the effect that it is possible to perform highly accurate personal identification and survival confirmation without complicating the device configuration.

  The authentication device according to claims 2 and 3 uses a still image as a background image. In this case, the identity of the place can be confirmed by determining whether the background image before the facial expression change is the same as the background image after the facial expression change. If there is a change in the background, it is identified as a composite image taken at a different location. Therefore, here, it is possible to confirm the identity of the place by determining that the background has not changed over time. This determination can be performed by comparing / collating two images before and after the change, and has an advantage of not requiring complicated equipment or processing unlike three-dimensional analysis.

  The authentication device according to claims 4 to 6 uses a moving image as a background image. As in the case of still images, moving images can be authenticated with high accuracy by using the relationship with the passage of time as a determination requirement. This may be based on the elapsed time (t) from the first timing to the second timing, and may determine the match between the moving image that should appear after the elapsed time and the moving image acquired at the second timing, or The presence of the applicant at the same place may be confirmed by determining whether or not the facial expression and the background have both changed at the second timing with reference to the first timing. In either case, a highly accurate authentication result can be obtained by dynamic determination along the time axis.

  In the authentication device according to the seventh and eighth aspects, the reliability of the authentication can be further enhanced by using both the still image and the moving image as the background image.

  According to the authentication device of claim 9, by using two background images that are temporally connected as a password, it can be considered that the authentication is incompatible unless a correct password is detected. It becomes possible to complete the certification without judging the gender (or without waiting for the judgment result), which is particularly effective in the case where the applicants line up like immigration and admission.

  According to the authentication device of claims 10 and 11, by using the environment information (sound, vibration, atmospheric pressure, etc.) of the place where the applicant is authenticated as an element for determining the identity of the place, the place that can be authenticated is identified. The effect that it can be enlarged is obtained.

  The present invention according to claims 12 and 13 discloses a method for performing authentication based on a change in the facial expression of the applicant and a change in the background over time. Here, the elapsed time information of the background information can be set and managed on the authenticating side as a password judgment tool for the applicant, so that it is possible to obtain the effect of eliminating the password management burden imposed on the applicant.

  Since the present invention according to claims 14 to 16 uses the time from the first timing to the second timing as a requirement for the authentication determination, it is necessary to collate the environment information collected at the first timing and the second timing. Can determine whether or not the applicant is present in the same place. Furthermore, by changing this time arbitrarily, even if the shooting location is specified, it is not possible to impersonate the applicant unless the length of this time is known. Therefore, the confidentiality of the authentication method can be significantly enhanced. Further, since the background image obtained from the authentication place and the environmental information such as sound, vibration, and atmospheric pressure that can be detected by the sensor are used, there is no increase in complicated equipment. In addition, since environmental information can be used as a password as in the case described above, highly reliable authentication can be performed without imposing a burden on the applicant.

  Claims 17 and 18 disclose an authentication system using the authentication method and the authentication device according to the present invention. Here, since the environmental information obtained from the surrounding environment of the applicant's whereabouts such as images, sounds, vibrations, and atmospheric pressure, and the time for collecting this environmental information are used as a password, the application range is wide and high. It is possible to provide a reliable and highly confidential security authentication system.

  As described above, the background information (environmental information) used as the password is not static information at a specific timing but is treated as dynamic information linked by a time axis, and therefore the elapsed time (t) The password itself is a part of the password, and it is a great feature of the present invention that the confidentiality of the password can be greatly improved by arbitrarily changing the time. Therefore, it is possible to provide a secure and highly reliable authentication function at any place and in any place without imposing a password management burden on the applicant without installing an expensive and large-scale device. It is possible to obtain the effect.

1 is a schematic configuration diagram of a security authentication system to which the present invention is applied. It is a figure which shows the timing of imaging with respect to an applicant performed when implementing this invention. It is a functional block diagram of the applicant authentication device of the 1st form for implementing the present invention. It is a flowchart which shows the procedure of the authentication process regarding the 1st form of this invention. It is an internal functional block diagram of the facial expression determination part and the background determination part of the 1st form of this invention. It is a time chart which shows the flow of time progress in the background determination process of the 1st form of this invention. It is a schematic diagram of the authentication image in the 1st form of this invention. It is a flowchart which shows the authentication processing procedure regarding the 2nd form for implementing this invention. It is a time chart which shows the flow of the time passage of the background determination process in the 2nd form of this invention. It is a schematic diagram of the authentication image regarding the 2nd form of this invention. It is a flowchart which shows the authentication processing procedure regarding the 3rd form for implementing this invention. It is a schematic diagram of the authentication image regarding the 3rd form of this invention. It is a figure which shows the acquisition timing and acquisition pattern of the information regarding the facial expression and environment which are sent to this invention. It is a functional block diagram of the security authentication system in the 4th form for implementing this invention. It is a flowchart of the authentication processing procedure regarding the 4th form of this invention. It is a time chart which shows the mechanism of person determination and place determination in the 4th form of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, an outline of a security authentication system to which the present invention is applied will be described with reference to the system configuration diagram of FIG. The system includes a security service execution unit 20 for the applicant 10, a camera 30 that captures the facial expression of the applicant, and an authentication center 40 that receives the captured image and authenticates the applicant. Further, as a means for collecting environmental information for authenticating the applicant, a sound collecting microphone 50, a sensor 60, or the like may be provided as necessary. The camera 30 is mainly used to photograph the facial expression of the applicant together with the background, and the photographed image is transmitted to the authentication center 40 via a communication line. The sound around the shooting location is collected by the sound collecting microphone 50. Further, the vibration and the atmospheric pressure at the shooting location are detected by the sensor 60. The collected sound information, detected vibration, and atmospheric pressure information are transferred to the authentication center via a communication line. Here, the background image other than the applicant's facial expression, light, sound (song), vibration, atmospheric pressure, and other information are defined as “environmental information”. In addition, recently, since various sensors such as a microphone, a vibration sensor, an atmospheric pressure sensor, and a GPS are mounted on a mobile terminal such as a smart phone, the mobile terminal can be used as a means for collecting environmental information. .

  As a security service to which the present invention can be applied, the behavior of the applicant and the authentication procedure thereof will be described by taking an applicant (purchaser) performing cardless payment as an example. First, in FIG. 1, the applicant 10 is photographed by a camera in front of a service execution unit (for example, a service counter at a sales floor). The shooting procedure is started when the applicant 10 stands in front of the camera, as shown in FIG. When the camera 30 captures the facial expression of the applicant 10 (normal face), the camera 30 performs the first shooting at the first timing T1 (first shooting). At this timing T1, the facial expression of the applicant and its background are photographed at the same time. Next, a signal is sent to the applicant 10 to change the facial expression by displaying a chime or a lamp. Based on this signal, the applicant 10 changes the facial expression (e.g., smile or wink). The camera 30 photographs the face and the background of the applicant after the facial expression change at the second timing T2 when the time t has elapsed from the first photographing timing and the facial expression of the applicant has changed due to the signal (second photographing). The applicant's behavior in cardless payment is over. In FIG. 2, the image before the facial expression change captured at the first timing is the information P1 when the background is a still image, the information P3 when the background is a moving image, and the information P3 when the background is a still image and a moving image. The information P5 is transmitted to the authentication center 40 via the communication line. Similarly, the image after the facial expression change captured at the second timing is the information P2 when the background is a still image, the information P4 when the background is a moving image, and the information P6 when the background is a still image and a moving image. Sent to.

  Next, the configuration of the authentication device 40A provided in the authentication center 40 will be described with reference to FIG. The authentication device 40A includes a photographing information receiving unit 41 that receives the image information transmitted from the photographing unit (camera) 30, and an element that extracts facial expression information and background information of the applicant from the received image information as respective elements. An extraction unit 42, a facial expression determination unit 43 that determines the facial expression of the applicant 10 based on the extracted facial expression information, a background determination unit 44 that determines a shooting location based on the background information, a facial expression determination unit 43, and An authentication determination unit 45 that confirms the identity of the applicant based on the determination result of the background determination unit 44.

  Next, a processing flow of the authentication device of FIG. 3 will be described with reference to FIG. Here, the case where the background is a still image, that is, the image information P1 and P2 will be described. The first element extraction processing is executed in the element extraction unit 42 on the image information P1 before the facial expression change captured at the first timing, and the facial expression image information before the change (1st facial expression) and the image captured at that time. The background image information (1st background) is separated. Further, for the facial expression image information P2 captured after the change, the facial expression image information (2nd facial expression) after the facial expression change by the second element extraction processing and the background image information (2nd background) after the facial expression captured simultaneously. ) And are separated.

  The facial expression determination process for confirming the identity of the applicant is executed on the first facial expression before the change and the second facial expression after the change. Here, the purpose of changing the facial expression is to determine whether the person at the shooting location is the same person before and after the change, and whether or not the person is alive (moving). For example, based on the movement of facial expression changes that commonly occur when a person smiles or winks from a normal facial expression, the smile change function and wink change function are calculated and compared with the actually captured image. It can be determined by obtaining the degree of similarity and analyzing whether or not the difference is within a predetermined range. In addition to this method, the technique disclosed in Patent Document 1 or 2 may be used. By such facial expression determination, it can be seen that the persons photographed at two different timings are the same person and are alive (moving) on the spot.

  Subsequently, the background determination shown in FIG. 4 will be described. The background information captured with the applicant at the first and second timings is cut out as the 1st background and the 2nd background by the element extraction processing. Since the background information is for a still image in this embodiment, a matching process of these two pieces of background information is performed in the background determination step, and a match or a mismatch is detected as a result of the matching.

  In response to the coincidence and non-coincidence signals respectively detected by the facial expression determination and the background determination, the authentication determination unit 45 transmits a signal instructing the appropriate countermeasure or the inappropriate countermeasure to the service execution unit 20. At this time, the condition for determining the appropriate response action (the action when the applicant is determined to be the principal) is that the facial expression determination matches and the background determination also matches. On the other hand, if at least one of the facial expression determination and the background determination does not match, the improper handling is forcibly performed (the processing when the applicant is determined not to be the principal).

  The above-described facial expression determination processing and background determination processing will be described in more detail with reference to the functional block diagram of FIG. As shown in FIG. 5, the facial expression determination unit 43 includes an applicant confirmation unit 431 and a facial expression change determination unit 432, while the background determination unit 44 includes a background confirmation unit 441 and a background change determination unit 442. Have In a registration database (registration DB) 70, image information of a normal face (face before change) of the applicant 10 is registered in advance. To register an image in the registration DB 70, the applicant may send an image of a normal facial expression to the authentication center 40 in advance and pre-register it. In this case, the background image may be any image. For example, it is possible to take a picture of one's facial expression using the camera function of the mobile terminal and send it to the authentication center 40 via the Internet line. As the registration DB 70, a magnetic disk, a file server, a cloud database server or the like capable of storing and accumulating transmitted image information can be used. Further, in the registration database 70, a background image used when actually photographing the applicant at the authentication location may be registered in advance. The background image information used to determine the authentication location can be determined by comparing the background image information before and after the facial expression change, and thus does not necessarily need to be registered in the registration DB 70 in advance.

As shown in FIG. 5, the facial expression determination unit 43 and the background determination unit 44 input the facial expression image (1st facial expression) and the background image (1st background) taken at the first timing and store them in the registration DB 70. It has an applicant confirmation unit 431 and a background confirmation unit 441 that compare and collate with the existing information. These are, for images actually taken by the camera 30 installed in the security authentication system of FIG. 1, whether or not they are applicants registered in the registration DB 70 in advance, and a place where the shooting location is correct. It is for confirming whether or not. As a result of the matching process by the applicant confirmation unit 431, if the first facial expression is different from the facial expression information of the applicant registered in the registration DB 70, or as a result of the matching process by the background confirmation unit 441, the first background is registered. If it is different from the information, it is determined that they do not match, and the process of forced authentication (NG) is forced.
The above process can be positioned as a pre-process for the facial expression and background change determination process. By introducing this pre-processing, it is possible to confirm in advance whether or not the subsequent change judgment processing is necessary, so it is possible to shake off impersonation lacking in cleverness in a short time, and reduce the waiting time for the next applicant. It can be shortened. However, as described above, the background confirmation as the preprocessing is not always necessary.

  When a match is confirmed by the above pre-processing, the expression change determination unit 432 and the background change determination unit 442 are activated by the match detection signal. The 1st facial expression information and the 2nd facial expression information acquired at the 1st and 2nd timings are input to the facial expression change determination unit 432, and the applicant photographed at the 1st and 2nd timings by the facial expression change determination process described above It is determined whether they are the same.

  On the other hand, the background change determination unit 442 activated by the coincidence detection signal from the background confirmation unit 441 inputs the 1st background information and the 2nd background information and executes the change determination process. A procedure in which the background change determination unit 442 determines the 1st background information and the 2nd background information as dynamic information along the time axis will be described in more detail with reference to FIG.

  For the 1st background image information obtained at the first shooting timing (time T1) and the 2nd background information obtained at the second shooting timing after the time t has passed, the sameness of these background information is determined. Both are compared and collated to make a determination. Here, when the background is a still image, it may be determined that the two background images have not changed even after the time t has passed (that is, the 1st background image information and the 2nd background image information are the same image). , It is a condition to prove that the applicant was filmed at the same filming location. On the other hand, when the background is a moving image, the proof condition is that these two images are different images. That is, the determination conditions may be opposite when the background is a still image and when the background is a moving image. Here, even if a person who has attempted a fraudulent act such as impersonation knows the background used at the authentication place, at what timing should the image be forged unless the value of the time t is known? I can't judge. Further, even if the background is a still image, it is difficult to determine whether or not the previous still image continues at the time t. In this way, by using the time t as a hidden password, it is possible to provide an authentication technique with extremely high confidentiality.

  Next, a specific example of authentication in the first embodiment will be described with reference to FIG. 7. 7 (a) and 7 (c) show images acquired at the first timing (before facial expression change). The facial expression of the applicant is facial expression A before the change, and the background thereof is the image C at the authentication center. Sent. Next, if the applicant's expression is changed from A to B, and the image taken at the second timing is (b) or (d), (b) the expression changes and the background changes. Therefore, it can be confirmed that the applicant is the same person photographed in the same place. On the other hand, in the image (d), although the facial expression B is correctly changed, the background is the background D different from the background C, and thus the image (d) is determined to be forged. If it is correct that the background changes to D after the time t has passed, conversely, it is determined that (b) is incorrect. Therefore, as long as the authentication center does not know which background is set and the time between the two timings is unknown, it can be understood that the background information functions as an effective password with extremely high confidentiality.

  FIG. 8 shows a determination processing procedure when the background image is a moving image. Since the background of the image captured in FIG. 2 is a moving image, it is transmitted as information P3 before the facial expression change at the first timing and as information P4 after the facial expression change at the second timing. The information P3 before the facial expression change is distinguished into the 1st facial expression information and the 1st moving picture information by the first element extraction processing, and the information P4 after the facial expression change is distinguished into the 2nd facial expression information and the 2nd moving picture information by the second element extraction processing. To be done. In the facial expression determination process, the identity of the applicant is determined and a match / mismatch detection signal is generated as in the first embodiment. On the other hand, in the background moving image determination processing, as will be described later, it is determined whether or not the moving image after the lapse of time is a proper image, and an OK detection signal indicating properness or an NG detection signal indicating improperness is generated. When both the match detection signal of the facial expression determination result and the OK detection signal of the background moving image determination result are satisfied, the authentication determination unit 45 generates a signal for instructing appropriate measures. When the facial expression determination does not match or the background moving image determination is NG, a signal instructing the improper handling is generated.

  The determination process for the case where the background is a moving image will be described in detail with reference to FIG. The information used for the determination process is the first background moving image information P3 acquired at the first timing (time T1) and the background moving image information P4 acquired at the second timing (time T2) after the lapse of time t. Is. The following two methods are effective as a method of determining whether or not the applicant exists in the same place by using these two moving image information as passwords.

  The first method is the screen information of the moving image that appears when the time t has elapsed from the first time (T1) when the first moving image information was acquired and the moving image information that was actually acquired at the second timing (time T2). This is a method of comparing and collating and to determine the identity (processing i in FIG. 9). The second method is a method of determining compatibility between a change in facial expression and a change in background (that two pieces of information change together).

  The transition of the determination image by the above determination method will be described with reference to FIG. FIG. 10A is an image acquired at the first timing (time T1) and shows a facial expression image A before change and a moving image E at that time. After that, after a lapse of time t, it is determined whether or not the background moving image G matches the moving image image appearing when the time t elapses by the first method for the image (b) acquired at the second timing T2. However, if they match, it can be seen that the applicant exists in the same place. If the moving image is an image other than G, the identity of the place is denied and it is determined that the person cannot be authenticated. Further, when the second determination method is used, the facial expression has changed but the background has not changed (while E remains, while FIG. 10C shows the information acquired at the first timing). ) Is confirmed, the identity of the place is denied, and it is processed as being unable to authenticate the person.

  As described above, by using the moving image as the background image, it is possible to provide a highly confidential background password by combining the background that changes along the time axis and the time t until the second timing. In particular, even if the background moving image used for authentication leaks, the safety can be maintained by changing the time t.

  FIG. 11 shows a processing flow when the authentication determination is performed using an image in which a still image and a moving image are combined as a background image. Here, the information P5 (the image captured before the facial expression change) acquired at the first timing and the information P6 (the image captured after the facial expression change) acquired at the second timing are the first and second information. The element extraction processing divides the facial expression, still image, and moving image into three elements. After that, three determinations of facial expression determination, background still image determination, and background moving image determination are performed on the three pieces of image information extracted as each element. In this example, the condition for determining the person in the authentication determination is when the facial expression determination, the background still image determination, and the background moving image determination all output a match detection signal.

  For example, with respect to the image before expression change (combination of expression A, background still image C, and background video E) shown in FIGS. 12A and 12C, the image after expression change shown in FIG. Assuming that the facial expression B is changed and the background still image C is still changed to the background moving image G), the image (d) is the same as the image (d) where the moving image E is not changed. It is possible to easily detect that the image was not captured (or is a forged one). Of course, if any one or both of the background still image C and the background moving image G are detected as inconsistent in the image after the facial expression change, it is possible to easily confirm that the applicant at the shooting location is not the person himself / herself. As described above, it is apparent that the impersonation prevention effect is further improved by using both the still image and the moving image in combination as the background determination password. Furthermore, by arbitrarily changing the time t, the confidentiality of the password can be significantly improved, and an extremely reliable authentication technique can be provided.

  In the above description, the technique of utilizing the background image that can be photographed at the same time as the applicant's photographing as a password for location determination has been described. It is easy for those skilled in the art that even environmental information that can be collected using well-known sensors such as ambient sound (music), vibration, atmospheric pressure, shooting position, etc. can be used as a password like an image. It is conceivable.

  In addition, according to the new idea of the present invention that the authentication place is not regarded as a simple static space for confirming the identity of the applicant, but as a dynamic space that is temporally connected, the facial expression and environment of the applicant. Providing an authentication technology that is superior in accuracy and confidentiality and expandable by dynamically changing one or both of them with a temporal relationship. Is possible. Hereinafter, application examples regarding changes in the facial expression and the environment will be described with reference to FIGS. 13 to 16.

  FIG. 13 shows three typical patterns as an application example of the present invention. The S1 pattern is a case where the environment is changed without changing the facial expression, the S2 pattern is a case where the environment is changed without changing the environment, and the S3 pattern is a case where both the facial expression and the environment are changed. Each one is shown.

  As the environmental information, for example, frequency information such as natural sound (or music) or vibration existing at the authentication place, quantifiable information such as atmospheric pressure, temperature, humidity, GPS, gyro sensor, etc. Information that can be detected mechanically, electrically, magnetically, acoustically, or chemically can be used, such as position information about latitude and longitude that can be collected. Furthermore, here, information obtained naturally from the environmental space without intentional (artificial) processing of these information is referred to as "no change" information, and intentional processing (for example, changing the sound Information that is dynamically changed by applying special vibration or changing the standing position for shooting is referred to as "changed" information.

  Next, the temporal matching between the photographed applicant image and the collected environmental information will be described. When using the background image at the time of shooting by the applicant as environmental information, the person and the background exist in the same screen, but the above-mentioned sounds, vibrations, atmospheric pressure, etc. are collected separately from the image. In this case, it is necessary to make the applicant's image information and environmental information have temporal relevance (matching). In order to provide temporal relevance, for example, environmental information is continuously detected at a predetermined timing (● on the time axis in FIG. 13) and transmitted to the receiving unit of the authentication center, and synchronized with this predetermined timing. By arbitrarily assigning the two timing points (T1 and T2) as the first and second photographing timings for photographing the applicant, it is possible to synchronize the photographed image information and the environment information. Furthermore, the environment information can be changed by changing the interval of time t arbitrarily (change of password).

  Now, in the case of actually authenticating the applicant, in the pattern S1 of FIG. 13, since the environment is dynamically changed, the applicant need only stand in front of the photographing camera (30 in FIG. 1). The camera 30 shoots the face image of the applicant at two arbitrarily designated timings (T1, T2) in synchronization with the acquisition timing of the environment information, and sends it to the authentication center as facial expression information F1 and F2. Further, as the environment information, the information K1 and K2 at the same timing are used as the authentication password. In this way, the “cue” for the applicant as shown in FIG. 2 becomes unnecessary.

  The pattern S2 in FIG. 13 shows an example in which the facial expression is changed without changing the environment, and it can be easily understood that the same effect as the effect described in the first embodiment can be obtained. In particular, it is excellent in that it is possible to confirm the survival of the applicant by the change in facial expression, and it is possible to confirm at the same time whether or not the survival place matches the authentication place.

  Further, by changing both the facial expression and the environment as in the pattern S3 of FIG. 13, it is possible to confirm the person's existence and the existence place (authentication place) as well as the person's confirmation. Moreover, in addition to changing both facial expression and background, it is possible to change the time, so the confidentiality as a password can be significantly improved. Can be built. Therefore, it is suitable for the immigration control of airports that require particularly strict security and the security authentication system in the off-limits area for those concerned.

  Here, an example in which the authentication system of the present invention is applied to immigration inspection of an airport that requires a particularly high level of authentication will be described with reference to FIGS. FIG. 14 is a block diagram showing an authentication function required in this authentication system. A camera for photographing immigrants installed at the examination place as the photographing unit 30, and a sound (song), vibration, atmospheric pressure, position, etc. obtained from the environment around the examination place as a camera for photographing the immigrants, and an environmental information collection sensor 50/60. It is possible to use a sensor having a function of sensing the environmental information. The timing of photographing the immigrant for examination is determined by the photographing instruction signal c1 from the timing control unit 141 of the authentication center 40. The captured image information F is transmitted to the authentication center 40 via the Internet line or the like. On the other hand, the environmental information K detected by the sensor 50/60 is transmitted to the authentication center at a predetermined time interval, but the environmental information may be transmitted in synchronization with the photographing instruction signal c1. The image information F and the environment information K transmitted to the authentication center 40 are stored and stored in a server (not shown) of the image and environment information receiving unit 142. The photographing for confirming the entry of the person is performed twice at a predetermined time interval (time t) as in the first to fourth embodiments.

  The information recorded in the image / environment information receiving unit 142 is sent to the element extracting unit 143 in response to the control signal c2 from the timing control unit 141, and extracted as the personal information F and the environmental information K, respectively. It is transferred to the determination unit 144 and the location identity determination unit 145. The authentication determination unit 146 makes a final determination according to the results of the person identity determination and the environment identity determination, and the identity of the immigrant is confirmed when these two determination results match. In the person determination (and in some cases, the environment determination), the person image of the immigrant is checked against the person information in the registration database (registration DB) 70 in which the person image of the immigrant is registered in advance. At the same time, the environment determination may be performed to confirm the examination shooting location.

  Next, the procedure of the authentication process in the authentication center 40 will be described with reference to FIG. Here, the procedure for performing the person determination and the environment determination in parallel will be described as an example, but these determinations may be performed in series or in combination.

  Based on the photographing instruction from the timing control unit 141, the two images photographed at intervals of time t are acquired as person images F1 and F2, and person collation processing is performed. In this processing, the person image of the immigrant who has been registered in the registration DB 70 in advance (for example, the face image of the passport) and the person image captured at the first time (first timing) are collated as preprocessing. At this stage, if an image inconsistency is detected, the action for refusal of immigration is taken. On the other hand, the environment identity determination processing is executed using the two environment information K1 and K2 extracted in synchronization with the shooting timing in response to the environment information reception instruction. Location verification may be performed as to whether or not the environmental information acquired in synchronization with the shooting of No. 1 matches the environmental information registered in advance as the environmental information of the examination location. In this collation, if it is determined that the locations do not match, the action of refusing entry is taken. However, since it is possible to confirm the location by determining the environmental identity using the two pieces of environmental information (K1, K2) that follow, it is possible to omit the preprocessing for location verification using the registration database. Is.

  In the person identity determination process that is performed subsequently, as shown in the timing chart of FIG. 16, the collation process of the captured images (person images) F1 and F2 acquired at the first timing T1 and the second timing T2 is performed. Done. In this processing, if there is no change in the facial expression of the person, the identities of the two images are determined, and if there is a change in the facial expression, the similarity between the two is determined. Further, in the place identity determination process, as shown in FIG. 16, the collation process of the environment information K1 and K2 extracted in synchronization with the first timing T1 and the second timing T2 is performed to If there is no change, the two are the same, and if there is a change in the environment, the environment information (which should be registered in the registration DB in advance) after the time t from the timing T1 and the actually acquired environment The matching process with the information K2 is performed, and the identity of the place is determined. In response to the result of the above determination process, the authentication determination unit 146 in FIG. 14 finishes the identity verification of the immigrant at the time when both the identity of the person and the identity of the place are confirmed, and the immigration permit is processed. Taken. However, in all other cases, immigration measures will be taken.

  In this way, by confirming the person and the place based on the temporal connection (relevance) between the two person information and the environment information acquired at the time interval t, it is possible to identify the photographed person. It is possible to determine the identity of the sex and the shooting location. In this case, it will be easily understood that the environmental information serves as the password described above. Furthermore, by making the two pieces of information acquired at the timings T1 and T2 have a temporal connection, it is possible to confirm the existence of the person at the same time.

  Here, when using the background image of an image taken for immigration authentication as environmental information, this background image can be used as a password for authentication, but information such as sound and vibration that is independent of the captured image When using, it is necessary to associate the relationship between the environmental information and the captured image. For this purpose, an image in which the facial expression of the photographer is changed may be used, or a background image at the time of photographing and environment information may have a specific relationship. Especially in the latter case, the environment information and the captured image can be linked by making the background image a moving image or changing the environment information according to the pattern of the background image.

  As is clear from the above description, the present invention can be applied to any place and scene where the authentication of the person is required. In particular, it is effective when it is necessary to authenticate the person in a remote place (for example, settlement in Internet mail order, automatic accounting processing in hospitals, entrance / exit management in parking lots, etc.).

10 Applicant 20 Service Execution Section 30 Camera for Photography 40 Authentication Center 40A Authentication Device 41 Imaging Information Receiving Section 42 Element Extraction Section 43 Facial Expression Determining Section 44 Background Determining Section 45 Authentication Determining Section 431 Applicant Confirming Section 432 Facial Expression Change Determining Section 441 Background Confirmation unit 442 Background change determination unit 50 Sound collection microphone 60 Sensor 70 Registration database 141 Timing control unit 142 Image and environment information reception unit 143 Element extraction unit 144 Person identity determination unit 145 Location identity determination unit 146 Authentication determination unit

Claims (18)

A first image of the applicant's facial expression and its background taken at the first timing, and a second image of the applicant's facial expression changed and its background taken at the second timing. An image information receiving unit for receiving,
An element extraction unit that extracts information about facial expressions and information about backgrounds from the first image and the second image;
A facial expression determination unit that determines the identity of the applicant based on the facial expression information extracted from the first image and the facial expression information extracted from the second image;
A background determination unit that determines the identity of the shooting location based on the background information extracted from the first image and the background information extracted from the second image;
An authentication determination unit that confirms the identity of the applicant in response to the determination results of the facial expression determination unit and the background determination unit,
The applicant authentication apparatus, wherein the background determination unit uses time information from the first timing to the second timing as a determination requirement.   The applicant authentication apparatus according to claim 1, wherein the background is a still image.   The applicant authentication according to claim 2, wherein the background determination unit determines the identity of the shooting location by collating the background information in the first image with the background information in the second image. apparatus.   The applicant authentication apparatus according to claim 1, wherein the background is a moving image.   The background determination unit compares the background information extracted from the second image with moving image information that appears with the lapse of time from the first timing to the second timing, and identifies the same shooting location. The applicant authentication apparatus according to claim 4, wherein the applicant authentication apparatus determines the sex.   By confirming that the facial expression information and the background information extracted from the second image both change with respect to the facial expression information and the background information extracted from the first image, the identity of the shooting location can be confirmed. The applicant authentication apparatus according to claim 4, wherein the determination is made.   The applicant authentication apparatus according to claim 1, wherein the background includes a still image and a moving image.   The applicant authentication apparatus according to claim 7, wherein the background determination unit determines the identity of a shooting location by combining a background of a still image and a background of a moving image.   9. The applicant authentication apparatus according to claim 1, wherein when the background determination unit cannot confirm the identity of the shooting location, the determination by the facial expression determination unit is ignored and authentication is not performed. An image of the facial expression of the applicant and environmental information at the time of the imaging are collected at a first timing, the facial expression of the applicant is changed, and the image of the changed facial expression and the environmental information at the time of the imaging are collected. An information collecting unit that collects at the second timing,
A facial expression determination unit that determines the identity of the applicant based on the relationship between the image acquired at the first timing and the image acquired at the second timing;
An environment determination unit that determines the identity of the location of the applicant by comparing the environment information collected at the first timing with the environment information collected at the second timing as dynamic information along the time axis. When,
An applicant authentication apparatus, comprising: an authentication determination unit that receives the determination results of the facial expression determination unit and the environment determination unit and determines the authentication of the applicant.   11. The applicant according to claim 10, wherein the environmental information is information that can be detected by a sensor, such as sound information, vibration information, or atmospheric pressure information obtained from the environment in which the applicant is photographed. Authentication device. In the authentication method that confirms the identity of the applicant based on the image information of the facial expression of the applicant and the background of the applicant at the designated place
A first image of the applicant's facial expression and its background taken at the first timing, and a second image of the applicant's facial expression and its background taken at the second timing by changing the facial expression of the applicant. A first step of receiving
A second step of extracting information about a facial expression and information about a background from the first image and the second image, respectively;
A facial expression determination step of determining the identity of the applicant based on the correlation between the facial expression information extracted from the first image and the facial expression information extracted from the second image;
The background information extracted from the first image and the background information extracted from the second image are associated with the passage of time from the first timing to the second timing to identify the shooting location. A background determination step of determining,
An applicant authentication method, characterized in that the applicant is authenticated based on the determination results in the facial expression determination step and the background determination step.

  When both the confirmation of the identity of the shooting location in the background determination step and the confirmation of the identity of the applicant in the facial expression determination step are established, the applicant is authenticated as the person himself / herself. The applicant authentication method according to claim 12. A first step of collecting information regarding the facial expression of the applicant photographed at the first timing and information regarding the environment at the time of the photographing;
A second step of collecting information about the facial expression of the applicant photographed at a second timing following the first timing and information about the environment at the second timing;
A third step of determining the identity of the applicant based on the facial expression information collected in the first step and the second step;
A fourth step of determining the identity of the applicant's location by collating the information about the environment collected at the first timing and the second timing;
A fifth step of making an authentication determination of the applicant in response to the determination results obtained in the third and fourth steps, the applicant authentication method.   15. The applicant authentication method according to claim 14, wherein the time from the first timing to the second timing is arbitrarily changed.   15. The applicant authentication method according to claim 14, wherein information that can be detected by a sensor from a place where an image is taken is used as the information about the environment. An image regarding the facial expression of the applicant and information regarding the environment of the place where the applicant is present are collected at a first timing, and an image regarding the facial expression of the applicant and environmental information regarding the place are collected at a second timing following the first timing. An information collection unit that collects and
The information collected by the information collecting unit is received, image information about the facial expression and environment information about the place are extracted from the received information, and the first timing and the information are extracted based on the extracted information about the two facial expressions. Whether the applicant exists in the same place based on the first determination that determines whether the images collected at the second timing are images of the same person and the extracted information about the two environments An information determination unit that performs a second determination that determines
Based on the determination result by the information determination unit, the applicant is determined to be the same person, and an authentication determination unit for instructing authentication permission when it is determined that the applicant's place is the same place,
And a service execution unit that executes a security service to the applicant in response to an instruction from the authentication determination unit.   18. The security authentication system according to claim 17, wherein the time from the first timing to the second timing can be changed.