JP4730812B2 - Personal authentication device, personal authentication processing method, program therefor, and recording medium - Google Patents

Description

  The present invention relates to a personal authentication apparatus and processing method using a face image, a program thereof, and a recording medium.

  Conventionally, personal authentication systems have been used in various fields. However, in recent years, various authentication systems using facial images have been proposed due to the lack of mental resistance to authentication. For example, Patent Document 1 proposes that a customer is recognized by a face authentication system and customer information is presented based on the recognition result.

  By the way, in the personal authentication using one still face image, erroneous authentication may be a problem in the case of twins. Therefore, for example, Patent Document 2 proposes to use a combination of two or more of face image authentication, voice authentication, ID number authentication, fingerprint authentication, and the like.

JP 2004-326208 A JP 2000-259828 A

  Although authentication by a combination of different types is effective, since a plurality of different types of operations are involved, the apparatus configuration and processing become complicated. Therefore, it is desirable that the authentication accuracy can be improved by using only the face image, and a method using a moving image instead of a single still image is conceivable as the method. This is because authentication using time-series face images can be expected to be more reliable than authentication using a single face image.

  In the authentication using moving images, for example, it is conceivable that authentication is performed using a registered smiley moving image. However, in the case of moving images, the reproducibility of the observed facial expression itself due to the nature of time-series data. May not be enough. In this case, there is a problem that recognition accuracy may not be improved despite the use of moving images.

  An object of the present invention is to propose an apparatus and processing method for realizing highly accurate personal authentication using time-series moving images, a program therefor, and a recording medium.

  According to the present invention, in order to improve the reproducibility of a moving image of a facial expression, a user speaks a keyword or the like, and authentication is performed using the moving image of the face at that time. In other words, in order to improve the reproducibility of facial expressions, a constraint condition of speech is given to the expression of facial expressions. This is different from the combination with voice authentication as in Patent Document 2.

Specifically, the invention according to claim 1 is a personal authentication device that performs person authentication based on a time-series face image at the time of speech, a means for acquiring a moving image, a means for acquiring sound, Means for acquiring the central time of the section in which each phoneme is pronounced, a frame image (hereinafter referred to as a reference frame) that constitutes a face image immediately before the utterance from the moving image, and each at the time of utterance Means for extracting a plurality of frame images (hereinafter referred to as phoneme frames) constituting the face image at the center of the section where the phoneme is pronounced, and extracting feature points of the reference frame and the face image of each phoneme frame; For each phoneme frame, a means for calculating a displacement of the feature point with respect to the reference frame (hereinafter referred to as a feature point displacement amount) and a feature point displacement amount of all phoneme frames are combined to generate one vector. Registered And having a means for performing person authentication by calculating the similarity between the template vector have.

  Here, the utterance may be prompted by the authentication system for the timing and the number of times, or the utterance target word itself may be designated by the authentication system each time. Alternatively, one or a plurality of words to be spoken are predetermined, and any one of them may be prompted or spoken. The predetermined words are preferably specific keywords that are difficult to forget such as name, address, date of birth, and password.

Therefore, the invention according to claim 2 is characterized in that in the personal authentication device according to claim 1, means for prompting speech is provided when a face image is included in a moving image . The invention according to claim 2 is intended to improve convenience when speaking.

  The invention according to claim 3 is the personal authentication device according to claim 1, characterized in that an object of the speech is unspecified. It is an object of the invention according to claim 3 to make it unnecessary to store a speech object.

  According to a fourth aspect of the present invention, in the personal authentication device according to the first aspect, the utterance target is a specific keyword. The invention according to claim 4 aims to improve the memory performance of the speech object.

  The inventions according to claims 5 to 8 provide a personal authentication processing method corresponding to the personal authentication device according to claims 1 to 4. It is an object of the inventions according to claims 5 to 8 to obtain the effects corresponding to the claims 1 to 4.

  The inventions according to claims 9 and 10 provide a program for causing a computer to execute the personal authentication processing method according to claims 5 to 8 and a recording medium recording the program. The inventions according to claims 9 and 10 are intended to realize a system having a configuration corresponding to each of claims 1 to 4 by a computer by installing the program in the computer.

According to the personal authentication device, personal authentication processing method, program, or recording medium of the present invention, the following operational effects can be obtained.
(1) Since a time-series face image corresponding to each phoneme at the time of speech is acquired and person authentication is performed based on the displacement of the feature points of the time-series face image with respect to the face image immediately before the speech, The expression accuracy of the facial expression is improved, and the human authentication accuracy can be improved.

(2) By providing a means for prompting speech, it is possible to improve convenience when speaking.

(3) By making the speech target unspecified, it is possible to make it unnecessary to store the speech target.

(4) By setting the speech target as a specific keyword, the memory property of the speech target can be improved.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows an apparatus configuration example according to the first embodiment of the present invention. In FIG. 1, a moving image pickup means 10, a face detection means 11, a frame detection means 12, a face feature point displacement calculation means 13, and a similarity evaluation means 14 are connected via a data bus. The voice acquisition means 15 and the voice division means 16 are connected, and the voice acquisition means 15 and the speech promotion means 17 are further connected. Here, each means 11-17 other than the moving image imaging means 10 is generally an internal configuration of a computer.

  FIG. 2 shows the overall processing flow of the first embodiment. The first embodiment will be described in detail below with reference to FIG.

  The moving image capturing means 10 is capable of capturing the user of the authentication system, for example, always capturing images of the entrance (step 101), and the images are input to the face detecting means 10 every moment.

  The face detection means 11 determines whether or not there is a face in the image of one frame. If the face is included, the area (mainly the area from the chin to the forehead) is determined. Cut out (step 102). FIG. 3 shows an example of face detection by the face detection means 11 and shows that an area surrounded by a white frame is detected as a face area.

  The face detection means 11 can be configured by a method described in, for example, “Kotaro Sabe, Kenichi Hidai“ Learning a real-time arbitrary posture face detector using pixel difference features ”, Image Sensing Symposium 2005 Proceedings”. . This is because an arbitrary two pixel difference values in a teacher image (= many face images) are input to a boosting learning device to learn a threshold value, and a predetermined threshold value after learning and an arbitrary input image to be detected are detected. These two pixel difference values are compared to determine whether or not a face image is included.

  When the face detection unit 11 detects a face in the frame, the face detection unit 11 sends a frame image with the region information (for example, the frame information in FIG. 3) to the frame extraction unit 12. A frame image without the face area information is sent to the frame extraction means 12. The frame extraction unit 12 buffers the input image by a predetermined number of frames.

  Further, when detecting the face, the face detecting unit 11 notifies the voice acquiring unit 15 of the detection. Thereby, the voice acquisition means 15 starts operation (step 103). The first embodiment is an example in which the utterance of a specific keyword is used. If the user does not utter within a predetermined time after the operation of the voice acquisition means 15 starts, the voice acquisition means 15 Notification is sent to the promotion means 17, and the speech promotion means 17 makes an announcement such as “Please tell your full name” to the user and prompts the keyword to be spoken. When the voice acquisition unit 15 acquires the user's speech, the voice acquisition unit 15 sends the voice information to the voice division unit 16.

  The voice dividing unit 16 decomposes the input voice information into phonemes using known DP matching (dynamic programming matching) or the like (step 104). Here, phonemes are consonants (k, s, t, u, h, m, y, r, w, g, z, d, b) including vowels (a, i, u, e, o) and muddy sounds. , P,...), Etc., and so on. The DP matching method uses the feature values of all phonemes registered in advance (for example, a phoneme spectrum of about 20 ms) as a template, and expands / contracts the time axis of the input audio information while expanding / decreasing the spectrum of the audio information after expansion / contraction Is calculated, the degree of coincidence with each template is evaluated, and the input speech is recognized as the phoneme of the template with the highest degree of coincidence. Since the spectrum is a vector whose element is the amplitude of each frequency band, the degree of coincidence of the spectrum can be evaluated when the inner product of the vectors is a predetermined value or more. In the first embodiment, since it is assumed that a predetermined keyword is uttered, the spectrum used as a template is determined from the beginning, and the time axis expansion / contraction with the highest matching is obtained.

  The voice dividing unit 16 divides the voice input into phonemes, and then sends the time immediately before the utterance and the center time of the section where each phoneme is pronounced to the frame extracting unit 12 (step 105). The frame detection means 12 extracts the frame at the time to which the face area information is attached from the buffered image (step 106), and sends it to the face feature point displacement calculation means 13. That is, the face feature point displacement calculating means 13 is sent with a frame immediately before the utterance (hereinafter referred to as a reference frame) and a frame corresponding to the number of phonemes (hereinafter referred to as a phoneme frame).

  As shown in FIG. 4, the face feature point displacement calculating means 13 sets a window for detecting eyebrows, eyes, and mouth in the face area of the frame (solid rectangle in FIG. 4) (dotted line in FIG. 4). (Step 107), an edge detection operator such as Laplacian is applied to the pixel values in the window, and binarization is performed with a predetermined threshold as shown in FIG. 5 to obtain an edge line image (Step 108). . The window in this embodiment is defined as having a relative position and size with reference to a rectangle detected as a face area, and the aspect ratio of the face area and the aspect ratio of each window are in a proportional relationship. . Accordingly, if the size of the detected face area itself is large, the window is set to be proportionally larger.

  The line drawing in each window is subjected to isolated point removal processing as necessary, and then orthogonally projected to the x-axis and the Y-axis (the arrows in FIG. 5 indicate the projection, and the bold lines indicate the projection results) ( Step 109) A total of 20 points are detected as left and right eyebrows, left and right eye eyes, and upper and lower left and right end points (circles in FIG. 5) as points at both ends of each projection.

In this way, the face feature point displacement calculating means 13 first detects the coordinates of the 20 feature points with respect to the frame immediately before the utterance (reference frame) and the frame corresponding to each phoneme (phoneme frame). . Here, the end point projected with respect to the x axis has only the x coordinate, and the end point projected with respect to the y axis has only the y coordinate.

Next, the face feature point displacement calculation means 13 calculates the difference in coordinates between the reference frame and the phoneme frame, that is, the movement distance in the x-axis or y-axis direction for each end point, and calculates the difference in the size of the face image area (x The movement distance in the axial direction is normalized by the size of the face area in the x-axis direction, and the movement distance in the y-axis direction is normalized by the size of the face area in the y-axis direction) to calculate the feature point displacement (step 110). This is sent to the similarity evaluation means 14. Twenty feature point movement amounts are calculated for one phoneme frame. Therefore, one utterance is expressed as an array for the number of phonemes of a 20-dimensional vector, and this is sent to the similarity evaluation means 14.

  In the first embodiment, for each individual, the 20-dimensional vector for each phoneme when a specific keyword (for example, consisting of n phonemes) is uttered in advance is registered as a template in the predetermined storage means of the recognition system. Yes. The similarity evaluation unit 14 calculates, for each phoneme spoken by the user, the similarity between the vector calculated by the face feature point displacement calculation unit 13 and the template vector as an inner product (step 111). The sum of squares for the number of n phonemes is calculated (step 112). This “sum of squares of inner product” is evaluated as a similarity index between the expression at the time of speech and the template, and the template having the maximum value is selected. Then, it is determined whether or not the value (similarity) is greater than or equal to a predetermined value (step 113). If the value is greater than or equal to the predetermined value, the user is authenticated as a template registrant (step 114) and is less than or equal to the predetermined value. In the case of the above, it is denied (step 115).

In the first embodiment, since the face image in the state of emitting the same phoneme as the template is extracted, the reproducibility of the face image used for authentication can be improved, and the authentication accuracy can be improved.
The predetermined value is a constant that should be determined by the allowable value of the false authentication rate that causes the authentication device to authenticate a person other than the person as the principal, and the false authentication rate and the threshold value have a negative correlation.

  FIG. 6 shows an apparatus configuration example according to the second embodiment of the present invention. The second embodiment is an embodiment in which the utterance of a specific keyword is not used, that is, the utterance target is unspecified, and the utterance designation means 18 is used as the utterance promotion means 17 in FIG. Is. The other configuration is the same as that of FIG.

  FIG. 7 shows the overall processing flow of the second embodiment. In the following, differences from the first embodiment will be mainly described.

  In the second embodiment, after the face detection by the face detection unit 11, speech specifying means 18 which is a kind of speech promoting means, such as "Hello, and, please saying" to the user, speech A word to be specified is designated (step 204).

  The voice acquisition means 15 acquires the user's utterance and sends the voice information to the voice division means 16. In the second embodiment, the word to be spoken by the user is sent from the utterance designation means 18 to the voice. Since it is input to the dividing means 16, a spectrum to be referred to as a template is determined based on the input, and the time axis expansion / contraction with the highest matching is also required.

As in the first embodiment, the frame extracting unit 12 extracts a frame for each phoneme divided by the audio dividing unit 16 (step 207), and the face feature point displacement calculating unit 13 extracts the feature point displacement with respect to the reference frame. A (feature point displacement vector) is calculated (step 211) and sent to the similarity evaluation means 14. In the second embodiment, as a template, 20-dimensional vectors related to all phonemes are registered in advance in the predetermined storage unit of the authentication system for each user. The similarity evaluation means 14 calculates the inner product of the template vector corresponding to the phoneme of the designated word and the vector calculated by the face feature point displacement calculation means 13 and calculates and evaluates the similarity. This is the same as in the first embodiment (steps 212 to 216).

  In the first and second embodiments, authentication is performed using the sum of squares of the degrees of similarity for each frame. However, in the third embodiment, 20 × consisting of elements of feature amount displacement vectors of all phoneme frames. Assuming one vector of phoneme dimension, template vector of the same dimension registered in the device in advance (if the speech target is a specific keyword, the template is already registered; if the speech target is unspecified, register it The inner product with the feature quantity vector of each phoneme is calculated), and the similarity is calculated and evaluated. FIG. 8 shows the overall processing flow of the third embodiment. Except for steps 310, 311 and 312, the overall processing flow is the same as in the first embodiment.

  The fourth embodiment uses a known framework for recognizing time-series vector data by a so-called Hidden Markov Model (HMM). In this example, it is assumed that the user speaks a specific keyword.

FIG. 9 shows a conceptual diagram of the HMM. The HMM in this example has n states S that are the same as the number of phonemes n constituting the keyword. Each state S has a probability a of transition to another state, and a state transition occurs probabilistically as time advances. The HMM outputs an observable symbol O in each state. In this example, the symbol O is the feature value displacement vectors, O _{1, O} 2, _..., exists only phonemes number as O _n. In FIG. 9, aij indicates the probability of transition from state i to state j, and bi (t) is the probability of outputting output O _{t in} state i. Incidentally, HMM, the initial state has a probability distribution if it was one of _S 1 to S _n.

  In order to use the HMM for recognition and authentication, it is necessary to first learn the HMM for each user of the authentication system. In this example, HMMs corresponding to the number m of users are prepared, and each HMM is learned by associating each user with one to one. The flow of HMM learning processing is shown in FIG.

Are prepared HMM of the number of users (step 400), first, keywords to the speech to each user (step 401), the feature displacement vectors _{O 1,} O 2 used for learning, ... to obtain the _{O n} (Step 402). Then, _{O 1,} O 2 for learning, ..., three parameters aij of HMM as the _{O n} prone, bi (t), estimates the probability distribution of the initial state (step 403). Here, a change in the likelihood of the parameter is reduced by an algorithm based on a known Baum-Welch parameter estimation method (Forward-Backward algorithm), and the parameter is estimated repeatedly until it can be regarded as convergence. Thus, displacement of _O 1 for _learning, O 2, ..., the _{O n} prone HMM parameters are obtained. This is repeated for all users, and the learning ends (step 404).

FIG. 11 shows an example of authentication processing by the HMM. At the time of authentication, after the user speaks (step 501), the facial feature point displacement calculating unit 13 calculates a sequence of feature amount displacement vectors from each phoneme frame (step 502), and the similarity evaluation unit 14 calculates the m pieces. The probability that each HMM outputs the displacement vector sequence is calculated (step 503). The probability is recursively calculated by a known Forward algorithm. Of the m probabilities obtained in this way, the owner of the learning data used for the HMM that calculated the highest one is the person to be authenticated. is there.

  Accordingly, the similarity evaluation means 14 determines whether or not the maximum of the m probabilities is equal to or greater than a predetermined threshold (step 504). Authentication is performed as a person associated with the HMM for which the probability is calculated (step 505). If it is equal to or less than the predetermined threshold value, authentication is impossible (step 506). Here, the predetermined value is a constant that should be determined by the allowable value of the false authentication rate that causes the authentication system to authenticate a person other than the person as the principal, and the false authentication rate and the threshold value have a negative correlation. Authentication using the HMM can improve the authentication accuracy by learning with actual data.

  In the above example, the sequence of displacements of the feature points of the phoneme frame with respect to the reference frame is set as the HMM generation target. It may be a generation target. FIG. 12 shows the flow of authentication processing in this case. Here, step 602 is different from that in FIG. 11 except that step 602 is the same as that in FIG.

  The processing functions of some or all of the means in the apparatus shown in FIGS. 1 and 6 can be configured by a computer program, and the program can be executed using the computer to realize the present invention. Needless to say, the processing procedures shown in FIG. 2, FIG. 7, FIG. 8, and FIG. 10 to FIG. 12 can be configured by a computer program and the program can be executed by the computer. In addition, a computer-readable recording medium such as an FD, MO, ROM, memory card, CD, or the like is stored in the computer. In addition, the program can be recorded and stored on a DVD, a removable disk, etc., and the program can be distributed through a network such as the Internet.

The figure which shows the apparatus structural example of Example 1 of this invention. FIG. 3 is a diagram illustrating a flow of processing according to the first embodiment. The figure which shows the example of face detection. The figure which shows the window setting to a face area | region. The figure which shows the example of the endpoint detection by line drawing of the image in a window, and an orthogonal projection. The figure which shows the apparatus structural example of Example 2 of this invention. FIG. 10 is a diagram illustrating a processing flow of the second embodiment. FIG. 10 is a diagram illustrating a flow of processing according to the third embodiment. The conceptual diagram of the hidden Markov model (HMM) used in Example 4 of this invention. The figure which shows the flow of a learning process of HMM. The figure which shows the flow (the 1) of the authentication process by HMM. The figure which shows the flow (the 2) of the authentication process by HMM.

DESCRIPTION OF SYMBOLS 10 Moving image pick-up means 11 Face detection means 12 Frame extraction means 13 Face feature point displacement calculation means 14 Similarity evaluation means 15 Voice acquisition means 16 Voice division means 17 Speech promotion means 18 Post utterance designation means

Claims (10)

In a personal authentication device that performs person authentication based on time-series face images at the time of speech,
Means for acquiring a moving image;
A means of acquiring audio;
Means for dividing the speech into phonemes and obtaining the central time of the section in which each phoneme is pronounced;
From a moving image, a frame image (hereinafter referred to as a reference frame) that constitutes a face image immediately before the utterance, and a plurality of frame images that constitute a face image at the center of the section where each phoneme at the time of utterance is pronounced ( Means for extracting the phoneme frame),
Means for extracting feature points of the face image of the reference frame and each phoneme frame, and calculating a displacement amount of the feature point with respect to the reference frame (hereinafter referred to as feature point displacement amount) for each phoneme frame;
Means for generating a single vector by combining the feature point displacements of all phoneme frames, calculating the similarity between the vector and a pre-registered template vector, and performing person authentication;
A personal authentication device characterized by comprising:   The personal authentication apparatus according to claim 1, further comprising means for prompting speech when a moving image includes a face image.   The personal authentication device according to claim 1, wherein the utterance target is unspecified.   The personal authentication apparatus according to claim 1, wherein the utterance target is a specific keyword. In a personal authentication method for performing person authentication based on time-series face images at the time of speech,
Acquiring a moving image;
Obtaining audio, and
Dividing the speech into phonemes and obtaining the center time of the section in which each phoneme is pronounced;
From a moving image, a frame image (hereinafter referred to as a reference frame) that constitutes a face image immediately before the utterance, and a plurality of frame images that constitute a face image at the center of the section where each phoneme at the time of utterance is pronounced ( A step of extracting a phoneme frame),
Extracting a feature point of the face image of the reference frame and each phoneme frame, and calculating a displacement amount of the feature point with respect to the reference frame (hereinafter referred to as a feature point displacement amount) for each phoneme frame;
Collecting the feature point displacements of all phoneme frames to generate one vector, calculating the similarity between the vector and a pre-registered template vector, and performing person authentication;
A personal authentication method characterized by comprising:   6. The personal authentication processing method according to claim 5, further comprising a step of prompting speech when a moving image includes a face image.   6. The personal authentication processing method according to claim 5, wherein the utterance target is unspecified.   6. The personal authentication processing method according to claim 5, wherein the utterance target is a specific keyword.   The program for making a computer perform the personal authentication processing method of any one of Claims 5-8.   A recording medium storing a program for causing a computer to execute the personal authentication processing method according to claim 5.

Images