JP2002288670A - Personal authentication device using facial image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a personal authentication device for automatically detecting the direction of a user's face as a subject, and executing the face recognition in accordance with the direction of the face to execute the personal authentication. SOLUTION: This authentication device is provided with one or plural cameras for imaging the user; a lighting means mounted to illuminate the user's face; a detecting means for detecting the direction of the user's face, on the basis of the image output from the camera; a means for sorting the face images into the front faces and the face images excluding the front faces, a database of the face image for checking, created by imaging the images of the user from the front; and an identifying means for collating the front face image sorted by the sorting means with the face image in the database of the face image for collating, and identifying the user. According to this device, only the facial image facing the front can be selected from the user's face image photographed by the camera for checking.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a human interface in general, and more particularly, to a personal authentication system using image recognition.

[0002]

2. Description of the Related Art A personal authentication system for identifying an individual can be used in various situations requiring security. One of the recognition methods used in such an individual authentication system is a method that uses face recognition for identifying an individual from a human face image.

[0003] In face recognition using a human face image, a method of marking a face or manually extracting features before executing face recognition has been proposed. In recent years, however, a completely non-contact method has been proposed. A fully automatic face recognition method has been proposed. Many of such fully non-contact, fully automatic face recognition methods normalize a face image after feature extraction by template matching or the like, and match the normalized image with a face database. Algorithms that can be used for matching with the face database include normalized cross-correlation, eigenspace method, neural network (NN), etc., and personal authentication is performed based on the result of such an algorithm.

In the conventional face recognition method, since the matching is performed based on face data (a face image, a face template, etc.) stored in advance, the position, orientation, and inclination of the face different from the stored face data are determined. There are drawbacks that are very weak in such changes. For example, when the face data stored in the face database used for face recognition is a face template facing the front, a face image in a state different from the face template (for example, a face image facing obliquely) is accurately detected. It is difficult to recognize. Therefore, the face recognition system of the related art generally performs face image normalization processing before recognition processing in order to deal with face images in different states.

When this normalization processing is executed, the face of the input image is converted into a predetermined face data format (predetermined face size, face position, face inclination, and image brightness). Although the rate is improved, the face recognition of the prior art basically does not have a function of detecting the human face direction itself in the input image, so that in the case of a face image in a horizontal direction, a decrease in the recognition rate is inevitable. . Therefore, when performing face recognition of the related art, the target user must always be conscious of facing the camera in a predetermined direction, which increases the burden on the user.

Conventional techniques for detecting the direction of a human face from an input image include JP-A-11-63927 and JP-A-2000-113186. Matsumoto et al., "Development of Face / Gaze Measurement System and Application to Motion Recognition" (5th Robotics Symposia, 2000
/ 3/26, 27) discloses a system that detects not only a human face direction but also a gaze direction from an input image and follows the face direction and the gaze direction in real time from the input image.

[0007]

However, all of these conventional methods detect the direction of the face or the direction of the line of sight of a person, and face recognition for identifying an individual person has not been realized.

Accordingly, an object of the present invention is to provide a personal authentication apparatus for automatically detecting the direction of a target user's face, performing face recognition according to the face direction, and performing personal authentication. And

[0009]

In order to solve the above-mentioned problems, an authentication device according to the present invention comprises one or more cameras for photographing a user, lighting means installed to irradiate the user's face, Detecting means for detecting the orientation of the user's face from the image output of the camera, and means for selecting the face image of the image output into a front image and a face image other than the front, according to the detected face direction, A database of a face image for comparison photographing the user from the front, and an identification unit for identifying the user by comparing the face image facing the front selected by the selection unit with the face image in the database of the face image for comparison. Is provided.

According to the present invention, it is possible to select and collate only a face image facing the front from a user's face image photographed by the camera. As a result, individual persons can be identified with higher accuracy as compared with the personal authentication device of the related art.

According to one embodiment of the present invention, the authentication device converts the detected face image of the user into a face image having the same size and position as the face image stored in the database of the face images for comparison. With normalizing means to adjust,
The identification unit is configured to identify the user by comparing the normalized image with a face image in the database of face images for comparison.

According to this aspect, the face image can be normalized according to the position and size of the face, and the face image photographed by the camera is stored in the face image database for comparison. Since the size and the position are the same, the recognition accuracy of the personal authentication device can be improved.

According to one embodiment of the present invention, the authentication device comprises: an extracting unit for extracting a predetermined image region from the image normalized by the normalizing unit; and a reference data for the predetermined image region. It is configured to include a success / failure judgment database to be stored, and success / failure judgment means for comparing the extracted image area with the reference data and judging success / failure of normalization by the normalization means.

According to this aspect, it is possible to confirm whether or not the normalized image has been correctly normalized by the normalizing means. Therefore, it is possible to reduce errors in the identification means included in the personal authentication device, and to provide a personal authentication device that operates stably.

According to one aspect of the present invention, the authentication device includes a database of face images for matching, which is obtained by photographing the user from each of a plurality of face directions, for each face direction, and wherein the identification means includes: It is configured to match the face image for matching for each face direction according to the detected face direction with the face image of the user facing sideways to identify the user.

According to this embodiment, personal authentication can be performed according to the face direction of the face image captured by the camera. Therefore, it is possible to provide a personal authentication device that performs personal authentication without the user being aware of the face orientation.

According to one embodiment of the present invention, the face direction detecting means of the personal authentication device converts a predetermined reference image of one or more facial features and three-dimensional coordinates corresponding to the reference image. An image corresponding to the reference image of the feature portion from the image output of the camera using the reference image of the feature portion as a template, including the stored feature point database; Extracting means for extracting a region, means for obtaining three-dimensional coordinates of the extracted image region based on the extracted image region, and corresponding to the three-dimensional coordinates of the obtained image region and the reference image Means for detecting the orientation of the user's face from three-dimensional coordinates.

According to this embodiment, the face direction detecting means comprises:
It is possible to detect the user's face direction in a dimension,
Accordingly, the personal authentication device can perform normalization and identification of the user's face image based on the three-dimensional face direction.

According to one embodiment of the present invention, the illuminating means of the personal authentication device is configured to irradiate the user's face with infrared light.

According to this embodiment, by using infrared light as illumination, an image having less influence on illumination fluctuation around the user can be obtained as compared with an image obtained using visible light. Further, the pupil of the user can be clearly extracted.

According to one aspect of the present invention, the identification means of the authentication device determines whether or not a face image corresponding to the face image from the image output exists in the database of the face images for comparison. Means for determining that the user is a registered user in response to the presence of a face image corresponding to the face image from the image output in the matching database, and corresponding to the face image from the image output In response to the face image not being present in the matching database,
It is configured to determine the user as an unregistered user.

According to this embodiment, it is determined whether the user has been registered or not registered in the database, and it is possible to execute different processes according to the attribute of the user.

According to one embodiment of the present invention, the authentication device includes an environment setting database storing a plurality of environment setting data associated with each of the registered users, and an environment of a vehicle according to the environment setting data. Environment setting means for setting, when the user is determined to be a registered user, permitting driving of a car, reading environment setting data on the registered user from the environment setting database, And setting the environment of the vehicle in accordance with the environment setting data.

According to this embodiment, when the user is a registered user, it is possible to automatically adjust environmental settings such as the angle of each mirror and the seat position according to each user.

According to one embodiment of the present invention, the authentication device includes authentication means for authenticating identification means for certifying the identity of the user, and when the user is determined to be an unregistered user, the identification means Requesting the user to present a means, and in response to the authentication means being authenticated by the identification means, giving the user permission to use the car, and the identification means not being presented by the user. And the user is prohibited from using the vehicle in response to the identification means not being authenticated by the authentication means.

According to this embodiment, it is possible to manage an unregistered user by identification means such as a key, a card key, and a password. The personal authentication device of the present invention can prohibit the use of a car unless these identification means are correctly presented by the user.

[0027]

Embodiments of the present invention will now be described with reference to the drawings. FIG. 1 shows one embodiment of a motor vehicle equipped with a personal identification device according to the present invention. The car in FIG.
An image input unit 1, a side mirror 2, a room mirror 3, a control device 5, and an infrared irradiation unit 6 are provided. In this embodiment,
A face image of the driver in the driver's seat is captured by the image input unit 1, and the driver in the driver's seat is recognized from the captured image. If the recognized driver is a person registered in advance in the database of the vehicle, the control device 5 refers to the environment setting (mirror angle, driving seat position, etc.) corresponding to the person, and sets the actuator to the actuator. The side mirror 2, the room mirror 3, etc., which are respectively connected, are adjusted.

FIG. 2 is a functional block diagram of an automobile provided with the personal authentication system according to the present embodiment. The functional block diagram includes an image input unit 1, an infrared irradiation unit 6, a face direction / sight line detection unit 11, a face feature extraction unit 13, and a personal identification unit 1.
5, an environment setting unit 17, a database 19, a side mirror actuator 21, a rearview mirror actuator 23, and a seat actuator 25 are included.

The information of each driver is registered in the database 19 in advance. The registered driver information includes face data of each driver, environment setting information corresponding to each driver, and the like. The face data is used by the personal identification unit 15 for comparison with the input image captured by the image input unit 1. The environment setting information includes the angle of each mirror according to the individual driver,
Includes settings such as position. When the driver in the driver's seat is specified by the personal authentication system according to the present invention, the environment setting unit 17 refers to the registered set value, and sets the side mirror actuator 21 and the rearview mirror for each driver. Actuator 2
3. Control the seat actuator 25.

The vehicle in this embodiment processes two types of operation modes using each functional block shown in FIG. The first operation mode is a face orientation / gaze detection mode executed by the face orientation / gaze detection unit 11. In this mode, the presence of a driver is detected, and the face orientation and the gaze direction are continuously detected. . The second operation mode is a personal authentication mode. In this mode, a driver detected in the face orientation / gaze detection mode is specified, and an environment setting suitable for the driver is executed.

The vehicle shown in FIG. 1 is normally operated in a face orientation / gaze detection mode, and monitors whether the driver is in the driver's seat. When the driver is in the driver's seat, the driver's face orientation and gaze direction are monitored. Is constantly monitored. The vehicle can determine the state of the driver based on the monitored face direction and line-of-sight direction, and can execute processing according to the state.

The personal authentication mode is started at the same time as the driver gets into the vehicle. When the personal authentication mode is activated, the automobile performs face recognition based on the face orientation obtained in the face orientation / gaze detection mode, and identifies the driver in the driver's seat. If a driver is identified,
The environment setting (adjustment of mirror angle and seat position) according to the driver is executed. If the driver is not identified by face recognition, it is highly likely that the person in the driver's seat is an unregistered driver in the database 19, so a sequence corresponding to an unregistered person is processed.

In both operation modes, an image captured by the image input unit 1 is used. The image input unit 1 and the infrared irradiation unit 6 used in this embodiment will be described below with reference to the drawings.

In this embodiment, the infrared irradiation unit 6 is used in order to reduce the deterioration of the image due to the illumination fluctuation in the vehicle.
Therefore, the infrared irradiator 6 is installed on the front of the driver so as to irradiate near-infrared light to the driver's face.

The first reason for using near-infrared light as illumination is to improve robustness against illumination variations. Generally, the brightness inside a vehicle such as an automobile fluctuates greatly due to environmental changes such as indoors and outdoors, or during the daytime or at night.
Further, when strong visible light from one direction shines on the driver's face, a gradation of a shadow is generated on the driver's face. Such variations in illumination and gradation of shadows significantly deteriorate the accuracy of image recognition. Therefore, in the present embodiment, the near-infrared light is irradiated from the front of the driver by the infrared irradiation unit 6 and an image is taken with the near-infrared light, whereby the gradation of the shadow on the face due to visible light from the surroundings is obtained. To reduce. As a result, this embodiment has an advantage that it is less affected by a change in illumination than an image obtained using visible light, and can improve the accuracy of face recognition.

The second reason for using near-infrared light is that the driver's pupil can be clearly extracted. Since the position of the driver's pupil is used to detect the gaze direction, it is important to clearly image the pupil. Further, in the normalization processing at the time of driver identification, the pupil can be used as reference coordinates, so that clear imaging of the pupil leads to an improvement in the normalization accuracy, that is, an improvement in the recognition accuracy.

FIG. 3 shows an embodiment of the image input unit 1. In this embodiment, a conventional stereo method is used to stereoscopically view an object to be photographed. Therefore, the image input unit 1
Is equipped with two video cameras (41, 42) installed in front of the driver. The two video cameras are fixed at predetermined positions so that the driver's face can be photographed stereoscopically. As described above, since near-infrared light is used as illumination, these video cameras use a visible light blocking filter 4 for blocking visible light having a wavelength other than near-infrared light.
The wavelength band is limited by 3 or the like. Each video camera has a camera control unit (45, 46)
Respectively. Each camera control unit is connected through an external synchronization signal line, and the left and right video cameras are synchronized by the synchronization signal. The personal authentication system of the present invention can process not only two-dimensional object recognition but also three-dimensional object recognition by using an image captured in this way as an input image.

As shown in FIG. 3, an image photographed by the image input unit 1 may be variously processed through a conventional image processing board 47. For example, the image processing board 47 shown in FIG.
Has not only a function as an input port of an NTSC video signal, but also a memory for storing images and a hardware circuit for performing complicated image processing, and can execute general image processing algorithms at high speed. . For example, an image processing algorithm by a hardware circuit includes processes such as an oblique projection mechanism, a Hough transform, a binary image matching filter, and an affine transform (rotation, enlargement, and reduction of an image). Such use of the image processing board 47 allows the image processing board 47 to perform a part of the image processing, thereby alleviating the calculation of the ECU and enabling high-speed image processing. In the personal authentication system of the present invention, the image processing board 47 is controlled by the ECU to process the image photographed by the image input unit 1 to realize the function of each functional block shown in FIG.

The processing in each operation mode will be described below in detail.

As shown in FIG. 2, the face orientation / gaze detection unit 11
Are the initial search unit 27, the face direction detection unit 29, and the gaze detection unit 3
1 and execute the face orientation / gaze detection mode. FIG. 4 and FIG. 5 show an overall flow chart of the face direction / sight line detection mode processed by the face direction / view line detection unit 11. In the face direction / gaze detection mode, the initial search unit 27, the face direction detection unit 29, and the gaze detection unit 31 operate in association with each other, and the face direction and the gaze direction can be determined in real time from the continuously input left and right input images. Can be detected.

FIG. 4 is a flowchart showing the processing of the initial search section 27. The processing of the face direction detection unit 29 is performed in step 1 of FIG.
11 to step 117, and the processing of the visual line detection unit 31 is shown by steps 121 to 125 in FIG. The real-time detection of the face direction and the line-of-sight direction in the face direction / line-of-sight detection mode is realized by a loop process from step 111 to step 125 shown in FIG. FIG.
Is used for the initial initialization immediately after the start of the face orientation / gaze detection mode and for error recovery of the face orientation / gaze direction detection in real time.

The processing of the initial search section 27 will be described with reference to FIG. The initial search unit 27 roughly searches the input image for an image area where a human face is captured. The processing here can also be said to be preprocessing for the face direction detection unit 29. Before the process of the face direction detection unit 29, the initial search unit 27 roughly searches the input image for an area where a face is captured, so that the face direction detection unit 29 performs detailed processing of the face in the input image. Analysis can be performed at high speed.

First, in step 101, the images of the left and right video cameras are input from the image input unit 1. Next, in step 103, a region where a human face is photographed is roughly searched from the entire input image. This is performed by two-dimensional template matching using the initial search template 51 stored in advance.

FIG. 6 shows an example of the template 51 for initial search. The image used for the initial search template 59 is an image obtained by partially cutting out a human face facing the front. This partial image includes characteristic regions of the human face such as eyes, nose, and mouth. Is included as one template. This initial search template 51 has been reduced in resolution in advance in order to increase the processing speed in template matching, and has been converted into a differential image in order to reduce the influence of illumination fluctuation. This template is created from a plurality of samples and stored in advance.

Since the search in step 103 is a two-dimensional template matching, either the right video camera image or the left video camera image is used. In the following, an example of template matching using the image of the right video camera will be described.

In the case of template matching using the image of the right video camera, template matching using the initial search template 51 is performed on the right image, and the matched area in the right image is included in the input image. Face candidates. Using this face candidate as a template, similar template matching is performed on the left image, and the three-dimensional position of the face candidate is obtained from the stereo matching result (step 105).

In step 107, the processing results are evaluated, and it is determined whether a face area has been detected from the input image. If it is determined that a face area has been found from the input image, the process proceeds to the processing of the face direction detection unit 29 (step 109). If no face area is found by the initial search unit 27, a series of processing is repeated using a new input image. The initial search unit 27 uses the image input unit 1
This series of processing is repeated until the photographing is performed.

Next, with reference to FIG. 5, the processing of the face direction detecting section 29 and the line of sight detecting section 31 will be described. As described above, in the face orientation / eye gaze detection mode, the loop from step 111 to step 125 shown in FIG. 5 is repeated in order to detect the driver's face orientation and gaze direction in real time. The face direction detection unit 29 can extract predetermined feature points from the input image, and determine the three-dimensional position of the face and the face direction from those feature points. The line-of-sight detection unit 31 includes the face direction detection unit 2
The gaze direction of the face is detected from the input image based on the feature points and the face direction obtained in step S9.

In step 111, the face direction detection unit 29
A feature point of the face is searched from the input image by template matching. The template used for this search is
3D facial feature point model 5 pre-stored in database
Use the image from # 3. FIG. 7 shows an example of the three-dimensional facial feature point model 53.

3D face feature point model 5 in this embodiment
3 is generated from partial images (81 to 91) in which a characteristic portion of a human face facing the front is locally cut out from the image. For example, as shown in FIG.
Such as the left outer corner 81, the left inner corner 83, the right outer corner 87, the right inner corner 85, the left end 89 of the mouth, the right end 91 of the mouth, etc., are generated by being locally cut out from a prepared face image. . Each of these partial images is the object being photographed within that image (in this example, the left and right outer and outer corners of the eye,
The three-dimensional coordinates representing the three-dimensional positions of both ends of the mouth) are stored in the database. In this specification, a partial image of the face feature area having these three-dimensional coordinates is called a face feature point, and a face model generated from the plurality of face feature points is called a three-dimensional face feature point model 53. . These three-dimensional facial feature point models 53 are generated from a plurality of samples and stored in a database in advance.

In step 111, the face direction detecting section 29
An image area corresponding to the three-dimensional face feature point model 53 is searched from the input image. This is performed by performing two-dimensional matching on the input image using the image of each feature point of the three-dimensional face feature point model 53 as a template. For this matching, either the image of the right video camera or the image of the left video camera may be used, but in this embodiment,
Use the image from the right video camera. As a result of the search in step 111, six images at the left and right inner and outer corners of the face and both ends of the mouth of the face captured in the input image are obtained.

In step 113, the image of each feature point obtained from the search in step 111 is used as a template.
Stereo matching is performed on the left image of the input image. Thus, three-dimensional coordinates of each feature point of the input image corresponding to each feature point of the three-dimensional face feature point model 53 are obtained.

In the case of this embodiment, stereo matching is performed using the images at the left and right corners of the face, the inner corners of the eye, and both ends of the mouth in the searched input image as templates. As a result of this stereo matching, three-dimensional coordinates of the left and right outer and inner corners of the eyes of the driver's face and both ends of the mouth are obtained.
Therefore, the three-dimensional position of the driver's face in the driver's seat can be obtained from the three-dimensional coordinates of the feature points of these input images.

After the three-dimensional coordinates of each feature point of the input image are obtained, it is checked in step 115 whether the extraction of the feature point is successful based on a predetermined criterion. If it is determined in step 115 that each feature point has not been found from the input image, the process returns to the initial search unit 27, and a region where the face is photographed is searched from the new input image.

If it is determined in step 115 that each feature point has been found from the input image, in step 117, three-dimensional model fitting is performed using the three-dimensional face feature point model 53, and the direction of the face is detected. . Hereinafter, the three-dimensional model fitting will be described.

As described above, the previously stored 3
The three-dimensional face feature point model 53 is generated from feature points of a face facing forward. On the other hand, the face captured in the input image does not always face the front. Therefore, the three-dimensional coordinates of each feature point (observation point) of the input image obtained in step 113 are determined by an arbitrary angle and displacement from the three-dimensional coordinates of each feature point of the three-dimensional face feature point model 53 facing the front. It has a gap. Therefore, when the three-dimensional face feature point model 53 facing the front is arbitrarily rotated and displaced, the angle and displacement corresponding to each feature point of the input image correspond to the direction and position of the face in the input image.

When fitting the three-dimensional face feature point model 53 to each feature point of the input image, the fitting error E is represented by the following equation.

(Equation 1) Here, N is the number of feature points, x _i is the three-dimensional coordinates of each feature point in the model, and y _i is the three-dimensional coordinates of each feature point from the input image. ω _i is a weighting coefficient for each feature point, and uses a correlation value in stereo matching when a three-dimensional position of the feature point is obtained from the input image. By utilizing this correlation value, the reliability of each feature point can be considered. The rotation matrix is
R (φ, θ, ψ), and the position vector is t (x,
y, z), which are the variables in this equation.

Therefore, if the rotation matrix R and the position vector t that minimize the fitting error E in the above equation are obtained, the face direction and the face position of the input image can be obtained. This calculation is performed by using a fitting method using a least square method or a virtual spring model, or the like.

In steps 121 to 125 shown in FIG.
The gaze detection unit 31 detects the gaze direction of the face from the input image based on the feature points and the face direction obtained by the face direction detection unit 29. At this time, the eyeball is modeled as a three-dimensional sphere, and the center position of the sphere is set as the rotation center. The three-dimensional eye model uses the three-dimensional relative position of the eyeball with respect to the face model, the radius of the eyeball, and the radius of the iris as parameters. FIG. 8 shows an outline of a method of obtaining the line-of-sight direction.

In step 121, the three-dimensional center position 61 of the eyeball is obtained based on the detection result of the face direction detection unit 29. The offset vector for finding the center position 61 of the eye is the characteristic points (55, 57) at both ends of the left and right eyes.
Is determined in advance as a three-dimensional vector from the midpoint of the straight line connecting the points to the center position 61 of the eye ((1) in FIG. 8).
Therefore, its offset vector, eyeball radius,
The center position 61 of the eyeball can be obtained using the face position and the face direction ((2) in FIG. 8).

The center position 59 of the pupil on the eyeball can be easily obtained by executing the Hough transform in the region at both ends because the positions (55, 57) at both ends of the eye have already been obtained as the feature points of the input image. (Step 123). The gaze direction is obtained as a three-dimensional vector (gaze vector) connecting the center position 61 of the eyeball and the center position 59 of the pupil (step 125).

In this embodiment, since the left and right images are used as the input images, it is possible to obtain the line-of-sight vector for each of the right image and the left image. Furthermore, since the line-of-sight vectors can be obtained for both eyes, a total of four line-of-sight vectors are obtained. In this embodiment, the average vector of the four line-of-sight vectors is used as the line-of-sight direction of the input image.

After the gaze direction of the face in the input image is detected in step 125, the process returns to step 111, and a series of processing is repeated using a new input image. As a result of this repetition, it is possible to continuously follow the driver's face direction, face position, and gaze direction in real time.

Next, the personal authentication mode will be described. In the personal authentication mode, the driver's face detected in the face orientation / line-of-sight detection mode is compared with face data registered in a database to identify the driver. When a driver is specified, the personal authentication mode executes environment setting (mirror angle, seat position, etc.) according to the driver. As shown in the functional block diagram of FIG. 2, the personal authentication mode includes a face feature extraction unit 13 for extracting features of an input image;
The process is performed using a personal identification unit 15 that compares and matches a driver based on the processing result of the face feature extraction unit 13 and an environment setting unit 17 that processes environment settings according to the result of the personal identification unit 15. In the case of the face orientation / eye gaze detection mode, the processing is executed using both the left and right images. In the case of the personal authentication mode, the processing is executed using one of the left and right images.

The face feature extraction unit 13 includes the front face feature extraction unit 3
3. Includes the profile feature extraction unit 35, and performs processing according to the direction of the face of the driver 10 in the input image. Driver 10 photographed in an input image from image input unit 1
Since the face direction has already been detected by the face direction / eye gaze detecting unit 11, the face feature extracting unit 13 can execute processing according to the face direction in the input image. The outline of the process of the face feature extraction unit 13 will be described below.

One of the processes of the face feature extraction unit 13 is a normalization process of the image input from the face direction / sight line detection unit 11. This normalization process converts the face in the input image into a predetermined face data format (predetermined face size, face position, face inclination, and image brightness) in order to improve face recognition accuracy. Here, “face inclination” refers to rotation of a face image. More specifically, the face in the image input by the face feature extraction unit 13 has an arbitrary face orientation and is inclined. For example, as shown in FIG. 9a, the face in the input image is
The user may face the front but tilt in the image, or tilt sideways as shown in FIG. 9c. The normalization process corrects “face inclination” in the image by rotating the input image on the image plane. For example, in the case of FIG.
The inclination of the face in the image can be corrected by rotating the image by the angle α with the intersection of the x-axis and the y-axis as the center of rotation (FIG. 9B). Similarly, in the case of FIG. 9C, the inclination of the face in the image can be corrected by rotating the image by the angle α around the intersection of the x-axis and the y-axis. In this specification, the angle of the face corrected by the rotation on the image plane is referred to as “face inclination”. Therefore, in this specification, even when the face is facing upward as shown in FIG. 9C, it is referred to as "landscape", and when the face is facing downward, it is similarly referred to as "landscape".

Another process of the face feature extraction unit 13 is a process of selecting a face direction in a face image. In this sorting process,
The input image is sorted into “front” and “side” according to the direction of the face. This processing is performed by the face direction / eye gaze detecting unit 1
1 is selected based on the face direction detected. In this specification, the front means a case where the face orientation obtained in the face orientation / gaze detection mode is within a predetermined range. Such a range is determined according to a recognizable range of a face recognition method used in frontal face recognition. Therefore,
Face orientations other than the predetermined range are determined to be sideways.

Further, the face feature extraction unit 13 executes a feature extraction process of the input image for the personal identification unit 15. The feature extraction process executed here varies depending on the face recognition method used in front face recognition or profile face recognition. For example, when the eigenspace method is used for the face recognition method, the normalized input image is vectorized by this feature extraction processing, and a weight vector representing the feature of the input image is calculated from the vector. The weight vector (feature component) obtained here is sent to the personal identification unit 15 and the database 19
Is compared with the face data stored in the device, and the individual driver is specified.

FIG. 10 shows an overall flow of the personal authentication mode. For example, when the personal authentication mode is started by the driver getting into the car, in step 301,
The detection result in the face orientation / eye gaze detection mode is the feature extraction unit 13
Is input to Here, the data input to the feature extraction unit 13 includes an image captured by the image input unit 1 and data on the driver's face direction, face position, and line-of-sight direction in the image. As described above, either the left or right image may be used as the input image.

In step 303, the feature extraction unit 13 executes a branching process according to the face direction. Since the personal authentication system according to the present invention can use different face recognition methods for each face direction, it is possible to execute face recognition with high accuracy.

For example, the personal authentication system can execute face recognition by using the eigenspace method in the case of a face facing front and by matching a three-dimensional model in the case of a face facing sideways. . Therefore, according to the present invention, it is possible to select an optimal face recognition method according to the face direction of an input image.

The face recognition method that can be used in the personal authentication system includes the eigenspace method, template matching, Gabor Wavelet method, and NN. The personal authentication system according to the present invention has already obtained information such as a face direction, a gaze direction, and a facial feature point in step 301. Therefore, if face recognition is executed by properly combining the basic information and the normalization processing, it is possible to identify individual drivers with high accuracy regardless of which face recognition method is used.

In the determination of the face orientation in step 303, the front means that the face orientation obtained in the face orientation / gaze detection mode is within a predetermined range. Such a range is determined according to a recognizable range of a face recognition method used in frontal face recognition. Therefore, it is determined that the face orientation other than the predetermined range is the landscape orientation.

If it is determined in step 303 that the face direction of the input image is facing the front, the front face feature extraction unit 33
Executes a normalization process and a feature extraction process for the front. When it is determined that the face direction of the input image is horizontal, the profile feature extraction unit 33 executes normalization processing and feature extraction processing according to the face direction. According to the determination in step 303, the normalization process, the feature extraction process, and the comparison / matching process are branched.
The feature extraction processing and comparison / collation will be described, and then the horizontal normalization processing, feature extraction processing, and comparison / collation will be described.

At step 305, the front face feature extraction unit 33
Converts an input image into a predetermined face format (face size, face position, face inclination). That is, the size, position, and inclination of the face of the input image are normalized so as to be the same as the size, position, and inclination of the face image stored in the database in advance. Further, the brightness of the entire image is adjusted to a predetermined value.

In the present embodiment, the eigenspace method is used as the face recognition method of the personal identification section 15. The normalization processing in step 305 is also executed as a primary transformation on the image plane. By using the feature points obtained by the face direction and gaze detection mode,
Since the size, position, inclination, and the like of the face in the input image can be easily obtained, this processing can perform normalization with higher precision than in the related art.

In one embodiment of this normalization process for the front, the midpoint of a straight line connecting the centers of the left and right pupils is used as a reference.
Affine transformation is performed so that the left and right pupils are horizontal and the distance is constant. Since the near-infrared image is used in the personal authentication system according to the present invention, the pupil can be clearly extracted, and normalization can be performed with high accuracy.

At step 307, it is checked whether the input image has been correctly normalized. This is performed by cutting out a predetermined area from the normalized image and comparing it with a success / failure determination template stored in a database in advance.

For example, a case will be described in which this success / failure judgment is executed using a template for the right eye. In this case, the success / failure determination template is generated by cutting out a predetermined size from an image in a predetermined face recognition format (face size, face position, face inclination), centering on the right eye. This success / failure determination template is created from a sufficient number of samples and used as a typical eye template. The created success / failure determination template is stored in advance in a success / failure determination database.

If the input image has been correctly normalized in step 305, the eyes in the normalized image should always be present at predetermined positions and in predetermined sizes. Therefore, if the area corresponding to the success / failure determination template is cut out from the normalized image and compared with the success / failure determination template, the success or failure of the normalization can be determined. If the normalization is successful, the correlation value between the extracted image and the success / failure determination template is a high value. If the normalization is not successful, the correlation value is a low value. Therefore, the success or failure of normalization can be determined by setting a threshold value of a correlation value as a determination reference in advance. Here, the right-eye was used to simply explain the success / failure judgment of normalization. However, by using a plurality of characteristic regions as success / failure judgment templates and using these plural success / failure judgment templates, more accurate judgment can be made. The determination of the success or failure of the normalization can be executed.

If it is determined in step 307 that the normalization of the face orientation with respect to the front has been successful, the front face feature extraction unit 33 executes a feature extraction process for comparison and collation by the personal identification unit 15 in step 308. If it is determined in step 307 that normalization of the face orientation with respect to the front has failed, the process returns to step 301, and a series of processes is repeated using a new input image.

As described above, since the face recognition for the front in this embodiment uses the eigenspace method, the feature extraction processing in step 308 and the comparison and collation processing in step 309 are performed in a manner corresponding to the eigenspace method. Will be implemented. The outline of the processing of steps 308 and 309 using the eigenspace method will be described below.

The face images of the front faces of the individual drivers to be recognized are stored in the database in advance.
In the eigenspace method, eigenvectors are generated from face images registered in the database. A face class representing each driver's face on the eigenspace is generated by projecting each driver's face stored in the database 19 onto the eigenspace. In the frontal face recognition using the eigenspace method, the face class and the features of the normalized image are compared and collated, and individual drivers are identified from the result.

Therefore, in step 308, the frontal face feature extraction unit 33 projects the normalized image into the unique face space, and extracts the feature of the normalized image as a weight vector. More specifically, first, the normalized input image is image-vectorized. For example, if the normalized input image is an image composed of n × n pixels, an n ^two- dimensional vector (one row,
N ² columns) are generated. Next, this image vector is projected onto the eigenface space, and a weight vector is generated. For example, when M eigenvectors are stored in the database 19, the image vectors are projected onto an M-dimensional eigenspace,
A weight vector having M vector components is generated. If M is sufficiently small relative to the number of dimensions N ² of the image vector, the weight vector becomes small dimensional vector from the original image vector while having facial features captured in the original image. The weight vector generated in this manner is sent to the personal identification unit 15.

At step 309, the personal identification section 15 compares and compares the weight vector obtained from the normalized image with the face class of each driver. In one embodiment of the comparison and collation, the distance between the face class of each driver and the weight vector in the unique face space is determined, and the driver in the image is determined based on the magnitude of the determined distance.

Next, at step 317, the personal identification unit 15
Determines whether the face class corresponding to the normalized image has been found by comparison and matching (step 309). If the face in the normalized image is a face of a person registered in the database 19, a face class corresponding to the person is found by the comparison and collation in step 309. However, when the face in the normalized image is a face of a person not registered in the database 19, a face class corresponding to the person cannot be found. Therefore, if the corresponding face class is not found in the comparison and collation, the process proceeds to step 321 to execute the sequence for an unregistered person. If the corresponding face class is found in the comparison and collation, the driver in the driver's seat is determined, and the registrant sequence in step 319 is processed. Registered and unregistered sequences
It will be described later in detail.

The face recognition based on the eigenspace method can reduce the amount of calculation according to the number of conversion parameters (weight components) at the time of projection, and thus requires the number of calculations corresponding to the number of pixels of the input image. High-speed processing can be realized as compared with the normalized correlation method.

Next, the normalization process for the horizontal orientation and face recognition will be described. As shown in FIG. 10, the flow of the normalization process and the face recognition process for the horizontal orientation is basically the same as the process for the front (steps 305 to 309). However, since the branching process is performed in step 303 according to the face orientation, the normalization process and the face recognition itself performed on the face facing sideways may use different methods than the process on the front face.

The embodiment of the face recognition for the horizontal orientation can be roughly divided into two types of embodiments, and the normalization process differs accordingly. The first embodiment is an embodiment in which a database is created in advance from images of individual driver's face directions, and the face direction databases are used according to the detected face directions to execute face recognition. is there. In this case, the normalization processing is executed according to the face direction detected from the input image.

For example, as shown in FIG. 9D, the reference point for positioning the face in the normalization processing (ie, the intersection of the x-axis and the y-axis in the figure) varies depending on the face orientation. Therefore, when normalizing a horizontal input image, the reference point for positioning the face must be changed according to the face orientation in the image. For this reason, the profile feature extraction unit 35 executes a normalization process in accordance with the face orientation detected by the face orientation / gaze detection unit 11. In this case, the success or failure of normalization is also executed using a different template depending on the face orientation. The normalization processing and the success / failure determination of the normalization are executed by the same processing as the face image for the front, except that the processing is performed for each face direction. In the comparison and collation for the horizontal orientation, the comparison and collation is executed based on a face direction database of face images photographed for each face direction.

In another embodiment of the face recognition for the horizontal orientation, a three-dimensional face model is generated from an input image, and a normalization process is performed using the three-dimensional face model. In this case, a predetermined feature point is extracted from the input image, and a three-dimensional face model of the input image is generated from the feature point.
The three-dimensional face model generated here is a face model generated in the face orientation when the input image was captured. Therefore, if this face model is rotated according to the face orientation obtained by the face orientation / gaze detection unit 11, a face model of the input image facing forward can be obtained. In this embodiment, generation of a three-dimensional face model from an input image and rotation processing of the model correspond to normalization processing in step 311. Since the rotated normalized image can be processed in the same manner as the face image facing the front, the success / failure determination of the normalization process is performed in the same manner as the success / failure determination of the normalization process of the face facing the front. . The face recognition in this embodiment can be realized by using a conventional face recognition method (for example, an eigenspace method) facing the front.

At step 317 after the comparison and collation for the horizontal orientation is performed, the personal identification unit 15 determines whether or not the corresponding face class has been found by the comparison and collation. Step 319 or step 321 is equivalent to step 317
It is processed according to the judgment of.

Next, an embodiment of the registrant sequence in step 319 will be described. FIG. 11 shows a flowchart of the processing of the registrant sequence. Step 3 as mentioned earlier
If the driver in the driver's seat is determined in 17, the personal identification unit 15 gives the driver in the driver's seat permission to drive the car in step 323. This permission allows the vehicle to be operated by the driver. In step 325, the personal identification unit 15 reads the environment setting data corresponding to the determined driver from the database 19, and sends the data to the environment setting unit 17. Step 3
At 27, the environment setting unit 17 controls each actuator to set the angle of the mirror and the seat position according to the environment setting data.

FIG. 12 is a flowchart showing the processing of the embodiment of the sequence for an unregistered person. Step 3 as mentioned earlier
In 17, if the face of the driver in the driver's seat is a face of an unregistered person not stored in the database, the personal identification unit 1
5 requests the driver to use the car by the manager. Here, the manager is a person who is responsible for the use of the vehicle, such as, for example, the owner of the vehicle and a manager of the vehicle. The administrator has any means of identifying himself to the car. For example, such identification means is a means capable of ensuring security such as a key, a card key, and a password. If an unregistered driver wishes to drive a vehicle according to the invention, the vehicle requires a proof of use by the administrator. In that case, the unregistered driver or administrator must prove the permission to use the vehicle by identification means. If the identification means is not presented to the vehicle, the personal identification unit 15 determines in step 335 that the unregistered driver is not permitted to use the vehicle by the administrator, and prohibits the driver from operating the vehicle. When the identification means is presented to the vehicle, the personal identification unit 15 determines whether the unregistered driver is a driver who wants to newly register. Therefore, if the person in the driver's seat is a new driver who wishes to newly register, the new registration of the driver is executed in step 341 via any means. If the person in the driver's seat does not desire a new registration, it is simply a driver who has been temporarily licensed to use the car by the manager, so step 3
At 39, the operation of the car is permitted with the restriction as a temporary driver.

In the above embodiment, an example of a vehicle equipped with the personal authentication device according to the present invention has been described.
The present invention is not limited to personal authentication in automobiles, and can be applied to other fields.

FIG. 13 shows an application example of the personal authentication device of the present invention to a computer. In FIG. 13, the image input unit 1 is connected to the computer 401 via the image processing board 47 to constitute a personal authentication device. Computer 401
Is a general-purpose computer to which user interfaces such as a monitor 402, a keyboard 403, and a mouse 404 are connected. For example, computers operate using an OS known as "Linux". In this example, when a certain user uses the computer 401, the image input unit 1 connected to the computer 401 automatically captures a face image of the user and performs personal authentication from those images. Only when the personal authentication succeeds, the use of the computer 401 is permitted, and the user can use the computer 401. Such personal authentication can be realized by executing the functional blocks of the face direction / gaze detecting unit 11, the feature extracting unit 13, the personal identifying unit 15 and the like on the computer 401 as described above.

As described above, the present invention is not limited to the above embodiment, but can be applied to various fields.

[0098]

According to the present invention, since the personal authentication device includes a function of detecting the face direction of the user, it is possible to execute face recognition according to the face direction. It is possible to improve the accuracy of personal authentication.

[Brief description of the drawings]

FIG. 1 shows one embodiment of a motor vehicle equipped with a personal identification device according to the invention.

FIG. 2 is a functional block diagram of an automobile including the personal authentication system according to the embodiment.

FIG. 3 is a diagram showing an embodiment of an image input unit.

FIG. 4 is a flowchart of processing of an initial search unit.

FIG. 5 is a flowchart of a process performed by a face direction detection unit and a gaze detection unit;

FIG. 6 is a diagram showing an example of an initial search template.

FIG. 7 is a diagram showing an example of a three-dimensional face feature point model.

FIG. 8 is a schematic diagram of a method of obtaining a line-of-sight direction.

FIG. 9 is a diagram illustrating an example of correcting a face inclination by normalization.

FIG. 10 is an overall flowchart of a personal authentication mode.

FIG. 11 is a flowchart of a registrant sequence process.

FIG. 12 is a flowchart of processing of a sequence for an unregistered person.

FIG. 13 is a diagram illustrating a computer including a personal authentication system according to the present embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image input part 6 Infrared irradiation part 10 User 11 Face direction / gaze detection part 13 Feature extraction part 15 Personal identification part 17 Environment setting part 19 Database

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference) EA43 FA24 FA25 FA67 FA69 HA08 JA09 JA16

Claims (9)

[Claims] 1. An apparatus for authenticating a user, comprising: a plurality of cameras for photographing the user; illuminating means installed to illuminate the user's face; Detecting means for detecting the direction of the face, means for selecting the face image of the image output into a face image other than the front and the front in accordance with the detected face direction, and a face image for comparison photographing the user from the front And the face image facing the front face selected by the selection means is compared with a face image in the database of the face image for comparison.
An authentication device comprising: identification means for identifying the user. 2. A normalizing means for adjusting the detected face image of the user to a face image having the same size and position as the face image stored in the database of the face images for comparison, wherein the identification means comprises: 2. The authentication apparatus according to claim 1, wherein the normalized image is compared with a face image in a database of the face images for comparison to identify the user. 3. An extraction means for extracting a predetermined image area from the image normalized by the normalization means; a success / failure judgment database for storing reference data of the predetermined image area; The authentication device according to claim 2, further comprising: a success / failure determination unit that compares an area with the reference data and determines success / failure of normalization by the normalization unit. 4. A database of face images for comparison of each user's face taken from each of a plurality of face orientations, wherein the identification unit is configured to determine each one of the face orientations according to the detected face orientations of the user. 4. The authentication apparatus according to claim 3, wherein the verification face image is compared with a face image other than the front face of the user to identify the user. 5. The face direction detecting means includes a feature point database storing one or more predetermined reference images of a characteristic portion of a face and three-dimensional coordinates corresponding to the reference image. Dimensional coordinates are respectively associated with the characteristic portions of the face, extracting means for extracting an image region corresponding to the reference image of the characteristic portion from the image output of the camera using the reference image of the characteristic portion as a template, Means for determining three-dimensional coordinates of the extracted image area based on the obtained image area; and determining the orientation of the user's face from the determined three-dimensional coordinates of the image area and the three-dimensional coordinates corresponding to the reference image. The authentication device according to claim 1, further comprising: a detecting unit. 6. The authentication device according to claim 1, wherein the illumination unit irradiates the user's face with infrared light. 7. The face image from the image output, wherein the identification means includes means for determining whether or not a face image corresponding to the face image from the image output exists in the database of the face images for comparison. In response to the presence of a face image corresponding to the user in the comparison database, the user is determined to be a registered user, and a face image corresponding to the face image from the image output does not exist in the comparison database. The authentication device according to claim 1, wherein the user is determined to be an unregistered user in response to the request. 8. The authentication apparatus, wherein: an environment setting database that stores a plurality of environment setting data associated with each of the registered users; environment setting means that sets an environment of a vehicle according to the environment setting data; When the user is determined to be a registered user, driving of a car is permitted, environment setting data regarding the registered user is read from the environment setting database, and according to the environment setting data regarding the registered user. The authentication device according to claim 7, further comprising setting an environment of a vehicle. 9. The authentication device includes authentication means for authenticating identification means for certifying the identity of a user, and when the user is determined to be an unregistered user, presents the identification certification means to the user. Requesting the user to be permitted to use the vehicle in response to the identification means being authenticated by the authentication means, wherein the identification means has not been presented by the user, and 8. The authentication apparatus according to claim 7, wherein use of the vehicle by the user is prohibited in response to the authentication means not being authenticated.