JP5955133B2 - Face image authentication device - Google Patents

Description

  The present invention relates to a face image authentication apparatus that authenticates a person based on a face image registered in advance.

2. Description of the Related Art Conventionally, a face image authentication device that authenticates a person based on a pre-registered face image is used in an access control system. This face image authentication device calculates a similarity between a face area image that shows a person's face extracted from an image of a person and a registered face image registered in advance, and the person is registered in advance. Determine whether you are a registrant.
In such a face image authentication apparatus, an unregistered unregistered person tries to be determined as a registrant by holding up a printed face image of the registrant's face or a registered face image in front of his / her face. It is assumed that they try to cheat.

For example, Patent Document 1 discloses a personal authentication device that authenticates a person using an image of a person who has stopped in front of a camera, and improves security against fraud using a face photograph or a printed material on which a face is printed. A technique for making this happen has been proposed. This personal authentication device includes a camera that captures an image of the person to be authenticated and two lights that illuminate the person to be authenticated from the right side and the left side. When the verification button is pressed by the person to be authenticated, the personal authentication device illuminates only the light that illuminates the person to be authenticated from the right side, images the person to be authenticated, and then lights only the light that illuminates the person to be authenticated from the left side. And image the person to be authenticated. Then, in both of the captured images, if the difference obtained by subtracting the average luminance of the non-illuminated area from the average luminance of the non-illuminated area is equal to or greater than a threshold value, It is determined that the person is a human, otherwise it is determined that the object to be imaged is a photograph or printed matter.
However, the personal authentication device according to Patent Document 1 has a problem that the face of the person to be authenticated must be specified in advance, and the illumination must be controlled on that, which is inconvenient and expensive.

Japanese Patent Laid-Open No. 2003-242491

By the way, a so-called walk-through type face image authentication device for authenticating a person based on an image obtained by photographing a walking person has been developed. The walk-through face image authentication device sequentially captures passages and the like through which a person passes to acquire input images, and tracks a person area in which a person extracted from each input image is shown. Then, for each person area in which the same person is shown, the similarity between the face area image in which the face of the person extracted from the person area is shown and the registered face image registered in advance is calculated. It is determined whether or not the registrant is registered in advance.
In such a walk-through face image authentication device, the position of the person shown in the input image changes as the person walks. Therefore, for each input image, the positional relationship between the person and the lighting, and the position of the person and the camera The shooting conditions are different.
On the other hand, photographing conditions such as illumination direction, distance from the camera to the face, angle, and performance of the camera used are determined for the photograph, and the photographing condition when the face is photographed is fixed. That is, when a photograph showing a face is taken by a camera, the face photographing conditions at the time of the previous photography remain in the photographed face portion.
The inventor found that the face image captured in the photograph had the shooting conditions when the previous photograph was taken, whereas the face image obtained by shooting the actual face has the shooting conditions at the time of actual shooting changed to the face image. I found out that it appears.

  An object of the present invention is a walk-through type face image authentication apparatus, in which face images taken while a person who is trying to authenticate are walking are different in shooting conditions, and in the case of fraud using a face photograph. It is an object of the present invention to provide a face image authentication apparatus that prevents fraud using a face photograph by utilizing the fact that the previous photographing condition remains.

  The present invention for solving such a problem is a face image authentication device for authenticating a person based on a face image of a person shown in an input image obtained by photographing a predetermined photographing region and a registered face image of a person registered in advance. An image acquisition unit that sequentially acquires an input image obtained by capturing a shooting area, a person detection unit that extracts a person area image from the input image, and a person area image extracted from a plurality of sequentially acquired input images A tracking unit that generates sequence data in which the tracked person area image is associated, a face shooting condition calculation unit that calculates a feature indicating a shooting condition at the time of shooting a face from the face area image in the person area image, and a sequence For each data, the fraud determination means for determining whether or not the characteristics indicating the photographing condition of the person area image included in the series data are within a predetermined range that is substantially the same, and the fraud determination means If it is determined that the constant range, providing a face image authentication device having an authentication means for authenticating a fraud.

  In the face image authentication apparatus according to the present invention, it is preferable that the person extracting means extracts a face area image as a person area image.

  In the face image authentication apparatus according to the present invention, it is preferable that the face photographing condition calculating means obtains the light source direction from the face area image and includes at least the light source direction in the feature indicating the photographing condition. In the face image authentication apparatus according to the present invention, it is preferable that the face photographing condition calculation unit obtains a variance of luminance values from the face area image and includes at least the variance in the feature indicating the photographing condition.

  Further, in the face image authentication apparatus according to the present invention, a sample face obtained by directly photographing a person obtained by learning using a sample face area image obtained by directly photographing a person and a sample face area image obtained by photographing a photograph of the face. A storage unit that preliminarily stores a partial space that can discriminate between the region image and the sample face region image in which the photograph was taken is further provided, and the face photographing condition calculation unit is a portion when the face region image is projected onto the partial space. It is preferable to obtain the position in the space and include the position of the partial space in the feature indicating the photographing condition.

  The walk-through type face image authentication device according to the present invention has an effect of being able to appropriately determine the presence or absence of fraud using a face photograph.

(A) And (b) is a schematic diagram showing the example of installation of an imaging part in case the face image authentication apparatus to which this invention is applied is installed in the entrance of a room. (A), (b), and (c) are schematic views showing examples of images when a person who is not implementing a photographic plan is irradiated with illumination light from a specific position, and (d), (e) ) And (f) are schematic diagrams illustrating an example of an image when a person who is executing a photographic plan is irradiated with illumination light from a specific position. It is a schematic block diagram of the face image authentication apparatus to which this invention is applied. It is a figure which shows the processing flow of the face image authentication apparatus to which this invention is applied. It is a figure which shows the processing flow of the face image authentication apparatus to which this invention is applied. (A) And (b) is a figure explaining the calculation method of a light source direction. It is a schematic diagram of a history table. (A), (b), (c), (d), (e), and (f) are figures which show the example of a face area image.

Hereinafter, a face image authentication apparatus according to an embodiment when the present invention is applied to an access control system will be described with reference to the drawings.
The face image authentication device to which the present invention is applied extracts a region in which the same person's face is shown from each of a plurality of input images obtained by sequentially photographing a walking person and uses it as a face region image. For each face area image, the direction of the light source applied to the face of the person shown in the face area image is obtained. The light source direction obtained from the face region image represents the light source direction when an actual face having an uneven surface is photographed. By the way, in the face area image when photographing the face of a person who is actually walking, the shadow appearing in the face area image caused by unevenness such as nose and lips present on the face changes due to the change of the illumination position accompanying walking As a result, the light source direction obtained also changes. On the other hand, in the face area image when a face photograph is taken, the light source direction when the face photograph is taken is the overall light and darkness that appears in the face area image shown in the photograph even if the illumination position changes due to walking. Changes, but the shadow basically does not change, and the direction of the light source obtained thereby does not change basically. That is, the light source direction is one of the characteristics indicating the shooting conditions at the time of face shooting. Therefore, in the face image authentication device, the light source direction, which is one of the characteristics indicating the shooting conditions for the face area image extracted from each input image, must be changed in the face area image tracked as the same person. For example, it is determined that the face photograph is in the face area image. As a result, the face image authenticating apparatus appropriately determines the presence or absence of fraud using a face photograph.

FIG. 1A and FIG. 1B schematically show an installation example of the imaging unit 3 connected to the face image authentication device 1 in order to manage entrance / exit of the room. The state of the shooting area to be performed will be described. As shown in FIGS. 1A and 1B, the face image authentication device 1 manages the entrance and exit of the entrance 100 of the room. In this case, the imaging unit 3 is attached to the wall surface or ceiling near the entrance 100 so that the shooting direction is directed slightly downward and toward the passage side so that the passage leading to the entrance 100 can be taken as a shooting region. As a result, the imaging unit 3 can image a person heading toward the entrance 100 along the traveling direction 101. In addition, a downlight 102 is installed on the ceiling of the passage leading to the entrance 100 so that the irradiation direction is downward.
In FIG. 1A, the imaging unit 3 sequentially photographs the same person 110, 111, 112 heading for the entrance 100 at times (t1), (t1 + 1), and (t1 + 2) at predetermined time intervals. Is shown. On the other hand, in FIG. 1B, the imaging unit 3 sequentially photographs the same person 120, 121, 122 heading for the entrance 100 at times (t2), (t2 + 1), and (t2 + 2) at predetermined time intervals. It shows how to do. These persons 120, 121, and 122 are unregistered persons who are not registered in the face image authentication apparatus 1, and the registered person's face photograph 103 registered in advance in the face image authentication apparatus 1 is placed in front of his / her face. ing. That is, this person shows a state of performing an illegal act in which the imaging unit 3 is photographed in place of his / her face by the imaging unit 3 and authentication is attempted by the registrant's facial photograph 103. ing. Such fraud using a face photograph is hereinafter referred to as a photographic plan.

Hereinafter, a positional relationship between the person walking in the passage and the downlight 102, that is, a temporal change in the light source direction, which is one of the imaging conditions of the imaging unit 3, will be described. In FIG. 1A, when viewed from the imaging unit 3, the person 110 is positioned behind the downlight 102 at the time (t1), and the front of the face is illuminated by the downlight 102. At time (t1 + 1), the person 111 is positioned immediately below the downlight 102 and is irradiated from above the head by the downlight 102. At time (t1 + 2), the person 112 is positioned on the front side of the downlight 102 and is irradiated from behind by the downlight 102.
Similarly, in FIG. 1B, when viewed from the imaging unit 3, at time (t2), the person 120 is positioned behind the downlight 102, and the front face of the registrant's face photograph 103 is illuminated by the downlight 102. Is done. At time (t2 + 1), the person 121 is positioned immediately below the downlight 102 and is irradiated from above by the downlight 102, but the back surface of the photograph 103 is irradiated. At time (t2 + 2), the person 122 is positioned on the front side of the downlight 102 and is illuminated from behind by the downlight 102, and the back surface of the photograph 103 is illuminated.

FIG. 2 shows a person area image obtained by extracting a person area from an input image taken by the imaging unit 3.
2 (a), 2 (b), and 2 (c) are taken when the persons 110, 111, and 112 who have not performed the photography plan in FIG. 1 (a) are irradiated from the downlight 102 at the respective positions. 2 (d), 2 (e), and 2 (f) show examples of person area images extracted from the input image. The figure shows an example of a person area image extracted from an input image photographed when being illuminated from the downlight 102 at a position. In the figure, the shadows due to illumination are indicated by diagonal lines from the upper right to the lower left, and the darker the shadows are, the thicker the diagonal lines are.

FIG. 2A shows a person region image 200 extracted from an input image obtained by photographing the person 110 at time (t1) in FIG. The person area image 200 is an image in which the person 110 is irradiated with the light of the downlight 102 from the upper front, so that the face area is bright and there is little shading due to facial irregularities such as eyebrows and nose.
FIG. 2B shows a person region image 201 extracted from an input image obtained by photographing the person 111 at time (t1 + 1) in FIG. In the person area image 201, the person 111 is irradiated with light from the downlight 102 from substantially above. Since the human face has irregularities and the concave part is not exposed to light from directly above, shadows are generated, so only the forehead, upper cheeks and nose are brightened, and there are shadows near the orbit, from the lower cheeks to the chin Has occurred. Thus, the person area image 201 is an image in which a shadow different from the person area image 201 appears on the face.
FIG. 2C shows a person area image 202 extracted from an input image obtained by photographing the person 112 at time (t1 + 2) in FIG. In the person area image 202, the person 112 is illuminated from behind, the downlight 102 is backlit, and unlike the person area image 201 and the person area image 202, the entire area becomes a shadow of illumination, and an overall shadow is generated.

FIG. 2D shows a person area image extracted from an input image taken with the person 120 holding the registrant's face photograph 103 in front of his / her face at time (t2) in FIG. 1B. 203 is shown. Since the person 120 holds the face photograph 103 in front of his / her face, the person area image 203 is not reflected by the downlight 102 in the face photograph 103 portion. The shadows are taken as they are. FIG. 2 (e) shows a person extracted from an input image taken at a time (t2 + 1) in FIG. 1 (b) with the person 121 holding the registrant's face photograph 103 in front of his / her face. An area image 205 is shown. The person 121 holds the face photograph 103 in front of his / her face, and the person area image 205 is illuminated from directly above the person 121, but the face photograph 103 which is a flat object is uniformly illuminated from behind. As a result, the entire face area 206 is shaded. Although a shadow is generated in the entire face photograph 103, the shadow at the time of photographing is stored as it is. That is, since the face photograph 103 is a flat surface and has no unevenness like a human face, the face region 206 is not shaded by the unevenness of the face as in FIG. Since the face photo 103 is illuminated by the downlight 102 on the back, the face region 206 is not as bright as in FIG. 2D, and the entire photo is slightly dark.
FIG. 2 (f) shows a person extracted from an input image taken at a time (t2 + 2) in FIG. 1 (b) with the person 122 holding the registrant's face photograph 103 in front of his / her face. An area image 207 is shown. Since the person 122 has the face picture 103 in front of his / her face, the face picture 103 is shown in the person area image 207 as a face area 208. In the person area image 207, the face photograph 103 is irradiated with light from behind, the downlight 102 is backlit, and the entire face area 208 is shaded. As in the case of FIG. 2 (e), a shadow is generated in the entire face photograph 103, but the shadow at the time of photographing is stored as it is. In this face area 208, a new shadow due to the unevenness of the face as in the case of FIG. 2B does not occur, and the entire photograph becomes dark.

  In other words, when no photographic scheme is performed, the relative position of the downlight 102 that irradiates the face of the person changes with the movement of the person, that is, the direction of the light source at the time of shooting changes. Therefore, the shading due to the unevenness of the face of the person in the photographed image changes. On the other hand, when a photographic plan is being implemented, the shadow that appears on the face in the face photo that the person raises changes so that the whole image becomes brighter or darker, but the light source when the photograph was taken The shade by is saved as it is. In other words, even if a person moves, the light source direction, which is one of the characteristics indicating the shooting conditions when shooting the face photo, does not change in the face reflected in the face photo that the person raises. The shading caused by the unevenness of the person's face in the face photo is basically unchanged. Therefore, the face image authentication device tracks an area in which the same person is captured from each captured image, and if the light source direction estimated from the shadow on the face of the person in each area does not change, the photograph It can be determined that the plotting is being performed, and if the plan is changed, it can be determined that no photographic plotting is being performed.

  Hereinafter, the face image authentication device 1 will be described in detail with reference to FIG. FIG. 3 is a diagram showing a schematic configuration of the face image authentication apparatus 1 to which the present invention is applied. The face image authentication device 1 includes an image processing unit 10, an output unit 20, and a storage unit 30.

  Here, the imaging unit 3 will be described. The imaging unit 3 is a surveillance camera that captures a predetermined area as shown in FIG. Hereinafter, the predetermined area that the imaging unit 3 captures is referred to as an imaging area. The imaging unit 3 is, for example, a two-dimensional array of photoelectric conversion elements (for example, a CCD sensor, a C-MOS, etc.) that outputs an electrical signal corresponding to the amount of received light, and an image of a shooting area on the photoelectric conversion element. An image forming optical system for forming an image. The imaging unit 3 is installed substantially in front of the person's passing direction so that the face of the person passing through the shooting area can be sequentially shot. And the imaging part 3 image | photographs an imaging | photography area | region and acquires an input image for every predetermined time interval (for example, 200 msec). The input image can be a grayscale or color multi-tone image. In this embodiment, the input image is a color image having 1280 pixels wide × 960 pixels vertically and having a luminance resolution of 8 bits for each of the RGB colors. However, an input image having a resolution and gradation other than this embodiment may be used. The imaging unit 3 is connected to the face image authentication device 1 and passes the acquired input image to the image processing unit 10.

The storage unit 30 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device.
In the storage unit 30, registrant data 31 such as the name, registrant ID, and registered face image of a person permitted to pass through the entrance 100, a history table 32 that is history information of person tracking data extracted from the input image, And a three-dimensional face shape model 33 in which a standard face shape of a person is modeled. Although not shown, the storage unit 30 stores in advance a computer program and various parameters for controlling the face image authentication apparatus 1.

  The output unit 20 is an interface connected to an electric lock, an authentication result display lamp or the like, which is an external connection device, and a control circuit thereof. When the output unit 20 receives a signal indicating successful authentication of the person from the image processing unit 10, the output unit 20 outputs a signal for performing unlocking control on the connected electric lock. Further, when the image processing unit 10 obtains an authentication result of authentication OK, authentication NG, or fraud (photographing plan), the output unit 20 turns on or turns off an authentication result display lamp (not shown) according to the result. Alternatively, the authentication result may be output to the monitoring center device.

The image processing unit 10 is configured by, for example, a so-called computer, and executes various processes on the input image from the imaging unit 3 while referring to the storage unit 30 and outputs the result from the output unit 20 to the outside. Specifically, the image processing unit 10 includes an image acquisition unit 11, a person detection unit 12, a face extraction unit 13, a tracking unit 14, a face photographing condition calculation unit 15, an illegal act determination unit 16, a face collation unit 17, an authentication unit 18, The result output means 19 is comprised.
Each means of the image processing unit 10 is a functional module implemented by a microprocessor, a memory, a peripheral circuit thereof, and software operating on the microprocessor. These means may be integrated by firmware. Moreover, you may comprise some or all of these means with an independent electronic circuit, firmware, a microprocessor, etc. Hereinafter, each unit of the image processing unit 10 will be described in detail.

  The image acquisition unit 11 is an interface for enabling the image processing unit 10 to process an input image captured by the imaging unit 3 and is not necessary when the input image of the imaging unit 3 can be directly captured. . In the present embodiment, since images are sequentially input from the imaging unit 3 in the order of shooting, the input images are converted into image data that can be processed in the order of input and passed to the person detection means 12. In the present embodiment, the input images are output in the order in which the image capturing unit 3 took the images. However, when acquiring the input images from a medium such as a hard disk, a function of acquiring the input images from the medium in the order of shooting is added. You may let them.

  Each time the person detection unit 12 acquires an input image from the image acquisition unit 11, the person detection unit 12 determines whether or not there is a region in which a person is captured from the acquired input image, and if there is, extracts the person region image. . Whether there is an area in which a person is present or not is determined by extracting a change area by a frame difference method and determining whether the area is a person by using the size, aspect ratio, and the like of the change area. There are various human detection methods such as a background subtraction method and the use of a discriminator, and any method may be adopted as the present invention. Here, since the person detection method is not an essential part of the present invention, a detailed description thereof will be omitted.

The face extraction means 13 detects a face area that is an area in which a person's face is reflected from the person area image, cuts it into a rectangular shape so as to include the face area, and extracts a face area image as shown in FIG. . Specifically, the face extracting means 13 performs edge pixel extraction on the person region image using a Sobel filter or the like so that the inclination direction of the luminance change can be known. Then, the face extracting means 13 detects an elliptical edge distribution having a predetermined size and approximating the contour shape of the head from the extracted edge pixels, and the face area is based on the area surrounded by the edge distribution. Extract images.
In this case, the face extraction unit 13 can detect an elliptical edge distribution using, for example, a generalized Hough transform. Since there are various face area image extraction methods, other methods may be employed as appropriate. For example, the face area may be detected using an Adaboost classifier. This method is described in a paper by P. Viola and M. Jones “Rapid Object Detection Using a Boosted Cascade of Simple Features” (Proc. The IEEE International Conference on Computer Vision and Pattern Recognition, vol.1, pp.511-518, 2001). ) Can be referred to.

  By the way, in the present embodiment, the face extraction unit 13 performs a face region extraction process on the person region image extracted by the person detection unit 12, but directly executes a face region extraction process from the input image. May be. Then, the face area image extracted by the face extraction unit 13 may be used as a person area image to perform processing such as tracking processing. In this case, the face extraction unit 13 also serves as the person detection unit 12.

The tracking unit 14 performs tracking processing on the person region image extracted by the person detection unit 12 using a known tracking technique. Then, the person area image tracked as the same person in the input image to be processed sequentially is the person area image 74 of the history table 32, the shooting time of the input image in the imaging unit 3 is the time 72, and the position of the person area on the image Information is stored in the tracking position 73. Details of the series data in the history table 32 will be described later with reference to FIG.
The tracking unit 14 correlates each person area image of the input image of the current frame that is sequentially acquired at a predetermined time interval with the person area image extracted from the input image of the previous frame (the input image one point before). Perform tracking using. For example, for each person area, the distance between the centroid position of the person area image of the current frame and the centroid position of the person area image of the previous frame is obtained, and if the distance is equal to or less than a predetermined threshold, the person area is the same Corresponding as a series of data by a person. If the tracking unit 321 cannot associate the person with the person area, the person area of the current frame is assumed to include a person who has newly entered the shooting area, and the person is recorded in the history table 32. As a series data, a new row of trial number 70 is added.
There are many tracking techniques that use not only the center of gravity position but also various parameters such as color similarity, shape similarity, movement speed, etc., but these are not essential to the present invention and will not be described in detail. Omitted. Note that any method such as an optical flow method, a particle filter method, or the like may be adopted as long as the method can track a person area.

  The face photographing condition calculating means 15 is a means for calculating a feature indicating a photographing condition when a face is photographed from the face area image extracted by the face extracting means 13. In the present embodiment, the light source direction when a face is photographed is estimated from the shadow of the face in the face area image. The “face” here refers to a real face having a three-dimensional shape, and does not mean a photograph.

The face photographing condition calculation unit 15 obtains a light source direction representing a positional relationship between a face shown in the face region image and a light source that gives a shadow to the face with respect to the face region image extracted by the face extraction unit 13. .
First, the face photographing condition calculation unit 15 performs a process such as rotation, enlargement / reduction, and the like on the three-dimensional face shape model 33 stored in the storage unit 30, and a plurality of face shape features included in the three-dimensional face shape model 33. A point and a feature point in the face area image corresponding to the face shape feature point are aligned, and an aligned shape model 600 as shown in FIG. 6 is created.

Next, a method for calculating the light source direction will be described with reference to FIG. FIG. 6A is a conceptual diagram of a situation in which light is applied from the light source 601 to the aligned shape model 600 and is shaded (hatched). Further, the face photographing condition calculation unit 15 generates a shadow image shown in FIG. 6B, which is projected from the aligned shape model 600 with the shadow shown in FIG. 6A onto a two-dimensional planar image.
The face photographing condition calculation unit 15 obtains a shadow image in the aligned shape model 600 caused by the unevenness of the face that is generated when light is irradiated from various light source directions to the aligned shape model 600. Further, for each shadow image, the direction vector to the light source 601 when the shadow is added is stored as the light source direction.
Then, the face photographing condition calculation means 15 compares the face region image extracted by the face extraction means 13 with the shadow image generated for each light source direction in FIG. Is identified. Then, the light source direction of the corresponding registered shape model 600 is stored in the light source direction 75 of the history table 32.

The light source direction is a vector represented by a three-dimensional orthonormal coordinate system (X, Y, Z) of the aligned shape model. As shown in FIG. 6, in this coordinate system, the center of gravity of the aligned shape model 600 is the origin, the horizontal direction of the aligned shape model 600 is the X axis, the vertical direction is the Y axis, and the X axis and the Y axis are orthogonal. The direction to do is the Z axis. The light source direction is expressed as (l _X , l _Y , l _Z ) using the X-axis component l _X , the Y-axis component l _Y and the Z-axis component l _Z. Such a light source direction has a characteristic as a feature indicating a photographing condition when a face is photographed.
Note that the method for estimating the light source direction is not limited to this, and various known methods can be used.

The fraud determination means 16 refers to the light source direction 75 in the same series data in the history table 32 of the storage unit 30 related to the person area image being tracked, and reads all the light source directions stored in the series data. Then, the fraud determination means 16 determines whether or not there are three or more records of the read light source direction, and if there are three or more, the light source direction is within a range of ± 5 degrees (an example of a predetermined range). ) Is further determined. If there are not three or more, the fraud determination means 16 does not determine whether it is within a range of ± 5 degrees. Then, if it is within the predetermined range, it is determined that the photographic plan has been taken, and the effect of the photographic plan is output to the authentication unit 18 to the authentication unit 18. On the other hand, if it is not within the predetermined range, no special processing is performed.
Here, the predetermined range is preferably a range that can be considered as a range of calculation error in the light source direction. This range takes into account the amount of variation in the light source direction in the environment where the image pickup unit 3 is assumed to be installed and the degree of calculation error in the face photographing condition calculation means 15, and from experiments or experience to the extent that these can be distinguished. It is set in advance.
In the present embodiment, it is determined that the number of records in the light source direction is three or more. However, the number of records may be two or more. This is set according to the time constraint and physical constraint necessary for issuing the authentication result.

  The face collating unit 17 collates the face area image extracted by the face extracting unit 13 with all the registered face images stored in the registrant data 31 of the storage unit 30, and calculates the degree of similarity with each registered face image. The highest degree of similarity is stored in the similarity 76 of the history table 32, and the registrant ID of the registered person of the registered face image is stored in the candidate person ID 77 of the history table 32. As a similarity calculation method, facial feature points are extracted from each of the face area image and the registered face image extracted by the face extraction unit 13, and the similarity is calculated based on the degree of coincidence of the positional relationships between corresponding feature points. Adopt the desired method. In addition, any method may be employed as long as the degree of similarity between face images can be calculated.

When the authentication unit 18 receives the determination as a photo measure by the fraud determination unit 16, the authentication unit 18 causes the result output unit 19 to output a warning lamp lighting signal from the output unit and turns off the tracking flag 71 of the history table 32.
The authentication unit 18 refers to the history table 32, calculates the average similarity for each candidate person ID 77 in the series data in which the tracking flag 71 is ON, and the average similarity is an authentication threshold, for example, “0.9”. If it exceeds, it is determined that the user is a registrant of the ID, and the authentication result with authentication OK is stored in the history table 32. If the average similarity is equal to or lower than the authentication threshold value, it is stored in the history table 32 as an authentication result with the authentication NG.
Further, when it is determined that the user is a registrant, the result output means 19 outputs the unlocking signal of the electric lock from the output unit 20 and turns off the tracking flag 71 of the history table 32. On the other hand, if it is not determined that the user is a registrant, it is determined whether a predetermined authentication time has elapsed from the time when the series data in the history table 32 was created. If the authentication time has elapsed, an authentication failure signal is output as the result output means 19. Are output to the output unit 20 and the tracking flag 71 of the history table 32 is turned OFF. If the authentication time has not passed, no special processing is performed.

  Here, the history table 32 stored in the storage unit 30 will be described with reference to FIG. FIG. 7 shows the contents of the history table 32. 7, the history table 32 includes a trial number 70, a tracking flag 71, a time 72, a tracking position 73, a person area image 74, a light source direction 75, a similarity 76, a candidate person ID 77, and an authentication result 78 for each person being tracked. Is a table associated with

The trial number 70 is an identification number for distinguishing the series data of the person being tracked from the series data of other persons, and is tracked by the tracking unit 14 until the person area image first appears in the shooting area and disappears. This is a number for identifying the series data for each person who is marked.
The tracking flag 71 is an area for storing a flag indicating whether or not the tracking of the person being tracked is continued, and turns ON when the tracking unit 14 starts tracking the person for each series of person areas to be tracked. When the tracking is finished, it is set to OFF.
The time 72 is an area for storing the time when the imaging unit 3 acquires the input image obtained by extracting the person area image. It is stored for each person area image to be tracked extracted by the tracking means 14.
The tracking position 73 is an area for storing position coordinates where the person area image is located in the input image, and is stored for each person area image to be tracked.
The person area image 74 is an area for storing a person area image to be tracked. The person area image associated with the person area image one frame before in the tracking process by the tracking means 14 is stored as the same series data.
The light source direction 75 is an area for storing the light source direction calculated from the face area image of the person area image by the face photographing condition calculation means 15.
The similarity 76 is an area for storing the similarity (matching score) having the highest value among the similarities between the face area image calculated by the face matching unit 17 and each of the plurality of registered face images.
The authentication candidate person 77 has registrant data having a registered face image showing the highest similarity among the similarities between the face area image calculated by the face matching unit 17 and each of the plurality of registered face images. This is an area for storing the identification ID (registrant ID) of the registrant in 31.
The authentication result 78 is an area for storing any one of authentication OK, authentication NG, and fraud that is the result of the authentication process performed by the authentication unit 18.

  The operation of the authentication process by the face image authentication apparatus 1 to which the present invention is applied will be described below with reference to the flowcharts shown in FIGS. The flow of operations described below is controlled by a control unit (not shown) that operates on the microprocessor constituting the image processing unit 10 and controls the face image authentication apparatus 1 as a whole. Note that the history table 32 is initialized to be blank before the processing described below is performed. The operation described below is executed every time one input image is acquired.

  First, the image acquisition unit 11 of the face image authentication device 1 acquires an input image obtained by the imaging unit 3 capturing an imaging region (step S801). When the image acquisition unit 11 acquires an input image, the person detection unit 12 extracts a person area image in which a person is shown from the acquired input image (step S802). Next, the person detection means 12 determines whether or not one or more person area images have been detected (step S803). If no person area image is detected, the process proceeds to step S801, and the processes of steps S801 to S803 are repeated until a person area image is detected.

  On the other hand, when one or more person area images are detected in step S803, the tracking unit 14 in the series data in which the tracking flag 71 in the history table 33 is ON for all the detected person area images. Correspondence with the latest record (refers to the record in the row with the latest time 72) is performed (step S804). Then, the tracking unit 14 determines whether there is any series data in which the tracking flag 71 is ON that is not associated with any of the person area images of the current frame (step S805). If there is series data in which the tracking flag 71 is ON that is not associated with any of the person area images of the current frame, the person of the series data has entered the room with authentication success or is not authenticated. It is thought that he was away from the shooting area. Therefore, the tracking unit 14 sets the tracking flag 71 of the sequence data that is not associated with any of the person area images of the current frame to OFF, and thereafter does not perform the tracking process for the sequence data (step S806). ). On the other hand, if there is no image associated with any of the person area images in the current frame, the tracking unit 14 proceeds to step S807.

  Next, the tracking unit 14 determines whether or not all the person area images in the current frame have been associated with the series data for which the tracking flag 71 is ON (step S807). If there is any data that could not be associated with the series data for which the tracking flag 71 is ON, the tracking unit 14 adds the series data for the person area image that could not be associated to the history table 32 as new series data. Add That is, the tracking unit 14 assigns a new trial number 70 to the series data for the person area image that could not be associated. Then, the tracking flag 71 is set to ON, the shooting time of the input image is set to time 72, the position information of the person area image on the input image is stored in the tracking position 73, and the person area image is stored in the person area image 73. (Step S808). On the other hand, if there is no data that could not be associated with the series data for which the tracking flag 71 is ON in step S807, the tracking unit 14 proceeds to step 809.

The tracking means 14 sets the shooting time of the input image at time 72 for the sequence data that can be associated with the sequence data for which the tracking flag 71 is ON for the person region image of the current frame, The position information of the person area image is stored in the tracking position 73, and the person area image is additionally stored in the person area image 73 (step S809).
By carrying out the processes in steps S801 to S809 described above, series data tracking the same person is stored in the history table 33.

The following steps S810 to S817 shown in FIG. 5 are performed for each series data in which the tracking flag 71 is ON.
The face extraction unit 13 detects a face area from the latest person area image of the processing target sequence data, extracts the face area from the person area image, and extracts the face area image (step S810).
Next, the face collating means 17 calculates the similarity between the face area image and the registered face image stored in the registrant data 31, stores the highest similarity in the similarity 76, and the registered face image Are registered in the candidate person ID 77 (step S811).
Next, the face photographing condition calculating unit 15 calculates the light source direction as a feature indicating the photographing condition when the face region image extracted by the face extracting unit 13 is photographed, and the light source direction 75 in the history table 32 is calculated. Store (step S812).

  Next, the fraud determination means 16 determines whether or not the number of records in the light source direction calculated in the series data is 3 or more (step S813). If the number of records in the light source direction calculated in the series data is less than 3, the fraud determination means 16 proceeds to processing of the next series data without performing any particular process on the series data. On the other hand, when the number of records in the light source direction calculated in the series data is 3 or more, the fraud determination means 16 determines whether the light source directions are within a range of about ± 5 degrees where the light source directions can be determined to be substantially the same. (Step S814).

  If it is determined in step S 814 that it is within the range of about ± 5 degrees, the fraud determination means 16 stores the authentication result 78 of the history table 32 as an authentication result of the relevant series data as an illegal activity. The tracking flag is set to OFF (step S815). In step S815, when the authentication means 18 records the result of fraud in the authentication result 78 in the history table 32, the authentication means 18 determines that the photographic scheme has been performed and outputs a signal indicating that the photographic scheme has been performed. To the unit 20. When a signal indicating that the photographic scheme has been performed is output to the output unit 20, an alarm lamp (not shown) is turned on, a buzzer (not shown) is sounded, and the like via the output unit 20. Alternatively, a signal indicating that the photograph plan has been performed is output to the monitoring center device (not shown) via the output unit 20. As a result, the monitoring center apparatus can notify the administrator that the photo plan has been taken, and can record the history.

On the other hand, if it is determined in step S814 that the value is not within the range of about ± 5 degrees, the authentication unit 18 reads the similarity 76 and the candidate person ID 77 of the history table 32, and the registration stored in the candidate person ID. An average value of similarities having the same person ID is obtained. And it is judged whether the average value of the said similarity exceeds "0.9" which is an authentication threshold value (step S816).
If it is determined in step S816 that the authentication threshold has been exceeded, it is determined in step S817 that the person is a registered person, authentication OK is stored in the authentication result 78, and the tracking flag 71 is set to OFF. Then, the result output means 19 outputs an unlock signal to the electric lock and an authentication result OK signal via the output unit 20.
On the other hand, if it is determined in step S816 that the authentication threshold has not been exceeded, in step S818, the authentication unit 18 refers to the time 72, and the elapsed time from the initial time when the series data is created to the time of this frame. It is determined whether a certain authentication time is longer than a predetermined time. This predetermined time is a time for obtaining a human region image sufficient for face authentication even when a time for determining a change in the characteristic indicating the photographing condition has passed. If sufficient time has passed, the process proceeds to step S819, where the authentication NG is stored in the authentication result 78 and the tracking flag 71 is set to OFF. On the other hand, if it is less than the predetermined time in step S818, the process proceeds to processing of the next series data.
When the process is completed for all the series data, the process returns to step S801 in FIG. 4 to repeat the process for the next input image.

Next, an example of how the history table shown in FIG. 7 is processed by each unit of the face image authentication apparatus according to the present invention is shown in the schematic diagram of the shooting location shown in FIG. 1 and the person shown in FIG. This will be described using region images.
In the following, for ease of explanation, it is assumed that input images are acquired at 1-second intervals, and the history table 32 in FIG. 7 is also updated accordingly at 1-second intervals.

First, a case where a registrant is photographed will be described. The trial number 70 of the person is indicated by “1” in the history table 33.
As shown in FIG. 1A, the person 110 is photographed by the imaging unit 3 at time t <b> 1 and an input image is obtained via the image obtaining unit 11. In this embodiment, since the time t1 when the input image is acquired is “2012/06/20 10:12:35”, the time 72 is recorded.
Then, the person detection unit 12 extracts the person area image of the person 110 from the input image, and the tracking unit 14 obtains the position in the input image, that is, the center of gravity position of the person area image, and records it in the tracking position 73. In the present embodiment, the coordinates of the input image are defined such that the upper left is the origin, the X coordinate increases toward the right, and the Y coordinate increases toward the bottom, and the position of the person 110 is “(610, 320)”. The coordinate value is recorded at the tracking position 73.
The person area image extracted by the person detecting means 12 is recorded in the person area image 74 of the history table 33. The recorded person area image is as shown in FIG.

The face photographing condition calculation means 15 calculates a feature indicating the photographing condition from the face area image extracted from the person area image. In the present embodiment, coordinate information of a direction vector (size 1) representing the light source direction is calculated.
As shown in FIG. 1A, in the case of a person 110 at time t1, the downlight 102 is located slightly to the left and above the person 110. Since the XYZ coordinates of the direction vector representing the light source direction are “(0.09, 0.87, −0.5)”, the coordinate value is stored in the light source direction 75. The values of the XYZ coordinates are located in the direction of the pitch angle (elevation angle) 60 degrees upward from the horizontal direction with respect to the person 110 and the yaw angle 10 degrees to the left from the front direction viewed from the person 110. It corresponds to being.

The face collating means 17 obtains the similarity and stores the result in the similarity 76. In the present embodiment, since the lighting conditions and the like are suitable at the position of the person 110, a high similarity of “0.98” is required as a result of normalizing the similarity from 0 to 1.
The face matching unit 17 records “A” as an ID representing the registrant in the registered face image for which the similarity is obtained.

Next, it is assumed that a person has moved to the position indicated by reference numeral 111 at time t1 + 1 after one time has passed. In this embodiment, at time t1 + 1 when the input image is acquired, “2012/06/20 10:12:36” is recorded at time 72 when one second has elapsed from the time described above.
Then, “(630,530)” is recorded at the tracking position 73 as the coordinates of the center of gravity position of the person area image in the same manner as at time t1.
The person area image recorded in the person area image 74 is as shown in FIG.

Similar to the time t1, the face photographing condition calculation means 15 calculates the coordinate information of the direction vector representing the light source direction from the extracted face area image.
In the case of the person 111 at the time t1 + 1, the downlight 102 is positioned substantially immediately above the person 111. In this case, since the XYZ coordinates of the direction vector are “(0,1,0)”, the coordinate values are stored in the light source direction 75. The values of the XYZ coordinates are such that the downlight 102 is positioned with respect to the registrant 111 in a pitch angle (elevation angle) 90 degrees upward from the horizontal direction and a yaw angle of 0 degrees from the front direction viewed from the person 111. It corresponds to being.

The face collating means 17 obtains the similarity as in the time t 1 and stores the result in the similarity 76. In this embodiment, since there is a lighting device almost directly above the registrant 111, an unnatural difference in brightness occurs in the face portion as shown in FIG. 2 (b). Although a high degree of similarity cannot be obtained, since it is a registered person, a degree of similarity of “0.85” is required.
Further, the face collating unit 17 records “A” in the candidate person ID 77 as the ID representing the registrant in the registered face image for which the similarity is obtained.

Next, it is assumed that one time has passed and the person has moved to the position indicated by reference numeral 112 at time t1 + 2. In the present embodiment, at time t1 + 2, “2012/06/20 10:12:37” is recorded at time 72 when one second has elapsed from the time described above and the input image has been acquired.
Then, “(620,710)” is recorded at the tracking position 73 as the coordinates of the center of gravity position of the person area image in the same manner as at time t1.
The person area image recorded in the person area image 74 is as shown in FIG.

Similar to the time t1, the face photographing condition calculation means 15 calculates the coordinate information of the direction vector representing the light source direction from the extracted face area image.
In the case of the registrant 112 at time t1 + 2, the downlight 102 is located behind and above the person 112. In this case, since the XYZ coordinates of the direction vector are “(0, 0.98, 0.17)”, the coordinate values are stored in the light source direction 75. The values of the XYZ coordinates are such that the downlight 102 is positioned with respect to the person 112 at a pitch angle (elevation angle) 100 degrees upward from the horizontal direction and a yaw angle of 0 degrees from the front direction viewed from the person 112. It corresponds to that.

The face collating means 17 obtains the similarity as in the time t 1 and stores the result in the similarity 76. In the present embodiment, since the lighting device is behind and above the person 112, the face portion is entirely dark as shown in FIG. 2C, and the texture information of the face is taken into account due to the dynamic range of the imaging unit 3. May be somewhat missing, and a high degree of similarity is not obtained as at time t1. However, since it is a registered person, a similarity of “0.91” is required.
Further, the face collating unit 17 records “A” in the candidate person ID 77 as the ID representing the registrant in the registered face image for which the similarity is obtained.

The fraud determination means 16 refers to the light source direction 75 whose trial number 70 is “1” and grasps the change of the light source direction. In the present embodiment, at t1, t1 + 1, and t1 + 2, the elevation angles based on the horizontal direction are 60 degrees, 90 degrees, and 100 degrees, respectively, and values according to changes in the positional relationship with the downlight 102 that is the actual light source. It has become.
Therefore, the fraud determination means 16 determines that the person whose trial number 70 is “1” does not perform the fraud, and outputs the fact to the authentication means 18.

  The authentication means 18 refers to the similarity 76 having the trial number 70 of “1” and the candidate person ID 77 to identify the person being photographed. That is, in the present embodiment, the result that the similarity with the person whose registrant's ID is “A” is the highest is obtained at any of the three times of time t1, t1 + 1, and t1 + 2. The average of the similarities is 0.91. This exceeds the threshold value 0.9 for authentication determination. Therefore, the authentication unit 18 determines that the person whose trial number 70 is “1” has the registrant's ID “A”, and records “authentication OK (A)” in the authentication result 78.

Next, a case where a suspicious person is taking a picture plan will be described. The suspicious person has a trial number 70 indicated by “2” in the history table 33.
As shown in FIG. 1B, the person 120 holds the registrant face photograph 103 in front of his / her face. In the present embodiment, the time t2 when the input image is acquired is “2012/06/20 11:25:14”, so that time 72 is recorded.
As in the case of the person whose trial number 70 is “1”, “(580,310)” is recorded in the tracking position 73 as the coordinates of the center of gravity of the person area image. The person area image recorded in the person area image 74 is shown in FIG. 2 (d).

The face photographing condition calculation means 15 calculates the coordinate information of the direction vector representing the light source direction from the extracted face area image.
In the case of the person 120 at time t2, the XYZ coordinates of the direction vector are “(0.99, 0.09, −0.09)”, and the coordinate value is stored in the light source direction 75. The value of the XYZ coordinates “(0.99, 0.09, −0.09)” of the direction vector indicates that the light source direction viewed from the face shown in the face area image 204 is a pitch angle (elevation angle) 5 degrees upward from the horizontal direction, This corresponds to the fact that it is located in the direction of the yaw angle of 85 degrees almost directly to the left from the front direction seen from the face. Although the value of the XYZ coordinates is obtained from the face area image 204 which is the face portion of the person area image shown in FIG. 2D, the shadow information appearing in the face area image 204 is the person 120. Is the shadow information when the photograph is taken, and the value of the XYZ coordinates of the direction vector is obtained based on the shadow information. The same applies to t2 + 1 and t2 + 2 described below.

The face collating means 17 obtains the similarity and stores the result in the similarity 76. In the present embodiment, since the face area image 204 is a registered person, a similarity of “0.92” is required.
The face matching unit 17 records “A” as an ID representing the registrant in the registered face image for which the similarity is obtained.

Next, it is assumed that at time t2 + 1 after one time has passed, the person has moved to the position indicated by reference numeral 121 while holding the photograph 103 in the same manner. In the present embodiment, at time t2 + 1, “2012/06/20 11:25:15” is recorded at time 72 as a case where one second has elapsed from the time described above and the input image has been acquired.
As in the case of the person whose trial number 70 is “1”, “(590,500)” is recorded at the tracking position 73 as the coordinates of the center of gravity of the person area image. The person area image recorded on the person area image 74 is shown in FIG. 2 (e).

Similarly to time t2, the face photographing condition calculation unit 15 calculates the coordinate information of the direction vector representing the light source direction from the extracted face area image 206.
Since the XYZ coordinates of the direction vector in the case of the person 121 at time t2 + 1 are “(0.99, 0.10, −0.09)”, the coordinate values are stored in the light source direction 75. The value of the XYZ coordinates “(0.99, 0.10, −0.09)” of the direction vector is such that the light source direction viewed from the face shown in the face area image 206 is 6 degrees above the horizontal direction and the pitch angle (elevation angle). This corresponds to the fact that it is located in the direction of the yaw angle of 85 degrees almost directly to the left from the front direction seen from the face.

The face collating means 17 obtains the similarity as in the time t 2 and stores the result in the similarity 76. In the present embodiment, since there is a lighting device almost directly above the suspicious person 121, the face portion where the photograph is taken is slightly dark with backlight and is as high as at time t2, as shown in FIG. Although the degree of similarity is not obtained, it is a photograph of a registered person, and therefore a degree of similarity of “0.84” is required.
Further, the face collating unit records “A” in the candidate person ID 77 as an ID representing the registrant in the registered face image for which the similarity is obtained.

Next, assume that one more time has passed and at time t2 + 2, the person has moved to the position indicated by reference numeral 122 while holding the photograph 103 in the same manner. In the present embodiment, at time t2 + 2, “2012/06/20 11:25:16” is recorded at time 72 when one second has passed from the time described above and the input image has been acquired.
Similarly to the time t2, “(595,650)” is recorded at the tracking position 73 as the coordinates of the center of gravity position of the person area image. The person area image recorded on the person area image 74 is as shown in FIG. It is.

Similar to the time t2, the face photographing condition calculation means 15 calculates the coordinate information of the direction vector representing the light source direction.
Since the XYZ coordinates of the direction vector in the case of the person 122 at time t2 + 2 are “(0.99, 0.09, −0.07)”, the coordinate values are stored in the light source direction 75. The value of the XYZ coordinates “(0.99, 0.09, −0.07)” of this direction vector is a pitch angle (elevation angle) 5 degrees upward from the horizontal direction as the light source direction viewed from the face shown in the face area image 208. This corresponds to the fact that it is located in the direction of the yaw angle of 86 degrees almost directly to the left from the front direction seen from the face.

The face collating unit 17 stores the result in the similarity 76 as in the time t2. In the present embodiment, since the lighting device is behind and above the person 122, the face portion is entirely dark as shown in FIG. 2F, and the texture information of the face is taken into account due to the dynamic range of the imaging unit 3. May be somewhat missing, and a high degree of similarity is not obtained as at time t2. However, since the face area image 208 is the face of a registered person, a similarity of “0.90” is required.
Further, the face collating unit records “A” in the candidate person ID 77 as an ID representing the registrant in the registered face image for which the similarity is obtained.

The fraud determination means 16 refers to the light source direction 75 whose trial number 70 is “2” and grasps the change of the light source direction. In the present embodiment, at t2, t2 + 1, and t2 + 2, the elevation angle with respect to the horizontal direction is 5 degrees, 6 degrees, and 5 degrees, respectively, and the yaw angle to the left with respect to the front direction of the face is 85 degrees and 85 degrees. In spite of the fact that the downlight 102 is present at a degree of 86 degrees and 86 degrees, the pitch angle (elevation angle) and yaw angle are set in advance. It is within the range of ± 5 degrees.
Therefore, the fraud determination means 16 determines that the person with the trial number 70 of “2” is performing the fraud, and outputs that fact to the authentication means 18.

  The authentication means 18 determines whether the result recorded in the similarity 76 having the trial number 70 of “2” and the candidate person ID 77 is determined to be a registrant. Instead, “injustice” is recorded in the authentication result 78.

Next, a case where an unregistered suspicious person is photographed will be described. Unlike the person whose trial number 70 is “2”, the person does not perform an illegal act of holding a photograph in front of the face, and the trial number 70 is indicated by “3” in the history table 33.
The process for the person is the same as the case of the registrant with the trial number 70 being “1” because no fraud has been carried out, and details are omitted, but the degree of similarity 76 and the candidate person ID 77 are greatly different. ing.

That is, since it is unregistered, the similarity does not increase with any of the registrant's face images. Accordingly, the similarity 76 is smaller than the threshold value 0.9 used for authentication determination at any of the three times, and does not exceed the threshold value even if the average or maximum similarity is obtained.
Therefore, the authentication unit 18 determines that the person is not a registered person, and records “authentication NG (unregistered person)” in the authentication result 78.

  As described above, the face image authentication apparatus to which the present invention is applied extracts a region in which the same person's face is captured from each of a plurality of input images obtained by sequentially capturing a walking person, In addition, the light source direction is obtained for each face area image. Then, the face image authentication apparatus determines that a face photograph is included in the face area image if the light source direction of each face area image extracted from each input image does not change. Thereby, the face image authentication apparatus can appropriately determine the presence or absence of fraud using a face photograph.

In addition, the face image authentication device detects the change using the light source direction obtained for the face area image extracted from the continuously acquired input image, instead of detecting the change of the continuously acquired input image. Among them, the change may be detected by using the light source direction obtained for the face area image extracted from the input image respectively acquired at the timing when the person moves more than a certain distance. In this case, the face image authentication apparatus obtains the light source direction for a person region extracted at a certain timing for a certain series data. Then, the face image authentication device sets the direction of the light source for the series data only when the person area extracted thereafter is at a position more than a predetermined distance away from the position of the person area for which the light source direction was obtained immediately before. Ask. The position of each person area is stored in the tracking position information, and each time the face image authentication apparatus extracts a person area, the position of the center of gravity of the person area and the person area for which the light source direction was obtained immediately before are extracted. A distance (two-dimensional norm) between the center of gravity position is calculated, and the light source direction is obtained when the calculated distance is a predetermined value or more. As a result, the face image authentication apparatus can obtain a change according to the movement of the person, not a change according to the time of the light source direction in the face area image, and the face photograph is reflected in the face area image. It can be determined with higher accuracy.
Note that the face image authentication device may include a distance sensor, and may determine whether or not the person has moved a certain distance or more based on distance information acquired from the distance sensor. Accordingly, the movement distance of the person can be obtained with higher accuracy based on the actual position in the imaging region.

  In the present embodiment, an example in which the light source direction is obtained as a feature indicating the photographing condition at the time of photographing the face has been described, but the present invention is not limited to this. For example, the face photographing condition calculation means may obtain the variance of the luminance values of the pixels in the face area image in addition to or in place of the light source direction. In this case, the fraud determination means obtains a variance of a predetermined number or more of the face area images as the amount of change with respect to the variance of the brightness values obtained for each face area image. As shown in FIG. 2 (b), when a person who is not taking a photo plan is directly under the downlight, the face area image is bright and dark according to the unevenness of the face. The variance of the luminance values of the pixels in the face area image is larger than in the case where there is not. On the other hand, as shown in FIG. 2 (e), when the person who is taking a photo plan is directly under the downlight, the entire face area image becomes dark. The variance of the luminance values of the pixels in the face area image is not so different from that when no photographic scheme is performed. Therefore, this also allows the face image authentication apparatus to appropriately determine whether or not the face area image is a face photograph.

  Alternatively, the face photographing condition calculation means performs edge pixel extraction on the face area image using a Sobel filter or the like, generates image data including the extracted edge pixels (hereinafter referred to as edge image), and generates the generated edge An image may be used as one of the characteristics indicating the shooting conditions. In that case, the fraud determination means obtains a normalized correlation between the edge image for the face area image of the current frame and the edge image for the corresponding face area image of the previous frame for each series data, and calculates the reciprocal thereof for each series data. The amount of change between edge images. As shown in FIG. 2 (b), when a person who has not taken a photographic plan is directly under the downlight, the face area image is bright and dark according to the unevenness of the face. The distribution of the edge pixels to be extracted differs greatly depending on whether or not there is not. On the other hand, as shown in FIG. 2 (e), when the person who is taking a photo plan is directly under the downlight, the entire face area image becomes dark. The distribution of the extracted edge pixels is not so different as when no photo is taken. Therefore, this also allows the face image authentication apparatus to appropriately determine whether or not the face area image is a face photograph.

Alternatively, the face photographing condition calculation means may include the same person as the sample face region image obtained by directly photographing the person as a feature indicating the photographing condition in addition to, or instead of, the light source direction and luminance distribution obtained from the face region image. Obtain a partial space that can distinguish between a sample face area image obtained by directly photographing a person and a sample face area image obtained by taking a photograph, obtained by learning using a sample face area image obtained by taking a photograph of a face. Alternatively, a value obtained when a face area image is projected onto the partial space may be used.
For this purpose, a large amount of sample normal face area images showing the face of a walking person and sample plan face area images showing a face photograph of a walking person in front of his / her face Prepare to. The person shown in this face photograph is the same person as the person shown in the sample normal face area image. In addition, a sample normal face area image whose shadow changes greatly according to walking is prepared, and a sample plan face area image whose shadow does not change even when walking is prepared.
Then, the face image authentication device obtains a partial space by a pre-learning process as described below. Alternatively, the partial space may be obtained by a separate dedicated device and stored in the storage unit of the face image authentication device. Further, when a server client type device configuration is adopted, it may be obtained from a server machine. In the present invention, since any of the above methods is not limited, in the following description, devices for obtaining a partial space are collectively referred to as “computer”.

ここで、Dは振幅スペクトルの次元数を表し、画策特徴量空間の次元数D_Sより大きい値である。また、x_iは集合Lに含まれる各学習サンプルの顔領域画像から求められる各特徴量ベクトルを表し、nは集合Lに含まれる学習サンプルの数を表し、μは各学習サンプルについて求められた特徴量ベクトルの平均ベクトルを表す。また、iは1≦i≦nの整数であり、各特徴量ベクトルに付与された番号を表す。例えば、特徴量ベクトルとして、顔領域画像の各画素の輝度値を並べたベクトルを正規化したベクトルが用いられる。あるいは、顔領域画像の周波数スペクトル、ガボール特徴等を特徴量ベクトルとして用いてもよい。 The computer that obtains the partial space projects these sample face area images onto the eigenspace, and obtains the projection position at each base in the eigenspace. The computer has a small position variation for a plurality of sample plan face area images in which the same person is photographed, and a large position variation for a plurality of sample normal face area images in which the same person is photographed. To obtain a partial space (hereinafter referred to as a plan feature space).
The procedure for extracting the plan feature amount space from the eigenspace for the feature amount obtained from the face area image will be described below. When the set of learning samples of the face area image used to derive the eigenspace is L and the covariance matrix of the amplitude spectrum obtained from the face area image of each learning sample is Σ, the eigenvalue Λ = diag (λ _k ) (K is an integer satisfying 1 ≦ k ≦ D) and eigenvector Φ = (φ ₁ ,..., Φ _D ) are obtained from the following equations.

Here, D is represents the number of dimensions of the amplitude spectrum, a number of dimensions D _S greater than the scheme feature space. Further, x _i represents each feature vector obtained from the face area image of each learning sample included in the set L, n represents the number of learning samples included in the set L, and μ was obtained for each learning sample. This represents an average vector of feature vectors. Moreover, i is an integer of 1 ≦ i ≦ n and represents a number assigned to each feature vector. For example, a vector obtained by normalizing a vector in which the luminance values of the pixels of the face area image are arranged is used as the feature amount vector. Or you may use the frequency spectrum of a face area image, a Gabor feature, etc. as a feature-value vector.

画策特徴量空間で用いる基底を抽出するために用いる学習サンプルの集合をS、集合Sのうち人物jについてのサンプル画策顔領域画像の集合をP_j、サンプル通常顔領域画像の集合をN_jとすると、画策特徴量空間の基底e_k(k=1,…,D_S)は、固有ベクトルΦ=(φ₁,…,φ_D)のうち、x_i∈P_jに対する低次元表現u_lの分散が小さく、かつ、x_i∈N_jに対する低次元表現u_lの分散が大きくなるベクトルから求めるのが好ましい。例えば、コンピュータは、x_i∈P_jに対する低次元表現u_lの分散σ_Plをl=1,…,Dについて算出し、算出した全ての分散を小さい順に並べたときの分散σ_Plの順位をR_Plとし、さらにx_i∈N_jに対する低次元表現u_lの分散σ_Nlをl=1,…,Dについて算出し、算出した全ての分散を大きい順に並べたときの分散σ_Nlの順位をR_Nlとして、次式により、合成順位R_Alを求める。

ここで、αはサンプル画策顔領域画像とサンプル通常顔領域画像の重み付けのための定数であり、例えば0.5に設定される。 A low-dimensional representation u _l of the feature quantity obtained by projecting the feature quantity vector x _i onto the eigenvector φ _l corresponding to the l-th eigenvalue λ _l is expressed by the following equation.

A set of learning samples used to extract the base used in the plan feature space is S, a set of sample plan face area images for the person j in the set S is P _j , and a set of sample normal face area images is N _j Then, the basis e _k (k = 1,..., D _S ) of the plan feature space is the variance of the low-dimensional representation u _l for x _i ∈P _j among the eigenvectors Φ = (φ ₁ ,..., Φ _D ) Is preferably obtained from a vector in which the variance of the low-dimensional representation u _{l with} respect to x _i ∈N _j is large. For example, the computer calculates the variance σ _Pl of the low-dimensional expression u _l for x _i ∈P _{j with} respect to l = 1,..., D, and ranks the variance σ _Pl when all the calculated variances are arranged in ascending order. R _Pl, and the variance σ _Nl of the low-dimensional expression u _l for x _i ∈N _{j is} calculated for l = 1, ..., D, and the rank of the variance σ _Nl when all calculated variances are arranged in descending order is As R _Nl , the synthesis order R _Al is obtained by the following equation.

Here, α is a constant for weighting the sample plan face area image and the sample normal face area image, and is set to 0.5, for example.

あるいは、分散σ_Plの順位R_Plが所定順位内であり、かつ分散σ_Nlの順位R_Nlが所定順位内であるもののうち、順位R_Plと順位R_Nlの和が小さい順にD_S個のφ_l及びλ_lを選択して画策特徴量空間の基底e_kを求めてもよい。あるいは、分散σ_Plに対する分散σ_Nlの比σ_Nl／σ_Plをl=1,…,Dについて算出し、算出した全ての比を大きい順に並べたときの順位を上記の順位R_lとして画策特徴量空間の基底e_kを求めてもよい。このようにして求めた画策特徴量空間内では、同一人物が写っている複数のサンプル画策顔領域画像についての投影位置の変動は小さく、同一人物が写っている複数のサンプル通常顔領域画像についての投影位置の変動は大きくなる。従って、入力画像から抽出された顔領域画像の特徴量を画策特徴量空間に投影した位置は、顔領域画像から求められた光源位置や輝度の分散と同様に、顔の撮影時の撮影条件を示す特徴を表す。 The base e _k of the plan feature space is expressed by the following equation using the order R _l when the composition order R _Al is arranged in ascending order.

Alternatively, rank R _Pl of variance sigma _Pl is within the predetermined order, and variance sigma _Nl rank R _Nl Out of those which are within a predetermined order, the order sum rank R _Pl and charts R _Nl is small D _S pieces of φ The base e _k of the plan feature amount space may be obtained by selecting _l and λ _l . Alternatively, the variance sigma ratio of variance sigma _Nl for _Pl σ _Nl / σ _Pl and l = 1, ..., calculated for D, orchestrate characterized rank when arranged all ratios calculated in descending order as the rank R _l The base e _{k of the} quantity space may be obtained. In the plan feature amount space thus determined, the variation in the projection position for a plurality of sample plan face area images in which the same person is photographed is small, and for a plurality of sample normal face area images in which the same person is photographed. The variation of the projection position becomes large. Therefore, the position where the feature amount of the face area image extracted from the input image is projected onto the plan feature amount space is the same as the light source position obtained from the face area image and the variance of luminance, and the shooting conditions at the time of shooting the face are set. Represents the characteristics shown.

顔撮影条件算出手段は、画策特徴量u、vの相関値Cを次式により算出する。

不正行為判定手段は、算出した相関値Cの逆数をその特徴量の変化量とし、変化量が閾値未満である場合、写真画策が行われていると判定し、変化量が閾値以上である場合、写真画策が行われていないと判定する。つまり、不正行為判定手段は、算出した相関値Cが、顔領域画像に写っている顔とその顔に陰影を与える光源との位置関係が変化したことを表す場合、写真画策が行われていると判定し、位置関係が変化しなかったことを表す場合、写真画策が行われていないと判定する。なお、不正行為判定手段は、画策特徴量u、vをそれぞれの大きさで割ることにより正規化したベクトルを求め、求めたベクトルの差を特徴量の変化量としてもよい。
上記の説明では、分散σ_Nlを求めるためのサンプル通常顔領域画像と、分散σ_Plを求めるためのサンプル画策顔領域画像に写っている人物は同一人物として説明したが、それぞれの分散を求めるための顔領域画像が、十分に多くの人物について得られている場合には、人物ごとの分散σ_Nlと分散σ_Plのそれぞれを平均して、合成順位を求めるなどの処理を行っても同様に判定できる。 The face photographing condition calculation means obtains feature quantities x _A and x _B from the two face area images for each series data, projects each into the plan feature quantity space, and the plan feature quantity u = (u ₁ ,. , u _Ds ), v = (v ₁ ,..., v _Ds ). That is, the i-th components u _i and v _i of the plan feature quantities u and v are expressed by the following equations.

The face photographing condition calculation means calculates a correlation value C between the plan feature quantities u and v by the following equation.

The fraud determination means uses the reciprocal of the calculated correlation value C as the amount of change in the feature amount, and when the amount of change is less than the threshold, determines that a photo measure is being performed, and the amount of change is greater than or equal to the threshold , It is determined that no photography plan has been taken. In other words, if the calculated correlation value C indicates that the positional relationship between the face shown in the face area image and the light source that shades the face has changed, a photo measure is being taken. If it is determined that the positional relationship has not changed, it is determined that no photographic plan has been taken. Note that the fraudulent act determination means may obtain a normalized vector by dividing the plan feature quantities u and v by the respective sizes, and a difference between the obtained vectors may be used as a change amount of the feature quantity.
In the above description, the sample normal face area image for _obtaining the variance σ _Nl and the person shown in the sample plan face area image for _obtaining the variance σ _Pl are described as the same person. If the face area image is obtained for a sufficiently large number of people, the variance σ _Nl and variance σ _Pl for each person are averaged, and processing such as obtaining the synthesis order is performed in the same manner. Can be judged.

  In addition, when using the light source direction in the face area image or the variance of the luminance value of the pixel in the face area image as the feature indicating the shooting condition, the example of obtaining the change range and variance as the amount of change is shown. The present invention is not limited to this. For example, the fraud determination means sets all possible face area image pairs for the latest predetermined number of face area images in the sequence data, and is obtained for the set face area image pairs. Any one of the minimum value, the maximum value, the average value, and the median of the absolute values of the difference in characteristics indicating the photographing conditions may be obtained as the change amount. Alternatively, the cheating determination means sets an arbitrary one face area image pair for the latest predetermined number of face area images in the series data, and obtains the image obtained for the set face area image pair. You may obtain | require the absolute value of the difference of the characteristic which shows conditions as the variation | change_quantity.

  In addition, the fraud determination means does not use, for the determination of the fraud, the feature indicating the shooting condition obtained for the face area image that is not suitable for determining the presence / absence of the photographic plan among the face area images in the series data. You may do it. For example, the fraud determination means does not use the feature indicating the photographing condition obtained for the face area image having an extremely low average luminance value or the extremely high face area image for the determination of the fraud. Alternatively, the fraudulent act determination means does not use the feature indicating the photographing condition obtained for the face area image whose face direction is away from the front direction in the fraud determination. Note that the orientation of the face can be obtained by various known methods. For example, the number of pixels recognized as skin color in the face area image is counted, and the larger the ratio of the number of skin color pixels to the total number of pixels in the face area image is, the more the front direction is, and the smaller the distance is from the front direction. It can be judged. If the skin color pixel is more in the left half than the right half in the face area image, the right is displayed.If the skin color pixel is more in the right half than the left half, the left is displayed. If there are more than half the top half, it can be determined that the top is facing each other.

  As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Face image authentication apparatus 3 Imaging part 10 Image processing part 12 Face extraction means 14 Tracking means 15 Face imaging condition calculation means 16 Fraud determination means 20 Output part 32 History table

Claims (5)

A face image authentication apparatus for authenticating a person based on a face image of a person shown in an input image obtained by photographing a predetermined shooting area and a registered face image of a person registered in advance,
Image acquisition means for sequentially acquiring input images obtained by imaging the imaging region;
Person detecting means for extracting a person region image from the input image;
Tracking means for tracking the person area image extracted from the plurality of input images sequentially acquired, and generating series data in which the tracked person area image is associated;
Distance calculating means for calculating a distance between the position of the person area image in the input image and the position of the person area image in the immediately preceding input image;
Face photographing condition calculating means for calculating a feature indicating a photographing condition at the time of photographing a face from a face region image in the person region image;
For each series data, it is determined whether or not the characteristics indicating the shooting conditions of the person area image whose distance is greater than or equal to a predetermined value among the person area images included in the series data are within a predetermined range where they are substantially the same. Fraud determination means to
A face image authenticating device having an authenticating means for authenticating a fraudulent activity when the fraudulent act determining means determines that it is within a predetermined range.   The face image authentication apparatus according to claim 1, wherein the person detection unit extracts a face area image as a person area image.   The face authentication apparatus according to claim 1, wherein the face photographing condition calculating unit obtains a light source direction from the face area image and includes at least the light source direction in a feature indicating the photographing condition.   The face image according to any one of claims 1 to 3, wherein the face photographing condition calculation unit obtains a variance of luminance values from the face area image and includes at least the variance in a feature indicating the photographing condition. Authentication device. Furthermore, the sample face area image obtained by directly photographing a person and the sample face area obtained by photographing the sample face area image obtained by directly photographing the person and the sample face area image obtained by photographing the photograph showing the face. A storage unit that stores in advance a partial space that can be distinguished from an image;
5. The face photographing condition calculating unit obtains a position in the partial space when the face area image is projected onto the partial space, and includes the position in the partial space in the feature indicating the photographing condition. The face image authentication device according to any one of the above.

Images