JP2000331167A - Method and device for collating facial image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the reliability of person identification of a facial image collating method which is applied to data processing at the time of performing the authentication, retrieval, etc., of persons by using facial image. SOLUTION: In the facial image collating method, unnecessary image data outside a face area are deleted by performing masking for extracting the facial area from two two-dimensional facial image (a collated image and a collating image) to be collated with each other based on prescribed mask data (S16) and erroneous assumption of a person caused by a difference in color information is prevented by performing monochromatization (S18). Then the fine structure (for example, wrinkles) of the face of the person is made to be inconspicuous by performing histogram smoothing processing (S20).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the identification of an object (for example, a human face) appearing in a given two-dimensional image and an object appearing in a separately prepared image. The present invention relates to a method of image collation performed on a computer and a device therefor.

[0002]

2. Description of the Related Art Objects of the same kind (including humans) appear.
An image matching device that compares images and determines whether those objects are the same is a person authentication for ensuring security, a criminal investigation using a criminal image taken with a security camera, etc. It is applied to various technical fields. For example, when the image matching apparatus is used to authenticate a person, an image (matching source image) of a matching target portion (for example, the entire face, iris, ears, and fingerprint of a face) of a certain person is obtained using an imaging unit or the like. Then, while comparing the collation source image with a large number of images (registered images) previously stored in a database, a value (evaluation value) indicating the degree of coincidence of images calculated using a predetermined evaluation function is determined. The calculation of each registered image is performed, and if a registered image whose value is higher than a predetermined value is found, the person is determined to be registered, and the person is authenticated. In addition, when using the image matching device for criminal investigation, generally,
A person who corresponds to a registered image with a high evaluation value by comparing an image of a criminal taken by a security camera or a fingerprint image collected from a crime scene with a registered image as a reference image and calculating an evaluation value for each registered image The image of the suspicious person is searched for in the procedure of setting as the suspect.

For example, when an image matching apparatus is applied to person authentication, it is important how to select a face matching target portion.
For example, in a method in which an iris image of an eye is to be compared, it is common to obtain an iris image by having a person to be authenticated look into a lens of an imaging camera. Some people feel psychologically resistant. On the other hand, an image showing the entire face (hereinafter referred to as a face image)
In the method in which is a collation target, a face image can be acquired only by having the person to be authenticated perform a simple operation such as standing in a predetermined place.

[0004]

Generally, a face image captured by an imaging camera includes not only a face but also a background around the face. Therefore, if the image is used as it is, even the background image is included in the comparison target, and the reliability of face identification is reduced. Also, even if the person appearing in the captured face image is actually a person already registered in the database, for example, the person's hairstyle or tanning condition is different from that at the time of registration, or the lighting condition at the time of authentication Is different from the time of registration, the person may be mistaken for being unregistered.
Further, when the registered image stored in the database is composed of one or a plurality of two-dimensional images in which a person's face is viewed from one or a plurality of predetermined directions (front, left and right sides, etc.),
If the direction of the face appearing in the face image to be collated does not match any of the predetermined directions, a correct determination result cannot be obtained.

The present invention has been made to solve such a problem, and an object thereof is to be applied to data processing when performing authentication or search of a person using a face image. An object of the face image matching method is to enhance the reliability of person identification.

[0006]

A face image collating method according to the present invention, which has been made to solve the above problem, collates two two-dimensional face images, ie, a collation source image and a collation destination image. A face image matching method for evaluating the degree of coincidence of two faces appearing in one face image, the method including matching the size of the face appearing in the two face images and the position of the face in each face image. Normalization processing, mask processing for extracting a face area by masking a predetermined area of the two face images based on predetermined mask data, and converting each face image to a monochrome image as necessary And a tone reduction process for reducing the tone of each face image that has been subjected to the normalization process, the mask process, and the monochrome process.

[0007]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In the face image collating method according to the present invention, the collation source image is a two-dimensional face image obtained by processing an output signal of an imaging camera for authentication or security, for example.
The collation destination image is, for example, a two-dimensional face image reproduced from face image data registered in advance in a database. Here, the face image data itself registered in the database does not necessarily have to be two-dimensional image data.
It may be three-dimensional image data representing the three-dimensional shape of the face. In this case, a two-dimensional face image (projection image of the face onto a predetermined two-dimensional plane) obtained by processing the three-dimensional image data by a predetermined method is used as a verification target image.

[0008] Each process executed on the image data of the collation source image and the collation destination image will be described below.

The normalization process is a process for matching the size of a face appearing in two two-dimensional face images and the position in each image. In this process, for example, a plurality of feature points included in the face (for example, the center and end points of each component of the face such as eyes, nose, mouth, and ears) are specified in advance, and the corresponding specification of two face images is specified. Enlargement / reduction, rotation, partial cutout, movement, and the like of one or both images are performed in such a manner that the positions of the feature points (coordinate values in each image) are as close as possible. If the verification target image is a two-dimensional face image obtained by processing the three-dimensional image data by a predetermined method, the face direction appearing in the verification source image is verified before this normalization processing. It is desirable to perform a direction correction process that matches the direction of the face appearing in the previous image.

The above-mentioned direction correction processing is performed, for example, in Japanese Patent Application No. 11-358, which is a patent application previously filed by the present applicant.
The localization method for a three-dimensional directed object described in No. 33 can be suitably applied. Here, the three-dimensional directional object is an object whose direction and rotation position can be defined in a three-dimensional space, and a human face is one example. In the method according to the invention described in the above-mentioned patent application, a plurality of feature points including four feature points that are not on the same plane in a three-dimensional space among feature points of a three-dimensional directed body (face) are designated as designated feature points in advance. And obtaining the two-dimensional coordinate data of the specified feature point in the collation source image which is a two-dimensional image of the first three-dimensional directed object (face), By processing the three-dimensional shape data, a three-dimensional image of the second three-dimensional oriented body (face) facing a certain direction in the three-dimensional virtual space is generated, and a projection image of the three-dimensional image on a predetermined plane is generated. Is generated as a reference image, the two-dimensional coordinate data of the specified feature point is acquired in the reference image, and the three-dimensional shape data and the designated image acquired in the reference image and the reference image are acquired. Process the two-dimensional coordinate data of the feature points with a predetermined method By this, a check value indicating the degree of matching between the positional relationship of the designated feature point in the collation source image and the positional relationship of the designated feature point in the collation destination image is optimized, Determining an orientation in the three-dimensional virtual space. According to this method, for example, the collation can be performed only by performing a simple operation such as designating a designated feature point in an image displayed on a screen of an image display device of a personal computer by operating an input device (such as a mouse). A two-dimensional image of the second face viewed from the same direction as the direction in which the first face appearing in the original image was captured is displayed. If the directions of the faces appearing in the two two-dimensional face images are matched in this way, the reliability of the person recognition is greatly increased.

The mask process is a process of extracting a face region by masking a predetermined region of two face images based on predetermined mask data. In this processing, for example, by processing image data of the collation source image by a predetermined method, a face area in the collation source image is recognized, and mask data for extracting the face area is generated. The mask is applied to the face image. Of course, mask data may be generated by processing the collation destination image in a similar manner. Furthermore, a logical sum or a logical product of two face regions obtained from each face image may be obtained, and mask data may be generated based on the logical sum or the logical product. By such a mask process, information of the background image unnecessary for the person identification is deleted, and the reliability of the identification is improved.

Recognition of a face area in a face image may be performed by conventionally known image data processing (for example, processing for detecting a pixel having a color corresponding to human flesh color). In addition, when registering the face image data in the database, it is preferable to perform a mask process for extracting only the image data of the face area in advance and register the same. The process of generating mask data from a collation destination image is extremely simple, and is therefore preferable.

The monochrome conversion process is a process for converting each face image into a monochrome image having a predetermined gradation. Such a process not only does not impair the reliability of person recognition but also enhances the reliability. That is, when color information is used for person recognition using a face image, a problem arises in that, for example, the color information of the face area changes according to a change in makeup or lighting, and the probability of a person being erroneously recognized increases. Therefore, by performing the above-described monochrome processing, the influence of a change in color information is eliminated, and the reliability of recognition is improved. Note that it is desirable that the gradation for monochrome conversion be as high as possible from the original image data. If the original image is already a monochrome image, there is no need to perform this processing.

The gradation reduction processing is processing for lowering the gradation of each face image which has been subjected to the normalization processing, the mask processing, and the monochrome processing. By performing this processing, for example, an effect that the difference in brightness of the face image due to the influence of the illumination disappears and an effect that the fine structure (for example, wrinkles) of the face becomes inconspicuous can be obtained. As an example of the gradation lowering processing, there is a histogram smoothing processing.

[0015]

As described above, in the face image collating method according to the present invention, by combining the three processes of the mask process, the monochrome process, and the gradation lowering process, the important information for the person identification is obtained from the image data. Since only those are extracted, high reliability is achieved in the person identification.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A face image collating apparatus for performing image processing according to an image collating method according to the present invention will be described with reference to the drawings. FIG. 2 is a diagram showing a schematic configuration of the face image matching device of the present embodiment. The device 1 shown in FIG. 2 includes a central control device 11 that performs data processing according to a predetermined program, an input device (mouse, keyboard, etc.) 12 for a user to input commands and data to the central control device (CPU) 12, an image, Display device (CRT, LCD, etc.) 13 for displaying
The personal computer 10 includes an external storage device (such as a hard disk drive) 14 in which various data and programs are stored. External storage device 1
4, a database 15 including three-dimensional image data and personal information data (name, address, etc.) of a large number of human faces collected in advance is constructed. Further, the CPU 11 is connected with a video reproducing device 16 and a scanner 17. It is assumed that Windows (registered trademark of Microsoft Corporation, USA) is installed on the personal computer 10 as an operating system.

The procedure of image collation using the face image collation apparatus 1 will be described below. First, when the user starts the image collation program by operating the input device 12, the CP
U11 displays a window 20 as shown in FIG. In this window 20, an area in which a collation source image is displayed (hereinafter, referred to as a 2D image display area 21) and an area in which a collation destination image is displayed (hereinafter, a 3D projection area 22)
), Eight command buttons 23 to 30, and a text box 31 for displaying the matching degree. In each image display area, the X axis is taken in the horizontal direction (the right direction in the drawing is positive),
It is assumed that the Y axis is taken in the vertical direction (the downward direction is positive in the drawing).

Next, the user issues a command to the CPU 11 to capture the reference source image. For example, if the data of the collation source image has already been stored in the external storage device 14,
A selection button 23 arranged on the D image display area 21 is clicked. When taking in an image from a photograph, the user sets the photograph on the scanner 17 and then clicks the scan button 24. When capturing an image from a video recorded on a videotape, the user sets the videotape on the video reproducing device 16 to reproduce the recorded video, and presses a capture button 2 at a scene to be captured.
5 Click By performing any of the above operations, 2D
The face image is displayed in the image display area 21. This face image becomes the collation source image.

Next, when the user clicks a DB selection button 26 arranged above the 3D projection area 22, the CPU
11 displays a file selection window (not shown) on the screen of the display device 13. In this window, the user
Select the data file of the face image of the person you want to match.
It is preferable that the file selection window be provided with a database browsing function for displaying the face image and the personal information of the selected file, because the face image can be easily selected. When a file of a certain person is selected in the file selection window, the CPU 11 reads out the three-dimensional image data corresponding to the certain person and places the three-dimensional image of the person's face in the three-dimensional virtual space.
A three-dimensional image (hereinafter referred to as a 3D original image) is generated,
Image of the person's face viewed from the front (2 placed in front of the face)
Two-dimensional image data representing a two-dimensional image projected on a two-dimensional plane is generated, and the image is displayed in the 3D projection area 22. Hereinafter, this image is referred to as a 3D projection image.

Next, the above-mentioned Japanese Patent Application No. 11-3583 is disclosed.
A preparatory work for performing the direction correction processing of the collation destination image by the method described in No. 3 is performed. First, when the user clicks the point designation button 29, the process enters a point designation mode. In this mode, the user sequentially clicks a plurality of feature points predetermined on the face in the collation source image displayed in the 2D image display area 21. In the present embodiment, the left eyelid tip point P1, the right eyelid tip point P2, the nose root point P3, the nose tip point P4,
It is assumed that the cleft median point P5, the left earlobe point P6, and the right earlobe point P7 are clicked. The 2D image display area 2 shown in FIG.
In the collation source image displayed in No. 1, the right earlobe point is hidden by the face and cannot be clicked. Therefore, the point P7 is dummy set outside the face area as a discard point not used in the direction correction processing. Further, the user similarly clicks seven feature points for the 3D projection image. Note that the coordinate information of the seven feature points may be stored in the database in advance, so that the operation of clicking the feature points may not be necessary for the 3D projection image.

When the user clicks the execution button 30 after the designation of the feature points is completed, the CPU 11 executes the image collation processing. The procedure of the image matching process executed by the CPU 11 will be described with reference to the flowchart of FIG. In the following description, it is assumed that the collation source image is a 24-bit (16777216 color) color image and has a size of 512 × 512 (pixels).

In step S10, a process of rotating the 3D original image in the three-dimensional virtual space (direction correction process) so that the face appearing in the 3D projection area 22 faces in the same direction as the face appearing in the reference original image. )I do. This processing is based on Japanese Patent Application Hei 11
It is performed according to the procedure described in US Pat.

In step S12, processing (normalization processing) for matching the position and size of the face in each image display area is performed. Specifically, for example, a parallel movement process for matching the coordinates of the nose cusp P4 in the image display area in both images, and a distance between two pre-designated feature points (for example, two eyelid tip points P1 and P2) Enlargement / reduction processing for matching (P2 distance) in both images is performed.

In step S14, a process of reducing the resolution of the image is performed to reduce the amount of data processing within a range that does not impair the reliability of the collation. In this embodiment, the resolution is reduced to 1/4. As a result, the size (pixel conversion) of the image is 128 × 128 (pixels), and the number of pixels of each image is 1/16 (n = 16384 pixels). Regarding the image after the resolution has been reduced, n pixel data constituting the 2D image display area 21 is P (i, j, bp),
The n pieces of pixel data constituting the 3D projection area 22 are represented by Q
(I, j, bq). Here, i and j are variables indicating the X coordinate and the Y coordinate, both of which are integers from 0 to 127. The variables bp and bq included in the pixel data P and Q are variables for storing a value indicating the brightness of the pixel.

In step S16, a coordinate group corresponding to the face area is extracted from the 3D projection area 22. This is 3D
It is obtained by calculation from the three-dimensional coordinate data of the original image. Next, n mask data M (i, i, n) for extracting this face region (pixel groups existing on and inside the contour of the face)
j, m). Mask data M (i, j, m)
Contains a corresponding coordinate (i, j) and a variable m for storing a value indicating whether to mask the pixel at the coordinate (i, j). The value of m of the mask data M (i, j, m) is set to 0 when the coordinates (i, j) are included in the face area,
Otherwise, it is set to 1. Based on the mask data M (i, j, m) generated in this way, a mask process is performed on the comparison source image displayed in the 2D image display area 21. Then, the matching source image is masked as indicated by hatching in FIG. 5, and only the face region emerges.

In step S18, both images are converted to monochrome. The gradation for monochrome conversion is as high as possible.
However, if the three-dimensional image data stored in the database is monochrome image data of a predetermined gradation, the monochrome conversion processing is performed only on the reference original image so that the gradation of the reference original image is the same as the predetermined gradation. Apply. Here, as an example, both images are assumed to be 256-tone monochrome images.

In step S20, the gradation of both images is reduced by histogram smoothing. In the case of this embodiment, for example, the gradation is reduced from 256 gradations to 64 gradations. For each pixel of the collation source image and the 3D projection image after the processing, a value indicating the brightness (for example, an integer from 0 to 63) is set to pixel data P and Q.
In the variables bp and bq.

In step S22, the coordinates (i, i) are set for the coordinates (m = 0) constituting the face area.
The brightness variable bp and the variable b of both images with j) as a parameter
A correlation coefficient R1 with q is obtained.

In step S24, a histogram indicating the brightness distribution of the projected image of the face image on the X axis is generated for each image, and a correlation coefficient R2 between the two histograms is obtained. In step S26, a histogram indicating the brightness distribution of the projected image of the face image on the Y axis is generated for each image, and the correlation coefficient R3 between the two histograms is obtained. As an example, FIG. 3 shows two histograms showing the brightness distribution of the projected image on the X axis and the Y axis of the 3D projected image displayed in the 3D projected area 22.

In step S28, a matching degree is calculated by an evaluation function using the above three correlation coefficients R1, R2 and R3 as arguments. The method of calculating the matching degree may be determined as appropriate. An example of the matching degree is a product of the above three correlation coefficients (R1 × R2 × R3). The value of the calculated matching degree is displayed in the text box 31.

Based on the value of the degree of matching calculated as described above, the user determines whether the two face images belong to the same person. If it is determined that the face images do not belong to the same person, the DB selection button 2
6 Click and select the file again. If the user wants to confirm the face image of the person who has already undergone the collation processing and the collation result again, he or she clicks the previous data button 27 and the next data button 28 as appropriate to redisplay the face image of the desired person and the collation result on the screen. be able to.

Although the embodiment of the present invention has been described above, the description of the above embodiment does not limit the present invention in any way. For example, in the above description, the gradation of the monochrome image is reduced to 64 gradations by the histogram smoothing process, but this gradation value is not limited to 64 gradations. In the above description, the reference source image is masked by the mask for extracting the face area of the 3D projection image (reference destination image). However, the image of the reference source image itself is subjected to image analysis to form a mask for itself. May be created. In the above description, the resolution of the image is reduced. However, this processing is performed to reduce the amount of data processing, and is not essential to the present invention. It is obvious to those skilled in the art that the face image matching device of the above embodiment can be used for a person authentication device and a person search device using a face image.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating a procedure of a matching process performed by a face image matching device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a schematic configuration of the face image matching device.

FIG. 3 is two histograms showing the brightness distribution of the projected image on the X axis and the Y axis of the face image (3D projected image) displayed in the 3D projection area.

FIG. 4 is a view showing a window displayed on the display device.

FIG. 5 is a view showing a state where an image is masked in the window.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Face image collation device 10 ... Personal computer 11 ... Central control device (CPU) 12 ... Input device 13 ... Display device 14 ... External storage device 15 ... Database

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B043 AA09 BA04 CA03 EA03 EA04 EA08 EA09 EA13 EA15 FA09 GA04 GA10 HA02 HA06 HA07 HA08 5L096 AA02 AA06 BA08 BA18 CA02 DA01 DA02 EA06 EA13 EA14 EA37 JA11 JA03

Claims (4)

[Claims] 1. A collation source image and a collation destination image, which are two two-dimensional face images, are collated and appear in the two face images.
A face image matching method for evaluating the degree of coincidence of two faces, a normalization process for matching the size of the face appearing in the two face images and the position of the face in each face image, a predetermined mask Mask processing for extracting a face area by masking a predetermined area of the two face images based on the data, monochrome processing for converting each face image to a monochrome image as needed, and the normalization processing, A face image matching method, comprising: a tone reduction process for reducing the tone of each face image that has been subjected to a mask process and a monochrome process. 2. The collation destination image is a two-dimensional face image generated from image data stored in a database after performing a mask process for extracting only image data of a face area in advance, and the mask data is The face image matching method according to claim 1, wherein the face image matching method is generated based on image data. 3. A plurality of feature points including four feature points that are not on the same plane in a three-dimensional space among feature points of a face are determined in advance as designated feature points, and a two-dimensional image of the first face is defined. The two-dimensional coordinate data of the designated feature point is obtained from the certain collation source image, and the three-dimensional shape data of the second face is processed to obtain the three-dimensional shape of the second face in a certain direction in the three-dimensional virtual space.
Generating a three-dimensional image, generating a projection image of the three-dimensional image on a predetermined plane, and using the generated three-dimensional image as the reference image, obtaining two-dimensional coordinate data of the designated feature point in the reference image, By processing the shape data and the two-dimensional coordinate data of the designated feature point acquired in the comparison source image and the comparison destination image by a predetermined method, the positional relationship between the designated feature point in the comparison source image and the The orientation of the three-dimensional image in the three-dimensional virtual space is determined so that a check value indicating a degree of coincidence with the positional relationship of the designated feature point in a collation destination image is optimized. 3. The face image matching method according to 1 or 2. 4. A collation source image and a collation destination image, which are two two-dimensional face images, are collated and appear in the two face images.
A face image collating device for evaluating the degree of coincidence of two faces, a normalizing means for matching the size of the face appearing in the two face images and the position of the face in each face image; Masking means for extracting a face area by masking a predetermined area of the two face images based on the mask data of the above, monochrome converting means for converting each face image into a monochrome image as required, and the normalization A face image matching apparatus, comprising: a tone reduction unit for reducing the tone of each face image that has been subjected to the processing, the mask processing, and the monochrome processing.