JP2009211148A - Face image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face image processor for creating a three-dimensional image of a candidate from the two-dimensional face image of the candidate without using any special surface measurement device. <P>SOLUTION: The face image processor includes: an image input means 210 for inputting a two-dimensional face image; a storage means 250 for storing a three-dimensional shape model and the feature point positions of the face in the three-dimensional shape model in advance; a derived model creation means 231 for creating a derived shape model by making the rate of length to a feature point at the upper part of a reference point, the reference point being set on the positive middle line of the face surface of the three-dimensional shape model, to length to a feature point at the lower part of the reference point different from that of the three-dimensional shape model; a selection means (232, 233) for comparing the derived shape model and the three-dimensional shape model with the two-dimensional face image, and for selecting the shape model which is most similar to the two-dimensional face image; and a personal model creation means 240 for compounding the selected shape model with the two-dimensional face image to create a three-dimensional face image corresponding to the face of the person. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a face image processing apparatus, and more particularly to a face image that creates a 3D face image by estimating the 3D face shape of the individual from a 2D face image obtained by photographing the individual and a 3D face shape model. The present invention relates to a processing apparatus.

  Conventionally, there has been proposed a face authentication device that authenticates a target person by collating a two-dimensional face image acquired by photographing the face of the target person with a registered face image. Such a face authentication apparatus registers the face image of the subject person in advance, and determines whether or not to authenticate based on the result of comparison with the face image acquired at the time of use. For this reason, if there are differences in face orientation, lighting conditions, facial expressions, and the like between the face image at the time of registration and the face image at the time of use, the error rate at the time of matching increases.

  Therefore, a sample image collection method for creating a plurality of two-dimensional images of the subject's face for use in face image matching has been proposed (see Patent Document 1). Such an image collection method creates a three-dimensional computer graphics model of the subject's face from the three-dimensional surface shape data and color information of the face acquired using the surface measuring device. Then, the image collection method creates a plurality of two-dimensional face images by rendering with the face orientation or illumination conditions varied based on the model.

  Further, a face image matching device has been proposed that uses a standard frame model representing a standard three-dimensional shape of a face to create a plurality of reference face images to be used for matching (see Patent Document 2). The face image matching device generates a three-dimensional face model (three-dimensional face image) corresponding to the face of the subject by combining the two-dimensional face image of the subject acquired at the time of matching with the standard frame model. . Then, the face image collation device creates a reference face image by rendering the three-dimensional face model in consideration of a variation factor such as a face direction, illumination conditions, or facial expressions.

JP-A-4-256185 JP 2003-6645 A

  In the image collection method described in Patent Document 1, it is necessary to create a three-dimensional computer graphics model of a face for each subject. However, since it is necessary to measure the face of the subject person using a special surface measuring device in order to create the model, it is not easy to create a three-dimensional computer graphics model of each subject person's face. For this reason, it has been difficult to apply such an image collection method to a commonly used face authentication apparatus.

On the other hand, in the face image matching apparatus described in Patent Document 2, one or more standard frame models corresponding to a standard face are prepared as models representing a three-dimensional shape. This standard frame model only needs to be created once and can be used for any of such face image collation apparatuses, so the above-mentioned problems do not occur. However, since the standard frame model corresponds to the standard face shape, for example, the characteristics of the face shape that each person has, such as the nose tip-to-eye spacing, the nose tip-to-mouth spacing, which is the tip of the nose. Does not reflect. In particular, the shape around the nose, which is located in the center of the face and has a prominent three-dimensional shape, has a large effect on positioning accuracy when combining a frame model and a two-dimensional face image. The solid shape cannot be reflected. Therefore, in order to synthesize the standard frame model and the two-dimensional face image, for example, even if an attempt is made to align the standard frame model and the two-dimensional face image around the nose tip that is dominant in the uneven shape of the face, the three-dimensional shape is different. For example, since the distance between each feature point of the face is different, there is a case where proper alignment cannot be performed.
In order to prepare a standard frame model having a similar face shape for a subject having any facial feature, it is necessary to acquire a large number of face shape data of a large number of people in advance. Moreover, it is desirable that the shapes of the faces from which the shape data is acquired are not similar to each other and are as different as possible. However, collecting such face shape data for a large number of people requires a great deal of labor. Also, as the number of standard frame models increases, the amount of calculation at the time of collation increases, so it is not desirable to increase the number of standard frame models too much.

Therefore, an object of the present invention is to use a small number of three-dimensional shape models representing the three-dimensional shape of a face, and to reproduce the target person's face shape well from the target person's two-dimensional face image. Is to provide a face image processing apparatus for generating
Another object of the present invention is to provide a face authentication apparatus having a low verification error rate using a three-dimensional face image created from a two-dimensional face image of a subject.

  To solve this problem, the present invention creates a three-dimensional face image corresponding to the face of a person by synthesizing a two-dimensional face image including the face of the person and a three-dimensional shape model representing the three-dimensional shape of the face. A face image processing apparatus is provided. The face image processing apparatus includes an image input unit that inputs a two-dimensional face image, a storage unit that stores in advance a three-dimensional shape model and a facial feature point position in the three-dimensional shape model, and a three-dimensional shape model. The length from the reference point to the feature point above the reference point and the feature point below the reference point with respect to the reference point set at a position where the feature point exists at least above and below the midline of the face surface of the three-dimensional shape model A derived model creating means for creating a derived shape model in which the ratio of the length to the 3D shape model is different from that of the 3D shape model, and comparing the derived shape model and the 3D shape model with the 2D face image, A selecting means for selecting a similar one, and a personal model creating hand that creates a 3D face image corresponding to the face of the person by synthesizing the selected derived shape model or 3D shape model with the 2D face image. With the door.

  The three-dimensional shape of the face varies from person to person. Therefore, the face image processing apparatus of the present invention creates a derived shape model in which the ratio of the lengths above and below the reference point is different from the pre-stored three-dimensional shape model of the face, so that the size and length of the three-dimensional shape are different. Variations in face shape can be created, and a 3D face image can be created by selecting a shape model more similar to the face shape of the subject.

In the present invention, the reference point is preferably a nose tip.
Further, in the present invention, the derivation model creating means includes a distance along the midline of the face from the nose tip to the feature point representing the nose root point or the center position of both eyes, and a distance from the nose tip to the feature point representing the mouth point. It is preferable to create at least one derived shape model with different ratios.

  Furthermore, in the present invention, the derived model creating means can create at least one derived shape model by transforming the contour shape of the face of the three-dimensional shape model so that the ratio of the length in the horizontal direction to the vertical direction is changed. preferable.

  The face image processing apparatus according to the present invention further includes a reference face image storage unit that stores a reference face image of at least one registrant facing a predetermined direction, and rotates the face direction of the three-dimensional face image in a three-dimensional space. The corrected face image creating means for creating a corrected face image in which the face of the person is directed in a predetermined direction is compared with the corrected face image and the reference face image to determine whether or not to authenticate the person as a registrant. It is preferable to further have a collation means.

The face image processing apparatus according to the present invention can create a three-dimensional face image that well reproduces the face shape of the subject from the two-dimensional face image of the subject using a small number of three-dimensional face models. There is an effect.
In addition, the face authentication apparatus according to the present invention has an effect that the verification error rate can be reduced by performing matching using a three-dimensional face image created from a two-dimensional face image of the subject.

Hereinafter, embodiments of a face image processing apparatus according to the present invention will be described with reference to the drawings.
When a face image processing apparatus to which the present invention is applied acquires a face image of a target person, the face image and a 3D shape model representing a 3D shape of a face prepared in advance are aligned and synthesized, A three-dimensional face model of the person is created. At that time, the face image processing device transforms the three-dimensional shape model so as to change the ratio of the length in the height direction with respect to a reference point such as the tip of the nose to change the distance between the feature points of the face and the size of the three-dimensional shape. By creating derivative shape models having different heights, a three-dimensional face model having a face shape that well matches the face shape of the subject is created. This three-dimensional face model is the three-dimensional face image of the present invention, and the face image of the subject obtained as a two-dimensional image is synthesized on a three-dimensional shape model that matches the face shape of the subject, By rotating the three-dimensional face model in an arbitrary direction, an image in which the orientation of the subject's face is changed can be output.

FIG. 1 shows a schematic configuration of a face image processing apparatus 200 to which the present invention is applied. As shown in FIG. 1, the face image processing apparatus 200 includes an image input unit 210, a face feature point extraction unit 220, an alignment information calculation unit 230, a personal model creation unit 240, and a storage unit 250.
Of these, the facial feature point extraction means 220, the alignment information calculation means 230, and the personal model creation means 240 are functional modules implemented by a microprocessor, a memory, peripheral circuits thereof, and software operating on the microprocessor, respectively. is there. Alternatively, these means may be integrated by firmware. Moreover, you may comprise some or all of these means with an independent electronic circuit, firmware, a microprocessor, etc. Hereinafter, each part of the face image processing apparatus 200 will be described in detail.

  The storage unit 250 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device. The storage unit 250 stores a computer program, various parameters, data, and the like for controlling the face image processing apparatus 200. The storage unit 250 stores a 3D shape model of the face and 3D position information of 3D face feature points corresponding to the 3D shape model. There may be a plurality of three-dimensional shape models or one. The three-dimensional shape model is a frame model representing the three-dimensional shape of a human face, and a wire frame model, a surface model, or the like is used. Note that the three-dimensional shape model is created, for example, from the face shapes of a plurality of persons, or by averaging face shape models simulating the face shapes of a large number of persons.

  In addition, the 3D face feature point represents any point of a characteristic part such as an eye, a nose, or a mouth that is different from the other in terms of shape or color component, for example, a center point or an end point of those parts. For example, the 3D face feature points include an eyebrow head, an eyebrow butt, a black eye center, an eye region center, an eye head, an eye corner, a nose tip, a nostril center, a mouth point, a mouth corner point, and the like. The type and number of 3D face feature points to be stored are not limited, but preferably include at least all the feature points of the same part as the face feature points that can be extracted by the face feature point extraction unit 220. In the present embodiment, as the 3D face feature points, the left and right eye region centers, the eyes and the corners of the eyes, the nose apex, the mouth point, and the left and right mouth corner points are stored.

  The image input unit 210 is an interface circuit connected to an image acquisition unit such as a surveillance camera, for example, and inputs the face image 100 of the subject acquired as a two-dimensional face image by the image acquisition unit to the face image processing apparatus 200. To do. In addition, when there is a recording medium that records history information including one or more face images taken in advance, the image input unit 210 includes a reading device for accessing such a recording medium, a keyboard, a mouse, and the like. And a user interface including a display. In this case, the user selects any one face image 100 from the history information via the user interface of the image input unit 210. The face image 100 includes the entire face, and each facial feature (such as eyes, nose, and mouth) can be distinguished from other feature parts. Further, the face image 100 can be a grayscale or color multi-tone image, but is preferably a color multi-tone image that easily extracts the characteristics of the human skin portion. In the present embodiment, the face image 100 is a color image having 128 × 128 pixels and having an 8-bit luminance resolution for each color of RGB. However, the face image 100 may have a resolution and gradation other than this embodiment.

  The face feature point extraction unit 220 extracts face feature points from the face image 100 acquired by the image input unit 210. Then, the face feature point extraction means 220 uses the extracted face feature point type and position information on the face image 100 (for example, a two-dimensional coordinate value with the upper left corner of the face image 100 as the origin) as position information calculation means. 230. In the present embodiment, the face feature point extraction unit 220 extracts face feature points corresponding to the respective 3D face feature points (10 locations such as the center of the eye region, the tip of the nose, the corner of the mouth) stored in the storage unit 250. . The face feature point extraction unit 220 can use various known methods for extracting face feature points from the face image 100. For example, the face feature point extraction unit 220 performs edge extraction processing on the face image 100 to extract edge pixels having a large luminance difference from surrounding pixels. Then, the face feature point extraction means 220 checks whether or not the feature amount obtained based on the position and pattern of the edge pixel satisfies a predetermined condition for a part such as an eye, nose, and mouth. Each facial feature point can be extracted by specifying the position. In addition, instead of performing edge extraction processing to extract edge pixels, the facial feature point extraction means 220 performs Gabor conversion processing or wavelet conversion processing to extract pixels with large local changes in different spatial frequency bands. May be. Furthermore, the face feature point extraction unit 220 may extract the face feature points by performing template matching between the template corresponding to each part of the face and the face image 100 to specify the position of each part of the face. Furthermore, the face feature point extraction unit 220 causes each user to specify the position of each face feature point through a user interface (not shown) including a keyboard, a mouse, a display, and the like. You may get it.

  The alignment information calculation unit 230 creates a plurality of derived shape models obtained by modifying the 3D shape model stored in the storage unit 250 so that the relative positional relationship of the 3D face feature points changes, and the derived shape model and Position alignment is performed so that the orientation of the face of the face image 100 acquired by the image input unit 210 is equal to the three-dimensional shape model. Then, the alignment information calculation unit 230 refers to the alignment information obtained as a result of the alignment, and selects the shape model with the least alignment error from among the derived shape model and the three-dimensional shape model as the individual model creation unit 240. Output to. For this purpose, the alignment information calculation unit 230 includes a derived model creation unit 231, an alignment unit 232, and a similar shape selection unit 233. The alignment unit 232 and the similar shape selection unit 233 function as the selection unit of the present invention.

The derived model creating unit 231 creates a plurality of derived shape models obtained by transforming the 3D shape model stored in the storage unit 250 so that the relative positional relationship of the 3D face feature points changes.
FIG. 2 shows a three-dimensional orthonormal coordinate system (u, v, w) that serves as a reference when creating a derived shape model. As shown in FIG. 2, the u axis of the orthonormal coordinate system (u, v, w) is set as a vertical axis along the midline of the face with respect to the three-dimensional shape model 110. The v-axis is set as a horizontal axis in the direction crossing the face left and right with respect to the three-dimensional shape model 110. The w axis is set with respect to the three-dimensional shape model 110 as a horizontal axis in a direction crossing the face back and forth.

Derived model creation means 231 can change the distance in the height direction to a 3D face feature point above the reference point as an example of a deformation method of the three-dimensional shape model. In the present embodiment, the reference point is the nose tip. In this case, when the derived model creation unit 231 reads the 3D shape model and the 3D face feature point from the storage unit 250, the distance L ₁ along the u axis between the midpoint of the center of each eye region and the nose tip point. Ask for. The derived model creating means 231 then enlarges the distance along the u-axis direction with respect to the nose apex (that is, the vertical distance) for each 3D face feature point above the nose apex according to a predetermined enlargement / reduction ratio M _1. Alternatively, a derived shape model is created by correcting the position of each 3D face feature point above the nose tip so as to reduce. At that time, the derivation model creation means 231 obtains the distance L ₁ ^′ along the u-axis between the center point of the left and right eye region center and the nose tip in the created derivation shape model, and the original three-dimensional shape model. The enlargement / reduction ratio M ₁ is set in N stages within the range of the distance L ₁ within the range of R _L to R _U (where R _U >1.0> R _L > 0.0). Then, the derived model creating means 231 creates N derived shape models based on the enlargement / reduction ratio M ₁ at each stage. Incidentally, the distance L _1, may be distance along the midline between the midpoint of the nose apex point and both inner corner or both outer corners of the midpoint. Also, if it contains nasion point to 3D face feature point, the distance L ₁ may be the distance between the nose cusp and root of the nose point.

As another example of the deformation method of the three-dimensional shape model, the derived model creating unit 231 can change the distance in the height direction to the 3D face feature point below the nose apex point that is the reference point. In this case, the derived model generating means 231, by reading the 3-dimensional shape model and 3D facial feature points from the storage means 250, obtains the distance L ₂ along the u axis of the mouth point and nose cusps. Enlargement and the derived model generating means 231, for each 3D face feature point below the nose cusp, distance along the u-axis direction with respect to the nose apex point (i.e., distance in the vertical direction) in accordance with a predetermined enlargement / reduction ratio M ₂ Alternatively, the derived shape model is created by correcting the position of each 3D face feature point below the nose tip so as to reduce. At that time, the derived model generating means 231, the distance L ₂ ^'along the u axis of the mouth point and nose apex point in the derived geometric model generated, the ratio of the distance L ₂ calculated for the original 3-dimensional shape model Within the range of R _L to R _U (where R _U >1.0> R _L > 0.0), the enlargement / reduction ratio M ₂ is set to N levels. Then, the derived model creating means 231 creates N derived shape models based on the enlargement / reduction rate M ₂ at each stage. Note that the distance L _2, may be distance along the midline between the midpoint of the nose apex point and both corners of the mouth point.
In addition, the derived model creating means 231 changes the magnification / reduction ratios M ₁ and M ₂ independently and both, and changes the vertical position of each 3D face feature point according to the enlargement / reduction ratios. A derived shape model may be created.

The ratio of the distances L ₁ and L ₂ varies depending on the individual. Further, even if the entire three-dimensional shape model is linearly deformed at the same enlargement / reduction ratio, the ratio between L ₁ and L ₂ does not change. Therefore, when the ratio of L ₁ and L ₂ in the face image 100 of the subject is different from the ratio of L ₁ and L _{2 in} the three-dimensional shape model, the face image can be enlarged or reduced even if the entire three-dimensional shape model is enlarged or reduced. 100 and the 3D shape model cannot be properly aligned. However, it is possible to obtain a shape model that can be properly aligned with the face image 100 by creating a plurality of derived shape models having different ratios of L ₁ and L ₂ by performing the above-described deformation. It becomes possible.

この式から明らかなように、変化量δを小さくするほど、あるいはR_UとR_Lの差を大きくするほど、作成される派生形状モデルの数Ｎは多くなる。そして、変化量δを小さくするほど、L₁とL₂との比を細かく設定できるので、顔画像１００の各顔特徴点との位置ずれ量が小さい３Ｄ顔特徴点を持つ派生形状モデルを作成することができる。また、R_UとR_Lの差を大きくするほど、L₁とL₂との比の値の範囲が拡がるので、様々な顔に対してL₁とL₂との比の差が小さい派生形状モデルを作成することができる。 Incidentally, creation speed N of the derived shape model, for example, the enlargement / reduction ratio M ₁ or M ₂ of the case of changing from R _L by R constant change amount in the range of _U [delta], represented by the following formula .

As is apparent from this equation, the smaller the amount of change δ or the greater the difference between R _U and R _L , the greater the number N of derived shape models created. Since the ratio between L ₁ and L ₂ can be set more finely as the amount of change δ is reduced, a derived shape model having a 3D face feature point with a small positional deviation from each face feature point of the face image 100 is created. can do. Also, R _U and the larger the difference between the R _L, L ₁ and the range of the ratio of the values of L ₂ is expanded, L ₁ and derived form the difference in specific small and L ₂ with respect to various face A model can be created.

The minimum value R _L and the maximum value R _U enlargement / reduction ratio M _1, M ₂ is set to cover a range where the ratio between L ₁ and L ₂ are obtained practically taken as a human face . For example, when the three-dimensional shape model has a standard face shape created by averaging the face shape models imitating the face shapes of many persons as described above, R _L = 0.8, R _U = 1.2 can be set. The amount of change δ comprehensively considers the accuracy required for the alignment between the face image 100 and the derived shape model, and the calculation time required to create the derived shape model and align the derived shape model with the face image 100. To be determined. For example, when R _L = 0.8 and R _U = 1.2, the change amount δ can be set to 0.05. However, R _L , R _U , and δ are not limited to the above values, and are appropriately optimized according to the ability of the microprocessor employed in the embodiment of the present invention and the face image processing apparatus 200.
The derived model creation unit 231 outputs the created plurality of derived shape models to the alignment unit 232.

  The alignment means 232 rotates, enlarges / reduces the three-dimensional shape model and each derived shape model so that the positions of the face image 100 and the corresponding face feature points most closely match, that is, the face direction is equal. Alignment is performed so that the alignment information is calculated. The alignment unit 232 calculates an alignment error when the face image 100 and the 3D shape model are aligned using each alignment information in order to evaluate each calculated alignment information.

  The alignment information includes, for example, the rotation angle, translation amount, and expansion / contraction along each axis of the three-dimensional orthonormal coordinate system (X, Y, Z) set for the three-dimensional shape model or the derived shape model. Includes reduction ratio. In the orthonormal coordinate system (X, Y, Z), for example, the center of gravity of a plurality of 3D face feature points on the three-dimensional shape model is set as the origin, and the X axis in the direction horizontal to the face and from left to right. A Y-axis is set in a direction horizontal to the face and from the back to the front, and a Z-axis is set in a direction from bottom to top perpendicular to the face.

（２）式において、α_iは、距離E_iに対する重み係数である。重み係数α_iは、例えば、顔特徴点抽出手段２２０において抽出された各顔特徴点の位置の信頼度が高いほど、大きな値となるように設定される。例えば、顔特徴点抽出手段２２０が顔の各部位に相当するテンプレートを用いたテンプレートマッチングにより各顔特徴点の位置を決定する場合、顔特徴点の位置の信頼度は、テンプレートと顔画像１００が最も一致する場合の一致度とすることができる。また、特定の顔特徴点（例えば、鼻尖点）の位置合わせを重視したい場合には、その重視したい顔特徴点に対応する重み係数α_iを、他の顔特徴点に対応する重み係数よりも大きくしてもよい。なお、顔特徴点の位置の信頼度が評価できない場合若しくは各顔特徴点の位置の信頼度がほぼ等しい場合、あるいは、全ての顔特徴点について位置ずれ量が均等になるように位置合わせしたい場合には、全ての重み係数α_iを同じ値（例えば'1'）に設定する。 The alignment unit 232 calculates the alignment error as follows, for example. First, the 3D face feature points on the derived shape model are projected onto the face image 100 which is a two-dimensional image. Then, the alignment means 232 calculates the distance E _i between each projected 3D face feature point and each corresponding face feature point, and obtains the total displacement E as a weighted sum of the distance E _i . The total positional deviation amount E is expressed by the following equation, for example.

In the equation (2), α _i is a weighting factor for the distance E _i . For example, the weighting coefficient α _i is set to be larger as the reliability of the position of each facial feature point extracted by the facial feature point extraction unit 220 is higher. For example, when the face feature point extraction unit 220 determines the position of each face feature point by template matching using a template corresponding to each part of the face, the reliability of the position of the face feature point is determined by the template and the face image 100. The degree of coincidence can be determined as the best match. In addition, when it is important to position a particular facial feature point (for example, the nose tip), the weighting coefficient α _i corresponding to the facial feature point to be emphasized is set to be higher than the weighting coefficients corresponding to other facial feature points. You may enlarge it. When the reliability of the position of the face feature points cannot be evaluated, or when the reliability of the positions of the face feature points is almost equal, or when it is desired to align all the face feature points so that the amount of displacement is equal. In this case, all the weighting factors α _i are set to the same value (eg, “1”).

The alignment means 232 rotates or rotates the three-dimensional shape model or the derived shape model along each axis of the three-dimensional orthonormal coordinate system (X, Y, Z) so that the positional deviation amount E is minimized. Translate or zoom in or out. The alignment unit 232 uses, for example, an energy minimization method such as simulated annealing or steepest descent method to change the posture change amount (rotation angle, translation amount, and translation amount) of the derived shape model when the positional deviation amount E is minimized. (Magnification / reduction ratio) can be obtained. The alignment unit 232 sets the positional deviation amount E, which is minimized for each shape model due to the above-described posture change, as the alignment error for the three-dimensional shape model aligned with the face image 100 or its derived shape model.
In addition, the registration unit 232 projects each facial feature point extracted from the facial image 100 into the three-dimensional space, and then the positional deviation amount with the 3D facial feature point on the three-dimensional shape model or the derived shape model is minimized. The alignment error may be calculated by rotating, enlarging / reducing the three-dimensional shape model or the derived shape model so that
The alignment unit 232 outputs the obtained alignment error to the similar shape selection unit 233.

  The similar shape selection unit 233 has the smallest alignment error among the three-dimensional shape model and the derived shape model (that is, the position of each 3D face feature point is the most similar to the position of each corresponding face feature point in the face image 100). That match) is output to the individual model creation means 240 as the most similar shape model.

The personal model creation unit 240 creates a 3D face model of the subject by mapping the face image 100 as a texture image onto the most similar face shape model. That is, the alignment information is set so that each face feature point extracted for the face image 100 and the 3D face feature point for the three-dimensional shape model or derived shape model selected as the most similar shape model are aligned. The face image 100 is used as a texture image and mapped to the most similar face shape model. Occlusion may occur where a part of the face is hidden due to a part protruding on the face such as the nose, or due to a shield or the like, and information on a part of the face is missing in the face image 100. Therefore, when a missing portion of face information or texture occurs in the three-dimensional face model to which the face image 100 is mapped, the personal model creation means 240 can detect pixels around the missing portion before mapping or after mapping. Interpolation processing (for example, spline interpolation or linear interpolation) is performed using the luminance information, and the luminance value of the missing pixel is calculated. Alternatively, the personal model creation means 240 may use the luminance value of the pixel corresponding to the symmetrical position of the missing portion as the luminance value of the pixel of the missing portion by utilizing the symmetry of the human face. Specifically, the luminance value of a pixel at a line-symmetrical position around the midline of the face with respect to the missing portion can be set as the luminance value of the pixel of the missing portion. By performing such an interpolation process, when the face image 100 is acquired, the subject faces the direction in which part of the face is hidden, or part of the subject's face is hidden behind the shielding object. Even in this case, a three-dimensional face model that represents the entire face of the subject can be created.
The personal model creation unit 240 stores the created three-dimensional face model in the storage unit 250 or outputs it to a face authentication device or the like that performs a matching process using the face image processing device 200.

Hereinafter, the operation of the three-dimensional face model creation process by the face image processing apparatus 200 to which the present invention is applied will be described with reference to the flowchart shown in FIG. The flow of operations described below is controlled by a control unit (not shown) that operates on the microprocessor constituting the face image processing apparatus 200 and controls the entire face image processing apparatus 200.
First, the face image processing apparatus 200 acquires the face image 100 of the subject as a two-dimensional image via the image input unit 210 (step S310). Next, the face feature point extraction unit 220 extracts face feature points from the acquired face image 100 (step S320).

  Next, each 3D shape model and 3D face feature points associated with the 3D shape model are read from the storage unit 250 (step S330). Then, the derived model creation unit 231 of the alignment information calculation unit 230 nonlinearly determines the positional relationship between the 3D face feature points above the nose apex and the positional relationship between the 3D face feature points below the nose apex in the vertical direction. A plurality of changed derived shape models are created (step S340). After that, the registration unit 232 of the registration information calculation unit 230 associates each 3D shape model and the derived shape model with each face feature point extracted from the face image 100, and each 3D shape model and the derived shape model. Each shape model can be rotated or translated along each axis of the orthonormal coordinate system (X, Y, Z) so that the positional deviation amount E of the 2D projection point of the 3D face feature point to be minimized Enlarge / reduce. The positioning means 232 calculates this as a positioning error when the positional deviation amount E is minimized (step S350).

  After step S350, it is determined whether or not alignment errors have been calculated for all three-dimensional shape models and all derived shape models created from these models (step S360). If there is a derived shape model or three-dimensional shape model for which the alignment error has not been calculated, the control returns to step S350, and the alignment error is not calculated for the derived shape model or the three-dimensional shape model in step S350. The process is repeated.

Next, the similar shape selection unit 233 of the alignment information calculation unit 230 selects, from among all the three-dimensional shape models and derived shape models, the one with the smallest alignment error as the most similar shape model (step S370). ).
Finally, the personal model creation means 240 creates a 3D face model of the subject by mapping the face image 100 as a texture image on the most similar face shape model (step S380).

  As described above, the face image processing apparatus 200 to which the present invention is applied changes the relative positional relationship between facial feature points in a non-linear manner with respect to a three-dimensional shape model prepared in advance. Then, the face image processing apparatus 200 changes the two-dimensional face into the one obtained by modifying the three-dimensional shape model so that the three-dimensional face feature point on the three-dimensional shape model and the corresponding face feature point on the two-dimensional face image most closely match. Since the subject's three-dimensional face model is created by combining the images, the subject's two-dimensional face model can be obtained without using a special surface measurement device or collecting face shape data of a large number of people in advance. From the face image, it is possible to create a three-dimensional face model that well reproduces the three-dimensional shape of the subject's face. In addition, a variety of derived shape models can be created by preparing multiple 3D shape models, providing a derived shape model in which the position of each facial feature point matches well with the 2D face image of the subject. More likely to be able to do it.

Next, a face image registration apparatus using the face image processing apparatus 200 will be described. Such a face image registration device creates a plurality of reference face images with different face orientations, illumination conditions or facial expressions from a three-dimensional shape model created based on the face image of the subject. The face image processing apparatus 200 itself may be provided with the functions of the following parts of the face image registration apparatus.
FIG. 4 shows a schematic configuration diagram of such a face image registration apparatus 400. The face image registration device 400 includes a three-dimensional face model creation unit 410 corresponding to the face image processing device 200, a variable face image creation unit 420, a face image registration unit 430, and a rendering condition storage unit 440. Here, the rendering condition storage unit 440 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device. The rendering condition storage unit 440 can also be configured by the same hardware as the storage unit 250 of the three-dimensional face model creation unit 410 (face image processing apparatus 200). The variable face image creation unit 420 and the face image registration unit 430 are functional modules implemented by a microprocessor, a memory, a peripheral circuit thereof, and software operating on the microprocessor. Furthermore, the fluctuating face image creation means 420 and the face image registration means 430 may operate on a microprocessor in which each functional module of the three-dimensional face model creation means 410 (face image processing device 200) is mounted. .
FIG. 5 shows an operation flowchart of reference face image creation processing by the face image registration apparatus 400. Hereinafter, each part of the face image registration apparatus 400 and reference face image creation processing will be described with reference to FIGS. 4 and 5.

  For example, when the face image registration apparatus 400 acquires the target person's two-dimensional face image 100 from the history information of the face image photographed by a surveillance camera or the like, the three-dimensional face model creation means 410 causes the target person's three-dimensional face image 100 to be obtained. A face model is created (step S510). Note that the three-dimensional face model creation unit 410 has the same configuration and function as the face image processing apparatus 200 described above, and thus detailed description thereof is omitted here.

  Next, the fluctuating face image creation unit 420 of the face image registration device 400 reads a plurality of rendering conditions from the rendering condition storage unit 440. Then, the varying face image creating means 420 creates a plurality of varying face images in which the face direction, lighting conditions, facial expressions, and the like are changed from the three-dimensional face model of the subject in accordance with each rendering condition (step S520). The rendering condition includes an illumination condition that defines the face direction, the illumination direction, and the luminance of the illumination light, and facial expression information that specifies an expression. Further, since various known methods can be used as the rendering method, detailed description of the rendering method is omitted.

  Then, the face image registration unit 430 registers each created variation face image, for example, as a reference face image of the face authentication device in association with the identification information of the target person (step S530). The face image registration device 400 may store the registered reference face image in a storage device provided outside the face image registration device 400, or the three-dimensional face model creation unit 410 (the face image processing device 200). ) May be stored in the storage means 250.

  As described above, the face image registration device to which the present invention is applied creates a reference face image according to an arbitrary rendering condition from a three-dimensional face model of a subject created without using a special surface measurement device. A large number of reference face images can be easily created by arbitrarily changing the face direction, lighting conditions, facial expressions, and the like.

Next, a face authentication apparatus according to another embodiment to which the present invention is applied will be described. The face authentication apparatus includes a face image having a predetermined orientation, lighting conditions, and facial expression created from a three-dimensional shape model created based on the face image of the subject, and a reference face image of the registrant registered in advance. The collation is performed to determine whether or not to authenticate the target person. The face image processing apparatus 200 itself may be provided with the functions of the following face authentication apparatuses.
FIG. 6 shows a schematic configuration diagram of such a face authentication apparatus 600. The face authentication apparatus 600 includes an image acquisition means 610, a three-dimensional face model creation means 620 corresponding to the face image processing apparatus 200, an input face image correction means 630, a face image collation means 640, and a reference face image storage. Means 650. Here, the reference face image storage means 650 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device. The reference face image storage unit 650 can also be configured by the same hardware as the storage unit 250 of the three-dimensional face model creation unit 620 (face image processing apparatus 200). The input face image correcting unit 630 and the face image collating unit 640 are functional modules implemented by a microprocessor, a memory, a peripheral circuit thereof, and software operating on the microprocessor. Further, the input face image correcting unit 630 and the face image collating unit 640 may operate on a microprocessor in which each functional module of the three-dimensional face model creating unit 620 (face image processing apparatus 200) is mounted. .
FIG. 7 shows an operation flowchart of authentication processing by the face authentication apparatus 600. Hereinafter, each unit and authentication processing of the face authentication apparatus 600 will be described with reference to FIGS. 6 and 7.

First, the two-dimensional face image 100 of the subject is acquired by the image acquisition unit 610 (step S710). The image acquisition unit 610 includes, for example, a detector having a two-dimensional array of photoelectric conversion elements such as a CCD and a C-MOS sensor, an optical system that forms an image on the detector, a processor, and its peripheral circuits. The image acquisition unit 610 forms an image of a predetermined area on the detector using an optical system, and acquires image data obtained by photographing the predetermined area. The processor of the image acquisition unit 610 acquires the face image 100 by cutting out an area corresponding to the face of the subject who entered the predetermined area from the obtained image data. Here, the extraction of the area corresponding to the face of the subject can be performed using various known methods. For example, a template corresponding to the face is prepared in advance, and the image acquisition unit 610 performs template matching with the entire image data using the template to calculate the degree of coincidence. Then, the image acquisition unit 610 cuts out an area in which the degree of coincidence of a predetermined level or more is obtained from the image data, and sets it as the face image 100.
Note that the image acquisition unit 610 may cut out only one face image from one piece of image data, or may cut out a plurality of face images from one piece of image data.

  Next, the 3D face model creation means 620 of the face authentication apparatus 600 creates the 3D face model of the subject from the 2D face image 100 of the subject (step S720). Note that the three-dimensional face model creation unit 620 has the same configuration and function as the face image processing apparatus 200 described above, and thus detailed description thereof is omitted here. Then, the input face image correction unit 630 of the face authentication device 600 rotates the subject's three-dimensional face model in a virtual three-dimensional space so that the subject's face faces a predetermined direction (for example, the front direction), Rendering as a two-dimensional image creates an input corrected face image (step S730). Note that when rendering, the input face image correcting unit 630 includes a specific part of the face (for example, the whole or part of the skin portion of the face, or the nose, eyes, The illumination condition may be adjusted so that the average luminance of the mouth and the like and the average luminance of the corresponding specific part included in the input corrected face image are substantially equal. Furthermore, the subject's facial expression may be adjusted to be a predetermined facial expression. Further, since various known methods can be used as the rendering method, detailed description of the rendering method is omitted.

When the input corrected face image is created, the face image collating unit 640 of the face authentication apparatus 600 collates the input corrected face image with each reference face image read from the reference face image storage unit 650 (step S740). Each reference face image is a registrant's face image registered in advance in the face authentication device 600. For example, the registrant's face is an image in a state where the face of the registrant faces the predetermined direction (for example, the front direction). . Further, the reference face image storage unit 650 is configured so that the orientation of the face of the subject in the input corrected face image deviates from the predetermined direction described above, or the illumination condition or facial expression of the subject when the face image 100 is acquired. Each registrant may have a plurality of reference face images in which the face direction, lighting conditions, and facial expressions are changed so that the matching can be accurately performed in accordance with the difference. Each reference face image can be created in advance by, for example, acquiring a registrant's face image using the face image registration device 400 described above. Alternatively, each reference face image may be an image obtained by photographing the registrant's face with a different angle, a different lighting condition, or a different facial expression. Further, various known collation methods can be used as the collation processing. For example, the face image matching unit 640 performs pattern matching between the input corrected face image and the reference face image. Then, the face image matching unit 640 calculates the number of pixels included in the face area of the input correction face image by calculating the sum of squares of the luminance values of each pixel in the face area included in the input correction face image and the corresponding pixel of the reference face image. The reciprocal of the normalized value divided by is obtained as the degree of coincidence. When the highest matching score among the matching scores obtained for each reference face image exceeds a predetermined reference value, the face image matching unit 640 corresponds the target person to the reference face image having the highest matching score. Authenticate as a registrant. On the other hand, the face image collating unit 640 does not authenticate the target person when any degree of coincidence does not exceed the predetermined reference value.
The authentication result is output from the face authentication apparatus 600, and is used in a security system that opens an entrance only when authentication is successful, for example.

  As described above, the face authentication device to which the present invention is applied deforms a three-dimensional shape model according to the position of a characteristic part of the subject's face, and obtains a two-dimensional face image of the model and the subject's face. Since matching is performed based on the synthesized three-dimensional face model, the matching accuracy can be increased. Furthermore, since the face authentication apparatus uses an input corrected face image adjusted to approximately the same face orientation as any of the reference face images for verification, it is possible to accurately authenticate regardless of which direction the subject faces at the time of shooting. it can.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, each three-dimensional shape model created by categorizing a number of face shape models and averaging the face shape models in the same category so that faces with high similarity in shape are in the same category May be used. Then, the alignment information calculation unit 230 may create a derived shape model for each three-dimensional shape model and calculate alignment information for each derived shape model.

The derived model creating means 231 uses a facial feature point other than the nose tip as a reference point, and changes the position of the other facial feature point so that the distance from the reference point is enlarged or reduced. May be created. For example, the reference point may be set to the nose root point. Then, the derived model creation means 231 changes the distance from the nose root point to a face feature point above the nose root point (for example, the eyebrow point), or the face feature point below the nose root point (for example, from the nose root point). The derived shape model may be created by changing the distance to the nose tip, mouth point, and genital point. The reference point may be set as the mouth point. Then, the derived model creating means 231 modifies the three-dimensional shape model so that the ratio between the length from the mouth point to the mental point and the length from the mouth point to the nose tip or root point is changed to obtain the derived shape model. You may create it.
Furthermore, the derived model creation means 231 may create a derived shape model in which the nose height of the three-dimensional shape model is changed. In this case, for example, the derived model creating means 231 may use the w-axis direction (see above) so that the nose height of the derived shape model is 0.8 to 1.2 times the nose height of the original three-dimensional shape model. That is, the three-dimensional shape model is enlarged or reduced along the front-rear direction of the face.

In addition, the derived model creating means 231 may create a derived shape model that not only changes the position of the characteristic parts such as the eyebrows, eyes, nose, and mouth inside the face but also changes the contour shape of the face. In particular, when the face authentication device to which the present invention is applied uses the face contour shape as matching information at the time of matching, the face contour shape in the input correction face image used for matching is also changed to the contour shape of the subject's face. Since it can be approximated, the authentication accuracy can be improved. When creating a derived shape model in which the contour shape of the face is also changed, the derived model creating means 231 changes the vertical enlargement / reduction rate above and below the nose apex, which is the reference point, in the same manner as other face feature points. Thus, the contour shape of the face can be changed. The derived model creating means 231 may deform the face contour shape separately from the other face feature points, and then combine them to form a derived shape model. In this case, in order to deform the contour shape of the face, a reference point is a portion that is less likely to be hidden by the hair, for example, the midpoint of the left and right jaw angle points. Alternatively, the center of gravity of the entire face area may be used as the reference point. The derived model creating means 231 changes the contour shape of the face by changing the vertical enlargement / reduction ratio above and below the reference point. Further, the derived model creation means 231 may change the horizontal enlargement / reduction ratio and the vertical enlargement / reduction ratio for the facial contour shape. By performing such deformation, it is possible to create derivative shape models having various face contour shapes such as round faces and thin faces.
As described above, various modifications can be made within the scope of the present invention according to the embodiment to be implemented.

1 is a schematic configuration diagram of an image processing apparatus to which the present invention is applied. It is a figure which shows the coordinate system used as the reference | standard of derivative shape model preparation. It is a flowchart which shows an example of the three-dimensional face model creation process in the image processing apparatus to which this invention is applied. It is a schematic block diagram of the face image registration apparatus to which this invention is applied. It is a flowchart which shows an example of the reference face image registration process in the face image registration apparatus to which this invention is applied. It is a schematic block diagram of the face authentication apparatus to which this invention is applied. It is a flowchart which shows an example of the authentication process in the face authentication apparatus to which this invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Face image 200 Face image processing apparatus 210 Image input means 220 Face feature point extraction means 230 Registration information calculation means 231 Derived model creation means 232 Position alignment means 233 Similar shape selection means 240 Individual model creation means 250 Storage means 400 Face image registration Apparatus 410 3D face model creation means (face image processing apparatus)
420 Fluctuating face image creation means 430 Face image registration means 440 Rendering condition storage means 600 Face authentication device 610 Image acquisition means 620 3D face model creation means (face image processing device)
630 Input face image correcting means 640 Face image collating means 650 Reference face image storing means

Claims (5)

A face image processing device that generates a 3D face image corresponding to a face of a person by synthesizing a 2D face image including the face of the person and a 3D shape model representing the 3D shape of the face,
Image input means for inputting the two-dimensional face image;
Storage means for previously storing the three-dimensional shape model and facial feature point positions in the three-dimensional shape model;
From the three-dimensional shape model to the feature point above the reference point with respect to a reference point set at a position where the feature point exists at least above and below the midline of the face surface of the three-dimensional shape model A derived model creating means for creating a derived shape model in which the ratio of the length of the reference point to the feature point below the reference point is different from the three-dimensional shape model;
Selecting means for comparing the derived shape model and the three-dimensional shape model with the two-dimensional face image to select the one most similar to the two-dimensional face image;
A personal model creating means for creating the 3D face image corresponding to the face of the person by combining the 2D face image with the selected derived shape model or the 3D shape model;
A face image processing apparatus comprising:   The face image processing apparatus according to claim 1, wherein the reference point is a nose tip.   The derivation model creation means is different in the ratio of the distance along the midline of the face from the nose tip to the feature point representing the nose root point or the center position of both eyes and the distance from the nose tip to the feature point representing the mouth point. The face image processing apparatus according to claim 2, wherein the at least one derived shape model is created.   2. The derived model creating means creates the at least one derived shape model by deforming a face contour shape of the three-dimensional shape model so that a ratio of lengths in a horizontal direction and a vertical direction is changed. The face image processing apparatus according to claim 1. Furthermore, a reference face image storage means for storing a reference face image of at least one registrant facing a predetermined direction;
A corrected face image creating means for creating a corrected face image in which the face of the person faces the predetermined direction by rotating the face direction of the three-dimensional face image in a three-dimensional space;
Collating means for collating the corrected face image with the reference face image and determining whether to authenticate the person as the registrant;
The face image processing apparatus according to any one of claims 1 to 4, further comprising:

Images