JP2012185545A - Face image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a face image processing device creating a three-dimensional face image of a target person in which, based on a two-dimensional face image of the target person, feature parts are satisfactorily placed as a real face of the target person by using a three-dimensional shape model indicating a three-dimensional shape of a face without using a special surface measuring device.SOLUTION: A face image processing device 100 has: position matching means 111 for matching positions of two-dimensional face feature points with positions of three-dimensional face feature points; derivation feature point calculation means 112 for generating a derivation face feature point by changing a position of the three-dimensional face feature point of which a position gap amount from the corresponding two-dimensional face feature point is equal to or more than a determined threshold; derivation model creation means 105 for creating a derivation shape model by combining the three-dimensional face feature points and the derivation face feature point; similar shape selection means 109 for selecting a shape model which is the most similar to the two-dimensional face image from the derivation shape model and a three-dimensional shape model; and personal model creation means 110 for creating a three-dimensional face image by synthesizing the selected derivation shape model or three-dimensional shape model with the two-dimensional face image.

Description

  The present invention relates to a face image processing apparatus, and more particularly to a face image processing apparatus that creates a 3D face image by mapping a 2D face image obtained by photographing an individual to 3D face shape data.

  Conventionally, there has been proposed a face authentication device that authenticates a target person by collating a two-dimensional face image acquired by photographing the face of the target person with a registered face image. Such a face authentication apparatus registers the face image of the subject person in advance, and determines whether or not to authenticate based on the result of comparison with the face image acquired at the time of use. For this reason, if there are differences in the face orientation, lighting conditions, facial expressions, and the like between the face image at the time of registration and the face image at the time of use, there is a problem that the error rate at the time of matching increases.

  Therefore, Patent Document 1 proposes a sample image collection method for creating a plurality of two-dimensional images of a subject's face for use in collating face images. In this image collection method, a three-dimensional computer graphics model of a subject's face is created from the acquired three-dimensional surface shape data and color information of the face using a surface measuring device. Then, the image collection method creates a plurality of two-dimensional face images by rendering with the face orientation or illumination conditions varied based on the model.

  Further, Patent Document 2 proposes a face image matching device that creates a plurality of reference face images to be used for matching using a standard frame model representing a standard three-dimensional shape of a face. This face image matching device generates a three-dimensional face model (three-dimensional face image) corresponding to the face of the subject by combining the two-dimensional face image of the subject acquired at the time of matching with the standard frame model. . Then, the face image matching device creates a reference face image by rendering the three-dimensional face model in consideration of the variation factors such as the face direction, the illumination condition, or the expression.

  Patent Document 3 discloses a face for generating a three-dimensional face image representing a face shape of a subject using a derived shape model obtained by deforming a three-dimensional shape model representing a three-dimensional shape of a face prepared in advance. An image processing apparatus has been proposed. This face image processing apparatus creates a derived shape model obtained by transforming a three-dimensional shape model so as to change the ratio of the height direction or the vertical length with respect to a reference point such as a nose tip. Then, the face image processing device combines the 2D face image with the model most similar to the 2D face image of the subject acquired from the image input means from the derived shape model or the 3D shape model, and generates the 3D face image. create.

JP-A-4-256185 JP 2003-6645 A JP 2009-2111148 A

  In the image collection method described in Patent Document 1, it is necessary to create a three-dimensional computer graphics model of a face for each subject. However, since it is necessary to measure the face of the subject person using a special surface measuring device in order to create the model, it is not easy to create a three-dimensional computer graphics model of each subject person's face. For this reason, it has been difficult to apply such an image collection method to a commonly used face authentication apparatus.

  On the other hand, in the face image matching apparatus described in Patent Document 2, one or more standard frame models corresponding to a standard face are prepared as models representing a three-dimensional shape. This standard frame model only needs to be created once and can be used for any of such face image collation apparatuses, so the above-mentioned problems do not occur. However, since the standard frame model corresponds to a standard face shape, it does not reflect the characteristics of the face shape of each person such as the extent of the eyebrows or cheekbones and the shape of the jaw. Patent Document 2 also discloses a method of selecting a standard frame model according to attribute information such as the sex and age of the subject as a standard frame model used for creating a three-dimensional face model. However, even if the attribute information of the subject is the same, the facial shape characteristics may be different. Therefore, a face image processing apparatus that can create a three-dimensional face model suitable for the face of the subject is further desired.

  Further, the face image processing apparatus described in Patent Document 3 creates a derived shape model by deforming a three-dimensional shape model in accordance with the position of a characteristic part of the subject's face. A suitable three-dimensional face image can be created. However, this derived shape model is obtained by changing only the ratio of the height direction or the vertical length to the reference point with respect to the three-dimensional shape model prepared in advance, and the position of each characteristic part. There is a further demand for a face image processing apparatus that can create a three-dimensional face image with better matching.

  Therefore, an object of the present invention is to use a three-dimensional shape model representing the three-dimensional shape of the face without using a special surface measuring device, and to determine the position of a characteristic part from the two-dimensional face image of the subject. An object of the present invention is to provide a face image processing apparatus that creates a three-dimensional face image that is more closely matched to the face of the subject.

  To solve this problem, the present invention creates a three-dimensional face image corresponding to the face of a person by synthesizing a two-dimensional face image including the face of the person and a three-dimensional shape model representing the three-dimensional shape of the face. A face image processing apparatus is provided. The face image processing apparatus includes an image input unit that inputs a two-dimensional face image, and a storage unit that stores in advance a three-dimensional shape model and a plurality of three-dimensional face feature points representing facial feature points in the three-dimensional shape model. A face feature point extracting means for extracting a plurality of 2D face feature points representing face feature points from the 2D face image, a 3D face feature point and a 2D face feature point corresponding to the 3D face feature point A positioning unit that calculates a positional deviation amount with respect to the position and performs positioning, and generates one or a plurality of derived face feature points whose positions are changed with respect to a three-dimensional face feature point having a positional deviation amount equal to or greater than a predetermined threshold, For the 3D face feature point, select the feature point selected from the 3D face feature point or the derived face feature point, for the 3D face feature point whose positional deviation amount is less than a predetermined threshold, Derived face combined to form the shape of Derived face feature point calculating means for generating a scoring point set and a derived model for generating a derived shape model by modifying the 3D shape model so that the 3D face feature point of the 3D shape model matches the derived face feature point set Creating means; comparing derived shape model and 3D shape model with 2D face image; selecting derived shape model or 3D shape model most similar to 2D face image; and selected derivation And a personal model creation means for creating a 3D face image corresponding to a human face by synthesizing a shape model or a 3D shape model and a 2D face image.

  Furthermore, in the face image processing apparatus according to the present invention, the derived face feature point calculation means corresponds to the three-dimensional face feature point as the positional deviation amount of the three-dimensional face feature point with respect to the corresponding two-dimensional face feature point increases. It is preferable that the derived face feature points are generated by being widely distributed.

  Furthermore, in the face image processing apparatus according to the present invention, the derived face feature point calculation means corresponds to the three-dimensional face feature point as the positional deviation amount of the three-dimensional face feature point with respect to the corresponding two-dimensional face feature point increases. It is preferable to generate the derived face feature points by increasing the number.

  The face image processing apparatus according to the present invention uses a three-dimensional shape model representing the three-dimensional shape of the face without using a special surface measurement device, and positions of characteristic parts from the two-dimensional face image of the subject. It is possible to create a three-dimensional face image that more closely matches the face of the subject.

1 is a schematic configuration diagram of an image processing apparatus to which the present invention is applied. It is a schematic diagram showing the relationship between a face image and a three-dimensional shape model. It is a graph which shows the relationship between the distance and the amount of positional deviation between a 3D face feature point and the projection line of a 2D face feature point. It is a schematic diagram which shows a derived face feature point and a derived face feature point set. It is a flowchart which shows an example of the three-dimensional face image creation process in the image processing apparatus to which this invention is applied.

Hereinafter, a face image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
When the face image processing apparatus to which the present invention is applied acquires the two-dimensional face image of the subject, the face image in the three-dimensional shape model representing the three-dimensional shape of the face prepared in advance from the two-dimensional face image. A feature point representing the feature point is extracted. Then, the face image processing apparatus performs alignment so that the arrangement of the feature points of the three-dimensional shape model most closely matches the arrangement of the feature points of the two-dimensional face image. A plurality of derived face feature points are generated by changing the positions of feature points whose positional deviation from the feature points of the face image is greater than or equal to a predetermined value, and a derived shape model is created using the derived face feature points. Then, the face image processing device creates a three-dimensional face image based on the derived shape model and the original three-dimensional shape model that are most similar to the two-dimensional face image. A three-dimensional face image is created in which the positions of the characteristic parts are well matched.

FIG. 1 is a diagram showing a schematic configuration of a face image processing apparatus 100 to which the present invention is applied. As shown in FIG. 1, the face image processing apparatus 100 includes a storage unit 101, an image input unit 102, a face feature point extraction unit 103, a registration information calculation unit 104, a derived model creation unit 105, a light source direction estimation unit 106, a shadow, and the like. An image creation unit 107, a similarity calculation unit 108, a similar shape selection unit 109, and a personal model creation unit 110 are included.
Among these, the facial feature point extraction means 103, the alignment information calculation means 104, the derived model creation means 105, the light source direction estimation means 106, the shadow image creation means 107, the similarity calculation means 108, the similar shape selection means 109, and the personal model creation. Each means 110 is a functional module implemented by a microprocessor, a memory, a peripheral circuit thereof, and software operating on the microprocessor. Alternatively, these means may be integrated by firmware. Moreover, you may comprise some or all of these means with an independent electronic circuit, firmware, a microprocessor, etc. Hereinafter, each part of the face image processing apparatus 100 will be described in detail.

  The storage unit 101 includes a semiconductor memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), or a magnetic recording medium and its access device or an optical recording medium and its access device. The storage unit 101 stores a computer program for controlling the face image processing apparatus 100, various parameters, data, and the like. The storage unit 101 also stores a three-dimensional shape model of the face and a plurality of three-dimensional face feature points representing the face feature points in the three-dimensional shape model. There may be a plurality of three-dimensional shape models or one. The three-dimensional shape model is a frame model representing the three-dimensional shape of a human face, and a wire frame model, a surface model, or the like is used. The three-dimensional shape model may be created, for example, from the face shapes of a plurality of persons, or may be created by averaging face shape models that imitate the face shapes of many persons. Alternatively, many face shape models may be categorized so that faces having a high degree of similarity in the same category, and the face shape models in the same category may be averaged.

  Here, the three-dimensional face feature point represents any point of a characteristic part different from others in terms of shape or color component such as eyes, nose and mouth, for example, a center point or an end point of those parts. For example, the three-dimensional face feature points include an eyebrow head, an eyebrow butt, a black eye center, an eye region center, an eye head, an eye corner, a nose tip, a nostril center, a mouth point (mouth center), a mouth corner point, and the like. The type and number of the 3D face feature points to be stored are not limited, but the 3D face feature points to be stored are the same parts as the face feature points of the 2D face image that can be extracted by the face feature point extraction unit 103. Preferably, it is a feature point, or at least all of these feature points are included. In the present embodiment, as the three-dimensional face feature points, the left and right eye region centers, the eyes and the corners of the eyes, the nose apex point, the mouth point, and the left and right mouth corner points are stored.

The image input unit 102 is an interface circuit connected to an image acquisition unit such as a surveillance camera, for example, and inputs the face image of the subject acquired as a two-dimensional face image by the image acquisition unit to the face image processing apparatus 100. . In addition, when there is a recording medium that records history information including one or more face images taken in advance, the image input unit 102 includes a reading device for accessing such a recording medium, a keyboard, a mouse, and the like. And a user interface including a display. In this case, the user selects any one face image from the history information via the user interface of the image input unit 102. Note that the face image may include the entire face, and the face direction may be front or oblique as long as each feature part (eyes, nose, mouth, etc.) of the face can be distinguished from other feature parts. Furthermore, the face image can be a grayscale or color multi-tone image, but is preferably a color multi-tone image that easily extracts the characteristics of the human skin. In this embodiment, the face image is a color image having 128 × 128 pixels and having a luminance resolution of 8 bits for each color of RGB. However, a face image having a resolution and gradation other than this embodiment may be used.
The image input means 102 outputs the acquired face image to the face feature point extraction means 103, the light source direction estimation means 106, the similarity calculation means 108, and the personal model creation means 110.

  The face feature point extracting unit 103 extracts a face feature point representing a face feature point (hereinafter referred to as a two-dimensional face feature point) from the face image output from the image input unit 102. Then, the face feature point extraction unit 103 uses the extracted two-dimensional face feature point type and position information on the face image (for example, a two-dimensional coordinate value with the upper left corner of the face image as the origin) as a registration information calculation unit. To 104. In the present embodiment, the face feature point extraction unit 103 is a two-dimensional face feature point corresponding to each three-dimensional face feature point (10 locations such as the center of the eye region, the tip of the nose, the corner of the mouth) stored in the storage unit 101. To extract. The face feature point extraction unit 103 can use various known methods for extracting a two-dimensional face feature point from a face image. For example, the face feature point extraction unit 103 performs edge extraction processing on the face image to extract edge pixels having a large luminance difference from surrounding pixels. Then, the face feature point extraction unit 103 checks whether or not the feature amount obtained based on the position and pattern of the edge pixel satisfies a predetermined condition for the part such as the eyes, nose, and mouth. By specifying the position, each two-dimensional face feature point can be extracted. Also, the face feature point extraction unit 103 performs Gabor transform processing or wavelet transform processing instead of performing edge extraction processing to extract edge pixels, and extracts pixels having large local changes in different spatial frequency bands. May be. Further, the face feature point extraction unit 103 may extract a two-dimensional face feature point by performing template matching between a template corresponding to each part of the face and a face image and specifying the position of each part of the face. . Furthermore, the face feature point extraction means 103 allows each two-dimensional face feature point to be specified by allowing the user to specify the position of each two-dimensional face feature point via a user interface (not shown) including a keyboard, a mouse, and a display. Face feature points may be acquired.

  The alignment information calculation unit 104 aligns the three-dimensional shape model stored in the storage unit 101 with the two-dimensional face image input from the image input unit 102, and a derived face feature used in a derived model creation unit 105 described later. Generate a set of points. The alignment information calculation unit 104 is configured such that the arrangement of the three-dimensional face feature points of the three-dimensional shape model stored in the storage unit 101 is the position of each two-dimensional face feature point extracted from the face image by the face feature point extraction unit 103. Align to best match the placement. The alignment information calculation unit 104 aligns the derived shape model created by the derived model creation unit 105 based on the derived face feature point set with the two-dimensional face image using the derived face feature point set. Details of the derived face feature point set will be described later. Then, the alignment information calculation unit 104 outputs the alignment information obtained as a result of the alignment to the derived model creation unit 105. For this purpose, the alignment information calculation unit 104 includes an alignment unit 111 and a derived face feature point calculation unit 112.

The alignment means 111 rotates, translates, enlarges / reduces each 3D face feature point of the 3D shape model, and the arrangement of each 3D face feature point most closely matches the arrangement of each 2D face feature point. Alignment is performed so that the alignment information is calculated. At this time, the alignment unit 111 maintains the relative positional relationship between the three-dimensional face feature points before and after the alignment, that is, the arrangement of the three-dimensional face feature points before the alignment and the position after the alignment. Each 3D face feature point is moved so that the arrangement of each 3D face feature point is similar to each other.
The alignment information includes, for example, a rotation angle along each axis of a three-dimensional orthonormal coordinate system (X, Y, Z) set for the three-dimensional shape model, a translation amount, and an enlargement / reduction ratio. . In this orthonormal coordinate system (X, Y, Z), for example, the center of gravity of a plurality of three-dimensional face feature points on the three-dimensional shape model is set as the origin, and the X axis in the direction from left to right with respect to the face The Y axis is set in a direction horizontal to the face and from the rear to the front, and the Z axis is set in the direction from the bottom to the top perpendicular to the face.

Then, the registration unit 111 uses the registration information to evaluate the registration information, and the registration error when the registration of the two-dimensional face feature points corresponding to the three-dimensional face feature points is performed. Is calculated.
The alignment unit 111 calculates an alignment error as follows, for example. First, the alignment unit 111 projects each two-dimensional face feature point on the three-dimensional space of the three-dimensional shape model.
FIG. 2 schematically shows the relationship between the face image and the three-dimensional shape model. As shown in FIG. 2, the alignment unit 111 projects the two-dimensional face feature points 11 and 12 of the face image 10 onto the three-dimensional space of the three-dimensional shape model 20 with the camera position 30 as the origin. The alignment unit 111 determines that the 3D face feature points 21 and 22 are based on the positional relationship between the 3D face feature points 21 and 22 and the projection lines 31 and 32 obtained by projecting the 2D face feature points 11 and 12 onto the 3D space with the camera position 30 as the origin. The arrangement of the feature points of the three-dimensional shape model 20 and the arrangement of the feature points of the face image 10 can be compared.

In the present embodiment, the three-dimensional shape model stored in the storage unit 101 assumes a model representing a standard three-dimensional shape of a face, and is not a model imitating the face shape of a subject. Therefore, the arrangement of the three-dimensional face feature points 21 and 22 and the arrangement of the two-dimensional face feature points 11 and 12 are generally different. Therefore, a feature point having a shortest distance from the projection line 31 such as the three-dimensional face feature point 21 at the left eye corner and a feature point having a large shortest distance from the projection line 32 such as the three-dimensional face feature point 22 at the left mouth corner. And exist. For this reason, it is generally impossible to perform alignment so that the arrangement of all the three-dimensional face feature points completely matches the arrangement of the two-dimensional face feature points.
On the other hand, when all the 3D face feature points are evenly aligned with the corresponding 2D face feature points, the desired 3D face image has a texture quality that is evenly shifted from the face shape. It will be bad. Therefore, the evaluation of the shortest distance from the projected straight line will be emphasized.

位置合わせ手段１１１は、この全体の位置ずれ量Eが最小となるように、各３次元顔特徴点を３次元の正規直交座標系(X,Y,Z)の各軸に沿って回転または並進させたり、拡大または縮小させたりする。位置合わせ手段１１１は、例えば、シミュレーティッドアニーリングまたは最急降下法などの最適化法を用いて、全体の位置ずれ量Eが最小となるときの３次元形状モデルの姿勢変化量（回転角、並進量、及び拡大／縮小率）を求めることができる。位置合わせ手段１１１は、上記の姿勢変化により最小となった全体の位置ずれ量Eを、３次元顔特徴点の２次元顔特徴点に対する位置合わせ誤差とする。 The alignment unit 111 calculates a positional deviation amount E _i for the two-dimensional face feature point corresponding to the target three-dimensional face feature point with respect to the target three-dimensional face feature point, and calculates the sum of the positional deviation amounts E _i . The total positional deviation amount E is obtained. The total positional deviation amount E is expressed by the following equation, for example.

The alignment means 111 rotates or translates each three-dimensional face feature point along each axis of the three-dimensional orthonormal coordinate system (X, Y, Z) so that the total displacement E is minimized. Or zoom in or out. The alignment unit 111 uses, for example, an optimization method such as simulated annealing or steepest descent method to change the attitude change amount (rotation angle, translation amount) of the three-dimensional shape model when the total displacement E is minimized. , And enlargement / reduction ratio). The alignment unit 111 sets the total positional deviation amount E, which is minimized by the above-described posture change, as the alignment error of the three-dimensional face feature point with respect to the two-dimensional face feature point.

（２）式において、d_iは、注目する３次元顔特徴点と、その注目する３次元顔特徴点に対応する２次元顔特徴点の投影直線との最短距離である。この場合、３次元空間における２次元顔特徴点に対応する点（以下、３次元投影点と称する）は、２次元顔特徴点の投影直線上の、その２次元顔特徴点に対応する３次元顔特徴点に最も近い点となる。またσは、距離d_iと位置ずれ量E_iの関係を定める係数である。 Further, in the equation (1), the positional deviation amount E _i is expressed by the following equation, for example.

In the equation (2), d _i is the shortest distance between the noticeable three-dimensional face feature point and the projection line of the two-dimensional face feature point corresponding to the noticed three-dimensional face feature point. In this case, a point corresponding to a 2D face feature point in the 3D space (hereinafter referred to as a 3D projection point) is a 3D corresponding to the 2D face feature point on the projection line of the 2D face feature point. The point closest to the face feature point. Σ is a coefficient that determines the relationship between the distance d _i and the positional deviation amount E _i .

FIG. 3 shows the relationship between the distance d _i and the positional deviation amount E _i . In FIG. 3, the horizontal axis represents the distance d _i , the vertical axis represents the positional deviation amount E _i , the graph 301 is for σ = 0.0017, the graph 302 is for σ = 0.017, and the graph 303 is for σ = 0.17. The relationship between the distance d _i and the positional deviation amount E _i is shown.
As shown in FIG. 3, the variation of the displacement amount E _i to changes in d _i will be larger in the region d _i is smaller than the area d _i is large. That distance d _i than closer more the large feature points, better to adjust the distance d _i is smaller feature points to be more closer is decline of the displacement amount E _i is increased, the whole positional deviation amount E It can be made smaller.
In FIG. 3, for example, in the region of distance d _i > 0.2, the positional deviation amount E _i of the graph 301 is asymptotic to 1, but the positional deviation amount E _i of the graph 303 is not asymptotic. That is, when σ is small, the positional deviation amount E _i gradually approaches a predetermined value in a region where the distance d _i is small. However, when σ is large, the positional deviation amount E _i does not asymptotic to a predetermined value unless the distance d _i is large. Thus, the relationship between the distance d _i and the positional deviation amount E _i can be adjusted by the value of σ, and the value of σ is appropriately determined according to the environment where the face image processing apparatus 100 is installed, its purpose, and the like. It is done. In this embodiment, σ = 0.018.

As described above, the alignment unit 111 performs alignment so as to minimize the total positional deviation amount E using the positional deviation amount E _i calculated by the expression (2), thereby reducing the three-dimensional distance d _i. The face feature points are preferentially aligned so as to approach the corresponding three-dimensional projection points. Thus the positioning means 111 may be aligned so as to match (2) about 3D face feature points calculated positional displacement amount E _i is smaller by formula, more corresponding two-dimensional face feature point.

（３）式及び（４）式において、αは、それぞれ距離d_iと位置ずれ量E_iの関係を定める係数であり、αの値は、顔画像処理装置１００が設置される環境、その目的などに応じて適宜定められる。
また距離d_iは、注目する３次元顔特徴点と、その注目する３次元顔特徴点に対応する２次元顔特徴点の投影直線との最短距離に限られない。例えば距離d_iは、注目する３次元顔特徴点と、その注目する３次元顔特徴点に対応する２次元顔特徴点の投影直線と３次元形状モデルの表面の交点との距離としてもよい。この場合、２次元顔特徴点の投影直線と３次元形状モデルの表面の交点が３次元投影点となる。
なお、鼻などの顔上に突起した部位に隠れて、あるいは遮蔽物などにより顔の一部が見えなくなるオクルージョンが発生し、顔画像において顔の一部分の情報が欠落しており、２次元特徴点の一部が抽出できないときは、対応すべき３次元特徴点については位置ずれ量E_iを算出しないこととする。 The positional deviation amount E _i may be any as long as it is calculated such that the smaller the distance d _i is, the smaller the amount of change in the positional deviation amount E _{i with} respect to the change in the distance d _i is. For example, you may use the sigmoid function represented by (3) Formula, or the function represented by (4) Formula.

In the equations (3) and (4), α is a coefficient that determines the relationship between the distance d _i and the positional deviation amount E _i , and the value of α is the environment in which the face image processing apparatus 100 is installed, its purpose It is determined appropriately according to the above.
The distance d _i is not limited to the shortest distance between the focused 3D face feature point and the projection line of the 2D face feature point corresponding to the focused 3D face feature point. For example, the distance d _i may be the distance between the 3D face feature point of interest and the intersection of the projection line of the 2D face feature point corresponding to the 3D face feature point of interest and the surface of the 3D shape model. In this case, the intersection of the projection line of the two-dimensional face feature point and the surface of the three-dimensional shape model becomes the three-dimensional projection point.
In addition, there is an occlusion that is hidden behind a protruding part on the face such as the nose or that part of the face cannot be seen due to an obstruction, etc., and information on the part of the face is missing in the face image. when a portion of the can not be extracted, the three-dimensional feature points should correspond to not calculate the positional shift amount E _i.

The alignment means 111 includes a rotation angle, a translation amount, and a translation amount of each three-dimensional face feature point along each axis of the orthonormal coordinate system (X, Y, Z) when the total positional deviation amount E is minimized. And the enlargement / reduction ratio is used as alignment information. In addition, the alignment unit 111 outputs the positional deviation amount E _i and the alignment information to the derived face feature point calculation unit 112 for each three-dimensional face feature point.

The derived face feature point calculation means 112 is a three-dimensional image in which the positional deviation amount E _i between the three-dimensional face feature point on the three-dimensional shape model and the two-dimensional face feature point corresponding to the three-dimensional face feature point is a predetermined value or more. Identify facial feature points. Then, one or a plurality of derived face feature points are generated by deriving only the three-dimensional position information of the specified three-dimensional face feature points to the peripheral positions.
That is, with respect to a 3D face feature point whose positional deviation amount with respect to the 2D face feature point is a predetermined value or more, with respect to one face feature point, the 3D face feature point that is the derivation source and the derived face feature There will be points.
Then, the derived face feature point calculating unit 112 creates a face feature point set that is a combination of a plurality of three-dimensional face feature points forming one three-dimensional shape model. For the 3D face feature points for which the derived face feature points have been generated, one of the 3D face feature points from which the derived face feature points are derived and the derived face feature points are sequentially selected for each face feature point. Generate a set. For a 3D face feature point for which no derived face feature point has been generated, a face feature point set is generated using the 3D face feature point as it is. Here, these face feature point sets are referred to as derived face feature point sets.
For example, the predetermined value can be set to a size of a range that is considered to be the same pixel on the two-dimensional face image when the three-dimensional shape model is projected onto the two-dimensional face image.

Since the position of a 3D face feature point with a large amount of displacement is significantly different from the 3D projection point of the corresponding 2D face feature point, the face of the subject and the 3D shape model around the 3D face feature point. The shape is likely to vary greatly. Therefore, for example, by changing each three-dimensional face feature point according to the amount of positional deviation, the three-dimensional shape model is mainly deformed at a feature point having a large amount of positional deviation, that is, a portion where the shape is highly likely to differ greatly. And a three-dimensional shape model similar to the face of the subject can be efficiently created. For example, in the example shown in FIG. 2, the face image processing apparatus 100 first performs alignment based on the three-dimensional face feature point 21 in the left eye corner, and then greatly varies the three-dimensional face feature point 22 in the left mouth corner. A three-dimensional shape model similar to the subject's face can be efficiently created. Therefore, in this embodiment, the derived face feature point calculating unit 112 has a 3D face feature point and a 3D face whose positional deviation amount E _i between the 2D face feature point corresponding to the 3D face feature point is a predetermined value or more. A plurality of derived face feature points whose positions are changed with respect to the feature points are generated.

ここでd(E_i)は位置ずれ量E_iから求められ、例えば次式により算出することができる。

ここでβは、正規分布の各次元の分散を定める係数である。βが大きいほど３次元顔特徴点P_iと派生顔特徴点P_i'の距離は大きくなり、βが小さいほど３次元顔特徴点P_iと派生顔特徴点P_i'の距離は小さくなる。βの値は、顔画像処理装置１００が設置される環境、その目的などに応じて適宜定められ、例えばβ＝0.7とすることができる。 For example, the derived face feature point calculation unit 112 generates a derived face feature point in which the position of the 3D face feature point is changed according to the probability distribution obtained from the positional deviation amount E _i for each 3D face feature point. For example, if a certain three-dimensional face feature point is P _i = (X _i , Y _i , Z _i ) ^T and a derived face feature point is P _i ′ = (X _i ′, Y _i ′, Z _i ′) ^T , The probability distribution p (P _i ′) can be expressed by a normal distribution obtained from the three-dimensional face feature point P _i and the positional deviation amount E _i as in the following equation.

Here, d (E _i ) is obtained from the positional deviation amount E _i and can be calculated by the following equation, for example.

Here, β is a coefficient that determines the variance of each dimension of the normal distribution. beta is too large 3-dimensional face feature point P _i and derived facial feature point P _i 'distance becomes large, beta is small enough 3D face feature point P _i and derived facial feature point P _i' distance becomes smaller. The value of β is appropriately determined according to the environment where the face image processing apparatus 100 is installed, its purpose, and can be set to β = 0.7, for example.

FIG. 4 is a schematic diagram showing the derived face feature point P _i ′ created by the equation (5). In FIG. 4, the derived face feature point is represented by a symbol “+”. As shown in FIG. 4, the derived face feature points 41 and 42 are distributed according to a three-dimensional normal distribution around the three-dimensional face feature points 21 and 22. In the vicinity of the three-dimensional face feature point 21 at the left corner of the left eye where the displacement amount E _i is small, the derived face feature points 41 are distributed in a narrow range, but the three-dimensional face feature point 22 at the left mouth corner where the displacement amount E _i is large. , The derived face feature points 42 are distributed over a wide range.

In Equation (5), the probability distribution p (P _i ′) is an example using a normal distribution in which the covariance matrix is limited to a diagonal matrix. A normal distribution with no restrictions on the matrix may be used.

例えば、顔特徴点抽出手段１０３が抽出した顔画像の２次元顔特徴点の位置の、対象者の顔における正しい特徴点の位置に対する誤差が小さい場合、３次元空間における正しい特徴点の位置は、３次元投影点の近傍に存在する可能性が高い。従ってその場合には、（７）式を用いて派生顔特徴点を求めることにより、対応する２次元顔特徴点と位置が合う派生顔特徴点を生成できる可能性がより高くなるので、顔画像処理装置１００は、より信頼性の高い派生形状モデルを作成することができる。 Further, the derived face feature point calculation unit 112 may set the three-dimensional projection point of each two-dimensional face feature point as the center of the normal distribution. That is, in this case, the point on the projection line of each 2D face feature point that has the shortest distance from the 3D face feature point corresponding to the 2D face feature point, or each 2D face feature point is represented in the 3D space. The intersection of the straight line projected onto the surface of the three-dimensional shape model is the center of the normal distribution. If the three-dimensional projection point of the two-dimensional face feature point is Q _i = (X _i , Y _i , Z _i ) ^T , the probability distribution p (P _i ′) can be expressed, for example, by the following equation.

For example, when the error of the position of the two-dimensional face feature point of the face image extracted by the face feature point extraction unit 103 with respect to the position of the correct feature point on the face of the subject is small, the position of the correct feature point in the three-dimensional space is There is a high possibility that it exists in the vicinity of the three-dimensional projection point. Therefore, in that case, by obtaining the derived face feature point using the equation (7), it is more likely that a derived face feature point whose position matches the corresponding two-dimensional face feature point can be generated. The processing apparatus 100 can create a derived shape model with higher reliability.

The derived face feature point calculation unit 112 may generate the derived face feature point P _i ′ using a non-normal distribution. For example, the side opposite to the side three-dimensional projection point corresponding viewed from 3D face feature point P _i is present, it is unlikely that the correct feature point exists. Therefore, the derived face feature point calculation unit 112 may not distribute the derived face feature points P _i ′ on the side where the corresponding 3D projection points do not exist as viewed from the 3D face feature points P _i . Thereby, the face image processing apparatus 100 can create a three-dimensional shape model more efficiently.

The derived face feature point calculation unit 112 may generate derived face feature points for all three-dimensional projection points, but this is not always necessary. For a three-dimensional feature point having a sufficiently small positional deviation from the corresponding two-dimensional feature point, even if a three-dimensional shape model is created by using the three-dimensional feature point as it is, it is similar to the face image of the target person. A dimensional shape model can be created. Therefore, the derived face feature point calculation unit 112 does not have to generate a derived face feature point for a three-dimensional feature point whose positional deviation amount is less than a predetermined value. As described above, the predetermined value can be set to the size of a range that is considered to be the same pixel on the two-dimensional face image when, for example, the three-dimensional shape model is projected onto the two-dimensional face image. .
Also, in this case, for example, the derived face feature point calculation unit 112 generates a predetermined number of derived face feature points in a predetermined grid shape with respect to a three-dimensional feature point whose positional deviation amount is a predetermined value or more. May be. The derived face feature point calculation unit 112 may increase the width of the grid for generating the derived face feature points as the positional deviation amount of the three-dimensional feature points is larger. Thereby, the face image processing apparatus 100 can create a three-dimensional shape model in which the arrangement of feature points is similar to that of the subject's face while suppressing the load of the three-dimensional face image creation processing.

  The derived face feature point calculation unit 112 may increase the number of derived face feature points as the positional deviation amount of the three-dimensional feature point is larger, and may decrease the number of derived facial feature points as the positional deviation amount is smaller. As a result, the face image processing apparatus 100 increases the number of derived shape models obtained by deforming the three-dimensional face feature points in a portion where the shape is likely to be different, and the three-dimensional face feature points in a portion where the shape is likely to be similar. The creation of a deformed derived shape model is suppressed. Therefore, the face image processing apparatus 100 can efficiently create a three-dimensional shape model in which the arrangement of feature points is similar to that of the subject's face while suppressing the load of the three-dimensional face image creation processing.

Then, the derived face feature point calculation unit 112 creates a derived face feature point set.
For example, as shown in FIG. 4, the derived face feature point calculation means 112 is a derived face feature point derived from the 3D face feature points 21 and 22 of the 3D shape model 20 or the 3D face feature points 21 and 22. A derived face feature point set 50 is created by combining 41 and 42, respectively. For example, when the same number of derived face feature points are generated for each three-dimensional face feature point, the number of derived face feature point sets 50 to be created is equal to the number of derived face feature points for each three-dimensional face feature point (three-dimensional face feature points). The value obtained by adding 1 (as the number of points) to the power of the number of three-dimensional face feature points that generated the derived face feature points. As the number of derived face feature point sets 50 to be created increases, the time required for the 3D face image creation process increases. However, the position of the characteristic part of the created three-dimensional face image is the characteristic part of the subject. The possibility of similarity to the position of is increased. On the other hand, the smaller the number of derived face feature point sets 50 to be created, the lower the possibility that the position of the characteristic part of the created three-dimensional face image will be similar to the position of the characteristic part of the subject. The time required for the three-dimensional face image creation process is shortened. Alternatively, it may be limited to derived face feature points at positions close to each 3D face feature point or 3D projection point, and a certain upper limit may be provided for the number of derived face feature point sets 50 to be created.

Here, returning to the description of the alignment unit 111, the alignment unit 111 includes, for each derived face feature point set acquired from the derived face feature point calculation unit 112, the three-dimensional face feature included in the derived face feature point set. Rotating, translating, and enlarging / reducing a point or derived face feature point to perform alignment so that the arrangement of the derived face feature point set most closely matches the arrangement of the corresponding two-dimensional face feature points. Is calculated. The alignment unit 111 calculates an alignment error when the derived face feature point set and the two-dimensional face feature point are aligned using the alignment information. Further, the alignment unit 111 outputs the positional deviation amount E _i , alignment information, and position information to the derived model creation unit 105 for each derived face feature point set. Note that the alignment method and the alignment error calculation method in this case are the same as in the case of aligning each 3D face feature point of the 3D shape model with respect to each 2D face feature point. Omitted.

For each derived face feature point set, the derived model creating unit 105 uses the positional deviation amount E _i , the alignment information, and the position information for the derived face feature point set output from the alignment unit 111 to store in the storage unit 250. A derived shape model is created by deforming the stored three-dimensional shape model corresponding to the derived face feature point set.
In order to create a derived shape model, first, the derived model creating means 105 deforms the 3D shape model by changing the position of each 3D face feature point so as to match the corresponding feature point of the derived face feature point set. . For example, the derivation model creation means 105 expresses a three-dimensional shape model with a three-dimensional mesh composed of three-dimensional face feature points, and the three-dimensional mesh is derived from the derivative face feature point set by Thin-Plate Spline, piecewise affine, or the like. A three-dimensional shape model is deformed by deforming into a configured three-dimensional mesh.
Next, the derived model creating means 105 uses the alignment information of the derived face feature point set for the deformed three-dimensional shape model along each axis of the orthonormal coordinate system (X, Y, Z) in the three-dimensional space. Then, it is rotated, translated, and enlarged / reduced so as to face the same direction as the face image, and this is used as a derived shape model.

  Derived model creating means 105 creates as many derived shape models as the number of derived face feature point sets output from the alignment means 111. The derived model creating unit 105 outputs the original three-dimensional shape model stored in the storage unit 101 and the created derived shape model to the light source direction estimating unit 106, the shadow image creating unit 107, and the similar shape selecting unit 109.

  The light source direction estimating means 106 is a light applied to the face reflected in the face image from the face image output from the image input means 102 and the original three-dimensional shape model and derived shape model output from the derived model creating means 105. The light source direction of is estimated. Various known methods can be used as a method for estimating the light source direction. For example, the light source direction estimating means 106 estimates the light source direction from the luminance distribution in the face image by the following method.

（８）式において、ρ(x,y)は、位置(x,y)における顔表面の反射率、l₀は光源係数、lは光源方向ベクトルを表す。またn(x,y)は、位置(x,y)に対応する元の３次元形状モデル又は派生形状モデル上の点における、顔表面に対する法線ベクトルを表す。あるいは、n(x,y)は、位置(x,y)に対応する元の３次元形状モデル上の点における、顔表面に対する法線ベクトルを近似値として用いてもよい。その場合、元の３次元形状モデルから作成した全ての派生形状モデルについて共通の法線ベクトルを使用でき、法線ベクトルの算出処理の負荷を軽減できる。 First, it is assumed that the face surface is a completely diffusing surface (Lambert surface) in which the intensity of light diffused by the surface is proportional to the cosine of the angle formed with the normal of the surface. For example, a two-dimensional set in which the upper left corner of the face image is the origin, the x axis is set in the direction from left to right horizontally with respect to the face, and the y axis is set in the direction from top to bottom perpendicular to the face In the coordinate system, the luminance E (x, y) at the position (x, y) on the face image is considered to be determined by the three-dimensional shape of the face, the direction of the light source, and the reflectance of the face surface according to the following equation. .

In equation (8), ρ (x, y) represents the reflectance of the face surface at the position (x, y), l ₀ represents a light source coefficient, and l represents a light source direction vector. N (x, y) represents a normal vector to the face surface at a point on the original three-dimensional shape model or derived shape model corresponding to the position (x, y). Alternatively, for n (x, y), a normal vector to the face surface at a point on the original three-dimensional shape model corresponding to the position (x, y) may be used as an approximate value. In this case, a common normal vector can be used for all derived shape models created from the original three-dimensional shape model, and the load of normal vector calculation processing can be reduced.

Here, it is assumed that the skin of the face is composed of the same component regardless of the location, and ρ (x, y) has a constant value γ in the equation (8). In this case, equation (8) is an equation with the light source coefficient l ₀ and the light source direction l as unknowns. Therefore, the light source direction estimating means 106 sets the equation (8) as a simultaneous equation at each pixel in the region corresponding to the facial skin in the face image, and solves the simultaneous equation to solve the light source coefficient for each derived shape model. l ₀ and the light source direction l can be obtained. Note that the constant value γ can be set to an average value, a mode value, a median value, or a vicinity value thereof in a region corresponding to the skin of the face in the face image.

  Further, when it is assumed that the reflectance ρ (x, y) of the face surface is constant, it is preferable to exclude regions corresponding to parts that are not skin, for example, regions such as eyes, mouth, nostrils, eyebrows, and hair. Therefore, the light source direction estimating means 106 uses the region corresponding to the non-skin portion as the light source estimation mask region, and does not use the pixels in the light source estimation mask region when the above simultaneous equations are established, so that the light source direction is accurately determined. The direction can be estimated.

  Alternatively, the light source direction estimating means 106 prepares a standard face image obtained by photographing a person's face as a model in various light source directions in advance or an equivalent face image obtained by simulation, The light source direction may be estimated by performing pattern matching and determining the most matching face image. Furthermore, the light source direction estimation means 106, when the positional relationship between the illumination light source and the camera that acquired the face image, or the relationship between the direction of the illumination light emitted from the illumination light source and the shooting direction of the camera is known in advance, The light source direction may be determined based on the relationship.

  Then, the light source direction estimation unit 106 outputs light source direction information indicating the light source direction to the shadow image creation unit 107 in association with the corresponding three-dimensional shape model and derived shape model.

（９）式において、l₀及びlは、それぞれ光源方向推定手段１０６により求められた光源係数及び光源方向ベクトルである。またγは、顔表面の皮膚に相当する部位の反射率であり、例えば、光源方向推定手段１０６において設定されたものと同一の値を有する。さらに、n(X,Y,Z)は、派生形状モデル上の点(X,Y,Z)における、顔表面に対する法線方向ベクトルを表す。あるいは、n(X,Y,Z)は、３次元形状モデル上の点(X,Y,Z)における、顔表面に対する法線方向ベクトルとしてもよい。その場合、その３次元形状モデルから作成した全ての派生形状モデルについて共通の法線ベクトルを使用でき、法線ベクトルの算出処理の負荷を軽減できる。 The shadow image creating unit 107 acquires the light source direction information output from the light source direction estimating unit 106 and the three-dimensional shape model and the derived shape model output from the derived model creating unit 105. The shadow image creating unit 107 renders the derived shape model according to the light source direction indicated in the corresponding light source direction information, and creates a shadow image that is a two-dimensional projection image when light is emitted from the direction. To do. For example, the shadow image creating unit 107 creates a shadow image in the following procedure.
When the derived shape model is irradiated with light from the light source direction indicated in the light source direction information, the reflection intensity of light at an arbitrary point (X, Y, Z) of the face shape model, that is, luminance E (X, Y, Z) can be expressed by the following equation.

In equation (9), l ₀ and l are a light source coefficient and a light source direction vector obtained by the light source direction estimating means 106, respectively. Further, γ is a reflectance of a portion corresponding to the skin on the face surface, and has the same value as that set in the light source direction estimating means 106, for example. Furthermore, n (X, Y, Z) represents a normal direction vector with respect to the face surface at the point (X, Y, Z) on the derived shape model. Alternatively, n (X, Y, Z) may be a normal direction vector with respect to the face surface at the point (X, Y, Z) on the three-dimensional shape model. In that case, a common normal vector can be used for all the derived shape models created from the three-dimensional shape model, and the load of the normal vector calculation processing can be reduced.

  The shadow image creating means 107 obtains the luminance at each point (X, Y, Z) on the derived shape model based on the equation (9). Then, the shadow image creating means 107 projects each point (X, Y, Z) on the derived shape model to a corresponding point (x, y) on the two-dimensional plane. After that, the shadow image creating unit 107 appropriately adjusts the luminance value and grayscales the projected point (x, y) so that the luminance does not overflow or underflow, thereby obtaining a shadow image.

Similarly, the shadow image creation unit 107 renders a three-dimensional shape model according to the light source direction indicated by the corresponding light source direction information, and creates a shadow image when light is emitted from the direction.
Then, the shadow image creation unit 107 outputs the shadow image obtained for each of the three-dimensional shape model and each derived shape model to the similarity calculation unit 108.

  The similarity calculation unit 108 calculates the similarity between the luminance distribution of the face image output from the image input unit 102 and the luminance distribution of the shadow image output from the shadow image creation unit 107 for each shadow image. For this purpose, the similarity calculation unit 108 converts a luminance value represented by RGB into a grayscale luminance value for each point of the face image to create a single-color face image. Then, the similarity calculation means 108 calculates the reciprocal of the mean square error of the luminance values of the corresponding pixels of the single-color face image and the shadow image, and sets it as the similarity.

Note that the similarity calculation unit 108 evaluates the similarity of the luminance value series of these two images, such as the normalized correlation value of the single-color face image and the shadow image, instead of the mean square error of the luminance value, as the similarity. Other possible indicators may be used.
Further, the similarity calculation unit 108 may exclude an area corresponding to the light source estimation mask area from the similarity calculation area. In this case, the similarity calculation means 108 obtains a similarity that more accurately reflects the similarity of the facial contour shape, etc., without depending on the part where the brightness of eyes, nose, mouth, etc. is significantly different from the skin of the face. Can do.
Further, the similarity calculation unit 108 may calculate one similarity for the entire area representing the face of the face image and the shadow image, or may calculate the similarity for each area by dividing into a plurality of areas. Also good.
The similarity calculation unit 108 outputs the similarity calculated for each shadow image to the similar shape selection unit 109.

  The similar shape selection unit 109 acquires the three-dimensional shape model and the derived shape model output from the derived model creation unit 105 and the similarity output from the similarity calculation unit 108. The similar shape selection unit 109 refers to the similarity calculated for each shadow image, selects a three-dimensional shape model or a derived shape model corresponding to the shadow image with the highest similarity, and selects the most similar face shape model. To do.

Note that the similar shape selection unit 109 may select M derived shape models in descending order of the corresponding similarity, and average them to obtain the most similar face shape model. The predetermined number M may be a fixed value such as 2, 3 or the like, or corresponds to a predetermined ratio (for example, 5% or 10%) in the total number of three-dimensional shape models stored in the storage unit 101. It is good also as a value to do.
Further, when the similarity calculation unit 108 divides the face image and the shadow image into a plurality of regions and calculates the similarity for each region, the similar shape selection unit 109 performs three-dimensional processing so as to correspond to the region. Divide the shape model and the derived shape model. Then, the similar shape selection unit 109 may select a three-dimensional shape model or a derived shape model corresponding to the shadow image with the highest similarity for each divided region, and may combine these to form the most similar face shape model. Further, the similar shape selection unit 109 selects and averages the M three-dimensional shape models or the derived shape models in descending order of the corresponding similarity for each divided region, and the synthesized result is the most similar. It may be a face shape model. Thus, by creating a similar face shape model for each of the divided areas, the face image processing apparatus 100 can create a similar face shape model with higher accuracy.
The similar shape selection unit 109 outputs the obtained most similar face shape model to the individual model creation unit 110.

  The personal model creation unit 110 creates a three-dimensional face image of the target person by mapping the face image as a texture image on the most similar face shape model output from the similar shape selection unit 109. That is, the individual model creation means 110 determines the position of each 2D face feature point of the face image and the 3D face feature point or derived face feature point set of the most similar face shape model corresponding to each 2D face feature point. In order to match, the face image is mapped to the most similar face shape model as a texture image using the alignment information.

Note that, as described above, occlusion that makes part of the face invisible occurs, and information on part of the face may be missing in the face image. Accordingly, when a face information or texture missing portion is generated in the 3D face image to which the face image is mapped, the personal model creating means 110 can detect the luminance of pixels around the missing portion before mapping or after mapping. Interpolation processing (for example, spline interpolation, linear interpolation) is performed using the information, and the luminance value of the missing portion pixel is calculated.
Alternatively, the personal model creation means 110 may use the luminance value of the pixel corresponding to the symmetrical position of the missing portion as the luminance value of the pixel of the missing portion by utilizing the symmetry of the human face. For example, the luminance value of the pixel at a line-symmetric position with respect to the midline of the face with respect to the missing portion can be set as the luminance value of the pixel of the missing portion.
Alternatively, the personal model creation unit 110 may perform an interpolation process on the missing part using the texture image of the corresponding part of the three-dimensional shape model.
By performing such an interpolation process, when the face image is acquired, the target person is facing the direction in which part of the face is hidden, or a part of the target person's face is hidden behind the shield Even so, it is possible to create a three-dimensional face image representing the entire face of the subject.
The personal model creation unit 110 stores the created three-dimensional face image in the storage unit 101 or outputs it to a face authentication device or the like that performs a matching process using the face image processing device 100.

Hereinafter, the operation of the three-dimensional face image creation process by the face image processing apparatus 100 to which the present invention is applied will be described with reference to the flowchart shown in FIG. The flow of operations described below is controlled by a control unit (not shown) that operates on the microprocessor constituting the face image processing apparatus 100 and controls the entire face image processing apparatus 100.
First, the face image processing apparatus 100 acquires a face image of the subject as a two-dimensional image via the image input unit 102 (step S501). Next, the face feature point extraction unit 103 extracts a two-dimensional face feature point from the acquired face image (step S502).

When the storage unit 101 stores a plurality of three-dimensional shape models, the following steps S503 to S513 are performed for each of them. When the three-dimensional shape model stored in the storage unit 101 is only one model representing a standard face three-dimensional shape model, step S514 described later is not necessary. The registration unit 111 of the registration information calculation unit 104 calculates the total positional deviation amount E obtained from the positional deviation amount E _i of each three-dimensional face feature point with respect to each two-dimensional face feature point extracted in step S50. Each three-dimensional face feature point is rotated or translated along each axis of the orthonormal coordinate system (X, Y, Z) or enlarged / reduced so as to be minimized. Then, the alignment unit 111 derives the positional deviation amount E _i of each three-dimensional face feature point corresponding to each two-dimensional facial feature point when the total positional deviation amount E is minimum, as a derived face feature point calculating unit 112. (Step S503).

The following steps S504 to S510 are performed for each derived shape model. The derived face feature point calculation unit 112 generates a derived face feature point in which the position of each 3D face feature point of the 3D shape model is changed based on the positional deviation amount E _i (step S504). Then, the derived face feature point calculation unit 112 creates a derived face feature point set in which the corresponding feature points selected from the corresponding three-dimensional face feature points or the derived face feature points are combined for each face feature point. Next, the registration unit 111 performs registration of the derived face feature point set with respect to the two-dimensional face feature point in the same manner as the process of step S503, and obtains alignment information of the derived face feature point set (step S503). S505). Next, the derivation model creation means 105 transforms the three-dimensional shape model so as to move the three-dimensional face feature points to the corresponding feature points of the derivation face feature point set, and further obtains alignment information of the derivation face feature point set. The derived shape model is created by using the transformation so as to face the same direction as the face image (step S506).

  Next, the light source direction estimation means 106 estimates the light source direction using the derived shape model based on the face image, and outputs light source direction information representing the estimated light source direction (step S507). Then, the shadow image creating unit 107 renders the derived shape model according to the light source direction information and projects it on the two-dimensional plane to create a shadow image (step S508). Thereafter, the similarity calculation unit 108 calculates the similarity between the created shadow image and the face image acquired by the image input unit 210 (step S509).

Then, it is determined whether or not a predetermined number N of derived shape models have been created and the respective similarities have been calculated (step S510). The predetermined number N is the number of derived shape models for which the similarity with the face image is calculated, that is, the number of derived shape models to be created by the derived model creation means 105. It can be. If a predetermined number N of derived shape models have not yet been created and the similarity has not been calculated, the processes in steps S504 to S509 are repeated to create a new derived shape model, and the similarity is calculated. .
On the other hand, when it is determined in step S510 that a predetermined number N of derived shape models are created and the similarity is calculated, the light source direction estimation unit 106 estimates and estimates the light source direction using the three-dimensional shape model. Light source direction information representing the light source direction is output (step S511). Then, the shadow image creating means 107 renders the three-dimensional shape model according to the light source direction information and projects it on the two-dimensional plane to create a shadow image (step S512). Thereafter, the similarity calculation means 108 calculates the similarity between the created shadow image and the face image (step S513).
Derived shape models are created for all the three-dimensional shape models, and it is determined whether or not the similarity with the face image is calculated for each three-dimensional shape model and each derived shape model (step S514). If a derived shape model has not yet been created, and there is a 3D shape model for which the similarity between the 3D shape model and each derived shape model and the face image has not been calculated, control returns to step S503. The processing from S503 to S513 is repeated.

  On the other hand, if it is determined in step S514 that the derived shape models have been created for all the three-dimensional shape models and the similarity between each three-dimensional shape model and each derived shape model and the face image has been calculated, similar shape selection means 109 determines the most similar face shape model from the three-dimensional shape model or derived shape model closest to the face image according to the similarity (step S512). Finally, the personal model creation means 290 creates a three-dimensional face image of the subject by mapping the face image as a texture image on the most similar face shape model (step S290).

  As described above, the face image processing apparatus 100 to which the present invention is applied includes the three-dimensional face feature point in the three-dimensional shape model stored in advance in the storage unit 101 and the two-dimensional face input from the image input unit 102. The amount of positional deviation from the two-dimensional face feature point extracted from the image is calculated. Then, the face image processing apparatus 100 generates a derived face feature point in which the position is changed for a three-dimensional face feature point having a positional deviation amount equal to or greater than a predetermined threshold. Further, the face image processing apparatus 100 combines the feature points selected from the original three-dimensional face feature points or derived face feature points with respect to the three-dimensional face feature points having a positional deviation amount equal to or greater than a predetermined threshold, and the positional deviation amount is a predetermined amount. For 3D face feature points less than the threshold, the original 3D face feature points are combined to generate a derived face feature point set. Then, the face image processing apparatus 100 creates a derived shape model by modifying the 3D shape model so that the 3D face feature point matches the derived face feature point set, and the created derived shape model or the original 3D shape A three-dimensional face image is created based on the model most similar to the two-dimensional face image. As a result, the face image processing apparatus 100 does not use a special surface measuring apparatus and does not collect shape data of a large number of people in advance. A three-dimensional face image can be created in which the position of is more closely matched to the face of the subject. As a result, it is also possible to create a three-dimensional face image having facial shape characteristics such as the extent of eyebrow or cheekbone overhanging and the shape of the jaw. In addition, a variety of derived shape models can be created by preparing multiple 3D shape models, providing a derived shape model in which the position of each facial feature point matches well with the 2D face image of the subject. More likely to be able to do it.

Further, the face image processing apparatus 100 increases the derived face feature point corresponding to the three-dimensional face feature point as the positional deviation amount of the three-dimensional face feature point with respect to the corresponding two-dimensional face feature point increases. Distribute to the range. As a result, it is more likely that a derived shape model in which the positions of the facial feature points are well matched with the two-dimensional face image of the subject can be provided.
Further, the face image processing apparatus 100 increases the number of derived face feature points corresponding to the three-dimensional face feature point as the positional deviation amount of the three-dimensional face feature point with respect to the corresponding two-dimensional face feature point increases. Increase. Thereby, it is possible to suppress the number of derivative shape models in which the three-dimensional face feature points are changed for a portion that is likely to be similar in shape to the subject's face, and to reduce the load of the three-dimensional face image creation processing.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, you may abbreviate | omit the light source direction estimation process by the light source direction estimation means shown to step S507 and S511. In that case, in steps S508 and S512, the shadow image creating means creates a two-dimensional image in which the derived shape model and the three-dimensional shape model are rendered without following the light source direction indicated in the light source direction information. In steps S509 and S513, the similarity calculation unit calculates the similarity of the luminance distribution between the two-dimensional image and the face image. Thereby, the face image processing apparatus can reduce the load of the three-dimensional face image creation process.

  In the derived face feature point generation process shown in step S504, the derived face feature point calculation unit may calculate all the derived face feature points for the three-dimensional shape model at once. In that case, if the similarity is not calculated for the predetermined number N of derived shape models in step S510, the face image processing device returns control to step S505, and repeats the processing of steps S505 to S509. Thereby, the face image processing apparatus can reduce the load of the three-dimensional face image creation process.

  Further, the derived face feature point calculation means does not output information on all the created derived face feature point sets to the alignment means, but the derived face feature point having the smallest positional deviation amount from the two-dimensional face feature point. Only information about the set may be output to the alignment means. In that case, the derived model creating means creates one derived shape model for each three-dimensional shape model. Then, the light source direction estimating means, the shadow image creating means, the similarity calculating means, and the similar shape selecting means are the light source direction estimating process, the shadow image creating process, the similarity degree only for the derived shape model and the original three-dimensional shape model, respectively. Calculation processing and similar shape selection processing are performed. Thereby, the face image processing apparatus can reduce the load of the three-dimensional face image creation process.

  As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

DESCRIPTION OF SYMBOLS 100 Face image processing apparatus 101 Storage means 102 Image input means 103 Face feature point extraction means 104 Registration information calculation means 105 Derived model creation means 106 Light source direction estimation means 107 Shadow image creation means 108 Similarity calculation means 109 Similar shape selection means 110 Personal model creation means 111 Positioning means 112 Derived face feature point calculation means

Claims (3)

A face image processing device that generates a 3D face image corresponding to a face of a person by synthesizing a 2D face image including the face of the person and a 3D shape model representing the 3D shape of the face,
Image input means for inputting the two-dimensional face image;
Storage means for storing in advance the three-dimensional shape model and a plurality of three-dimensional face feature points representing facial feature points in the three-dimensional shape model;
Facial feature point extraction means for extracting a plurality of two-dimensional facial feature points representing the facial feature points from the two-dimensional facial image;
Alignment means for performing alignment by calculating a displacement amount between the three-dimensional face feature point and the two-dimensional face feature point corresponding to the three-dimensional face feature point;
One or a plurality of derived face feature points whose positions are changed with respect to the three-dimensional face feature points having a positional deviation amount equal to or greater than a predetermined threshold value are generated, and the three-dimensional face feature points or the derivative are generated for the three-dimensional face feature points. For a feature point selected from face feature points, and for the three-dimensional face feature point whose positional deviation amount is less than a predetermined threshold, a derived face feature point set is formed by combining the three-dimensional face feature points to form a face shape. A derived face feature point calculating means for generating;
Derived model creation means for creating a derived shape model by deforming the 3D shape model so that the 3D face feature point of the 3D shape model matches the derived face feature point set;
Similar shape selection means for comparing the derived shape model and the 3D shape model with the 2D face image and selecting the derived shape model most similar to the 2D face image or the 3D shape model;
A personal model creating means for creating the 3D face image corresponding to the face of the person by combining the 2D face image with the selected derived shape model or the 3D shape model;
A face image processing apparatus comprising:   The derived face feature point calculation means distributes the derived face feature points corresponding to the three-dimensional face feature points more widely as the positional deviation amount of the three-dimensional face feature points with respect to the corresponding two-dimensional face feature points is larger. The face image processing apparatus according to claim 1, generated by   The derived face feature point calculation means increases the number of the derived face feature points corresponding to the three-dimensional face feature points as the positional deviation amount of the three-dimensional face feature points with respect to the corresponding two-dimensional face feature points increases. The face image processing device according to claim 1, wherein the face image processing device is generated.

Images