JP4623320B2 - Three-dimensional shape estimation system and image generation system - Google Patents

Description

  The present invention relates to a system, a method, and a program for estimating a three-dimensional shape of an object from a two-dimensional image showing the object. The present invention also relates to a system, a method, and a program for generating a two-dimensional image with different illumination conditions or a two-dimensional image with different illumination conditions and the direction of the object from a two-dimensional image in which a certain object is captured. .

  As a general technique for constructing and estimating a three-dimensional shape of an object from a two-dimensional image, for example, a technique disclosed in Japanese Patent Application Laid-Open No. 11-242745 (Patent Document 1) is known. According to this technique, a stereo / multi-view image captured by two or more cameras, a pattern irradiation image obtained by capturing a known pattern irradiated with visible light or infrared light, or the like is used. When the shape of the target is limited, it is possible to estimate the three-dimensional shape of the target from one image by using the restriction by the shape. For example, when a vertical line and a horizontal line intersect at right angles like a building, or when a specific pattern such as a repeating pattern is drawn on a plane, by using geometric information such as the vanishing point principle or cross ratio, Sometimes the three-dimensional shape of the object can be calculated. However, in the case of a “face”, that is, an object that does not have a regular geometrical restriction such as a plane or a sphere and does not have a specific pattern in color and brightness, the above-described general stereo / multi-view Images and pattern irradiation images are used.

  However, according to such a method, normal camera photography such as a plurality of cameras (stereo / multi-lens cameras), a pattern irradiator that emits pattern light, a camera for detecting light such as infrared light, etc. Requires a different instrument for measurement. This causes problems such as an increase in cost and an environment in which measurement is possible. Furthermore, it is necessary to store information at the time of measurement such as the position of the camera at the time of shooting and the irradiation position of the pattern irradiator. This causes problems that the measurement environment is restricted and that only images that have been previously captured in an environment intended for measurement can be used. The “face” is often captured as a single image without taking measurement into consideration. Therefore, in this prior art, it is impossible to estimate the three-dimensional shape of a face from such a single face image.

  In relation to the above, according to the technique disclosed in Japanese Patent Laid-Open No. 2001-84362 (Patent Document 2), an object having a three-dimensional shape including a portion having a substantially constant surface reflectance is substantially single. Photographed under one light source. Based on the image obtained by this photographing, the direction of the light source and the three-dimensional shape of the object are estimated. In addition, a face classification system disclosed in Japanese Patent Laid-Open No. 5-266173 (Patent Document 3) includes first means for positioning a face in a two-dimensional display of a three-dimensional frame, and a first means for detecting a face in the display. 2 means, a third means for generating a face feature vector, and comparing the face feature vector detected now with the face feature vector detected previously, And a fourth means for determining whether or not they match. In addition, International Publication WO-02 / 007095-A1 (Patent Document 4) discloses a face three-dimensional direction tracking device that sequentially estimates the face direction of a person from an input image captured in time series.

  The face does not have a regular geometric shape such as a plane or a sphere, but the general arrangement and topological shape of feature points such as eyes and mouth are the same. Individual different face shapes can be generated depending on the amount of deviation of the feature point arrangement from the standard face. Furthermore, as for the captured face image, there are a lot of images facing the front, so the topological shape of the face image is also limited. From such a viewpoint, a 3D face shape is estimated from a single face image by using a learning type model obtained from the 3D face shapes of a plurality of persons (Non-patent Document 1). However, according to this method, it is necessary to manually specify feature points in the face image in order to match the face image with the learning model.

Japanese Patent Laid-Open No. 11-242745 JP 2001-84362 A JP-A-5-266173 International Publication WO-02 / 007095-A1 T. Vetter et al., "A Morphable Model For The Synthesis of 3D Faces", ACM Conf. SIGGRAPH 99, pp. 187-194, 1999.

  One object of the present invention is to provide a technique capable of estimating the three-dimensional shape of an object from a single two-dimensional image showing the object without using a special measuring device.

  Illumination basis data common to a plurality of objects and three-dimensional shape information corresponding to the illumination basis data are prepared in advance. The combination of the illumination base data representing the image most similar to the two-dimensional image of an object, the weight of the illumination base data, and the illumination vector is obtained, and the illumination base of the object determined from the weight of the illumination base data and the illumination base data is partially differentiated Relative shape information is obtained. The positional relationship of feature points between the two-dimensional image and the relative shape information is obtained. The three-dimensional shape information of the certain object is corrected by correcting the three-dimensional shape information using the positional relationship.

  According to the present invention, it is possible to estimate the three-dimensional shape of an object from a single two-dimensional image in which the object is captured without using a special measurement device. Thus, the cost is reduced.

  A three-dimensional shape estimation system, a three-dimensional shape estimation program, a three-dimensional shape estimation method, an image generation system, an image generation program, and an image generation method according to the present invention will be described with reference to the accompanying drawings.

  First, the concept used in the description of the embodiment of the present invention will be described. In the embodiment of the present invention, a human “face” is exemplified as a target object whose three-dimensional shape is estimated.

  FIG. 1 is a diagram for explaining a “shape function” that expresses a three-dimensional shape of a certain face. The three-dimensional shape of such an object can be measured by, for example, a range finder. The range finder is a device that acquires the shape (relative position information) and color information of an object and loads them into a computer. By the range finder, the front surface coordinates x and y and the depth coordinate z of the object surface are obtained. That is, the three-dimensional shape of the face is represented in the xyz coordinate system as shown in FIG. 1, and the “shape function z” representing the three-dimensional shape is given by the following equation.

  The partial differential form of the shape function z is given by the following equation.

This function group (f _x , f _y ) indicates differential information of the shape function z and serves as a parameter for expressing the relative shape of the object such as the curvature of the object surface. Therefore, this function group (f _x , f _y ) is hereinafter referred to as “relative shape function”.

  FIG. 2 is a schematic view showing a two-dimensional image of a certain face. This two-dimensional image is expressed on the xy plane. In FIG. 2, the horizontal direction of the face corresponds to the x direction and the vertical direction of the face corresponds to the y direction. The number of pixels in the x and y directions is given by w and h, respectively. That is, the total number of pixels s of this two-dimensional image is given by s = w × h.

  The two-dimensional image of the face includes luminance information and color information. Since this luminance / color information is determined by reflection of the shape of the face, it is possible to estimate the face shape by photographing to a plurality of illuminations if the face position and posture are the same. One such idea is Peter N. Belhumeur et al., "What Is the Set of Images of an Object Under All Possible Illumination Conditions?", International Journal of Computer Vision, Vol. No. 28, pp. 245. -260, 1998, there is an "illumination basis". This illumination base (geodesic illumination basis, GIB) is a descriptor of illumination variation at each position of the face epidermis.

First, the diffuse reflectance of the face surface at the i-th pixel P _i of the two-dimensional image is given by a _i , and the normal vector n _i is n _i = (n _{i, x} , n _{i, y} , and it is given by n _{i, z).} Further, it is assumed that an illumination luminance vector s indicating an illumination direction at the pixel P _i is given by s = (s _x , s _y , s _z ). At this time, “luminance X _i ” at the pixel P _i is given by the following equation.

ここで、ベクトルｂ_ｉは、次の式で与えられる。

Here, the vector b _i is given by the following equation.

Therefore, the “luminance vector X” indicating the luminance X _i for all the pixels p _i included in the two-dimensional image is given by the following equation.

In these equations, the diffuse reflectance a _i and the normal vector n _i , that is, the vector b _i depend only on the shape / property of the object and do not depend on the intensity / direction of the illumination. The illumination intensity vector s changes when the illumination is changed. A set I of the luminance vectors X of the two-dimensional image created when the illumination is changed is expressed as follows.

ここで、Ｂは、次のように表される

Here, B is expressed as follows

That is, B _x = {b _{i, x} } = (b _{0, x} ,..., B _{s, x} ), and B _y = {b _{i, y} } = (b _{0, y} ,. b _{s, y} ), and B _z = {b _{i, z} } = (b _{0, z} ,..., b _{s, z} ). Then, the B is referred to as "illumination basis". That is, the illumination base (reflection information) is a descriptor of illumination variation, and can represent a change in luminance with respect to illumination variation. Further, as described above, the brightness of the image is determined by the illumination and the face shape. Therefore, it can be said that this illumination base (reflection information) is a parameter related to the face shape. In the present invention, this concept of “illumination base” is used.

  The present invention finally estimates the “three-dimensional shape (shape function f (x, y))” shown in FIG. 1 from the “two-dimensional image” shown in FIG. 2 with high accuracy. Objective. Another object of the present invention is to generate a two-dimensional image with different illumination conditions or a two-dimensional image with different illumination conditions and face orientations from a “two-dimensional image” shown in FIG. Such a technique can be applied to, for example, a security field such as a face personal authentication technique, a beauty / shaping field, an amusement field, and the like. Hereinafter, the configuration and operation of the present invention will be described in detail.

(First embodiment)
FIG. 3 is a block diagram showing the configuration of the three-dimensional shape estimation system according to the first embodiment of the present invention. The three-dimensional shape estimation system 1 includes a relative shape analysis device 10, a feature point position search device 20, and an absolute shape analysis device 30. The relative shape analysis apparatus 10 receives face image data 50 indicating a two-dimensional image of a certain face, and calculates a relative shape function (f _x , f _y ) of the face based on luminance information and learning reflection information accumulated in advance. presume. Relative shape data 51 indicating the estimated relative shape function (f _x , f _y ) is output to the feature point position search device 20 and the absolute shape analysis device 30. The feature point position search device 20 receives the face image data 50 and the relative shape data 51, and automatically detects “feature points” of faces such as both eyes and nose. Feature point position data 52 indicating the position of the “feature point” is output to the absolute shape analyzer 30. The absolute shape analysis apparatus 30 estimates the shape function f (x, y) of the face based on the relative shape function (f _x , f _y ), the position of the feature point, and the learned shape data accumulated in advance. The three-dimensional absolute shape data 53 indicating the estimated shape function f (x, y) is output from the absolute shape analyzer 30. Thus, according to the three-dimensional shape estimation system 1 according to the present embodiment, the three-dimensional absolute shape data 53 (see FIG. 1) is obtained from the face image data 50 (see FIG. 2).

  Hereinafter, the configuration and operation of each device will be described in detail.

  FIG. 4 is a block diagram showing the configuration of the relative shape analyzer 10 according to the present invention. The relative shape analysis apparatus 10 includes an input unit 11, a learning data creation unit 12, a storage unit 13, a relative shape calculation unit 14, and an output unit 15. The storage unit 13 stores relative shape data 51 and generalized illumination base data 61. The generalized illumination base data (first learning data) 61 indicates generalized illumination bases for a plurality of human faces (a plurality of objects of the same type), and is accumulated by learning in advance. The learning data creation unit 12 is a unit for creating the generalized illumination base data 61, and is realized by a CPU and a computer program. The relative shape calculation unit 14 is a unit for creating the relative shape data 51, and is realized by a CPU and a computer program. Examples of the storage unit 13 include a hard disk drive and a memory.

  First, the generalized illumination base data 61 is created. FIG. 5 is a flowchart showing the operation of the relative shape analyzer 10 when generating the generalized illumination base data 61, that is, the operation of the learning data generator 12.

Step S101:
First, illumination is applied to the faces of a plurality of persons from various directions, and photographing is performed. Thereby, a plurality of three-dimensional shape data 60 with texture is obtained. The three-dimensional shape data 60 of the faces of a plurality of people is input to the learning data creation unit 12 via the input unit 11.

Step S102:
Next, the learning data creation unit 12 calculates the illumination base B of each person's face from the three-dimensional shape data 60 based on the above equation (6).

Step S103:
Next, the learning data creation section 12, by methods such as principal component analysis, component illumination bases B of the face of the plurality of persons A _1, A 2, decomposes _... A _n. As a result, a general facial illumination base (generalized illumination base) B can be described as follows.

ここで、β_１，β_２，…β_ｍは、個人ごとに異なるパラメータである。

Here, β ₁ , β ₂ ,... Β _m are different parameters for each individual.

Step S104:
The learning data creation unit 12 outputs and stores the generalized illumination base data 61 indicating the generalized illumination base B obtained in step S103 in the storage unit 13.

Next, a method for estimating relative shape data 51 indicating a relative shape function (f _x , f _y ) of a person's face from face image data 50 indicating a two-dimensional image of a person's face is shown. The face orientation of this person is known, but the lighting base B of this person is unknown. FIG. 6 is a flowchart showing the operation of the relative shape calculation unit 14 of the relative shape analyzer 10.

Step S111:
First, face image data 50 (see FIG. 2) indicating a two-dimensional image of a person's face is input to the relative shape calculation unit 14 via the input unit 11.

Step S112:
The two-dimensional face image has luminance information and color information. Therefore, the relative shape calculation unit 14 can extract the brightness vector X = {X _i } represented by the above-described equation (5) from the face image data 50.

Step S113:
Next, the relative shape calculation unit 14 reads the generalized illumination base data 61 from the storage unit 13 and obtains the generalized illumination base F (β ₁ , β ₂ ,... Β _m , A ₁ , A ₂ ,... A _n ). get. Using the luminance vector X, the generalized illumination base F, and the illumination intensity vector s, the relative shape calculation unit 14 creates a functional E1 represented by the following equation.

Step S114:
Next, the relative shape calculation unit 14 obtains parameters β ₁ , β ₂ ,... Β _m that minimize the functional E1. When the calculation becomes nonlinear, the relative shape calculation unit 14 calculates these parameters using an iterative calculation method such as a hill climbing method. The relative shape calculation unit 14, the determined parameter beta _1, beta _2, illumination basis F in the case of _{... β m (β 1, β} 2, ... β m, A 1, A 2, ... A n) the , and illumination basis ^{B *} of the face. In this way, by using the generalized illumination base data (first learning data) 61, the illumination base B ^{* of the} face is calculated from the two-dimensional image of the face.

Step S115:
The illumination base B ^* can also be expressed by the above formula (7). According to the present invention, the relative shape calculation unit 14 calculates a relative shape function (f _x , f _y ) related to the face based on the illumination base B ^* by the following expression.

Thus, the relative shape function (f _{x, f y)} of the face from a two-dimensional image of a face is calculated. As shown in Expression (2), this relative shape function (f _x , f _y ) represents a partial differentiation of the shape function f (x, y). The above processing is effective not only for the method based on the illumination base but also for all models that link the image and differential information that is a relative shape.

Step S116:
The relative shape calculation unit 14 outputs the relative shape data 51 indicating the obtained relative shape function (f _x , f _y ) to the feature point position search device 20 and the absolute shape analysis device 30 via the output unit 15. . The relative shape calculation unit 14 may store the relative shape data 15 in the storage unit 13.

As described above, according to the relative shape analysis apparatus 10 according to the present invention, the illumination base B of an object can be estimated from a single two-dimensional image in which the object is reflected. Further, from the single two-dimensional image, the relative shape function of the object (f _{x, f y)} it is possible to estimate the. Here, it is not necessary to photograph the object with a special photographing apparatus such as a stereo / multi-lens camera. Therefore, the cost is reduced. In addition, it is not necessary to store information at the time of shooting, and there is no restriction on the shooting environment of the two-dimensional image.

  FIG. 7 is a block diagram showing a configuration of the feature point position search apparatus 20 according to the present invention. The feature point position search device 20 includes an input unit 21, a feature point extraction unit 22, a storage unit 25, and an output unit 26. The feature point extraction unit 22 includes a color / luminance feature point extraction unit 23 and a shape feature point extraction unit 24. The storage unit 25 stores feature point position data 52 indicating the positions of a plurality of “feature points” in a certain face. The feature point extraction unit 22 is a unit for creating the feature point position data 52, and is realized by a CPU and a computer program. Examples of the storage unit 25 include a hard disk drive and a memory.

A “feature point” of a face is a characteristic point of the face such as the center of the eye, the apex of the nose, or under the nose. This feature point includes a “color / luminance feature point (first feature point)” characterized by color and brightness and a “shape feature point (second feature point)” characterized by shape. Examples of the color / luminance feature points include eyes and mouths. For example, lips have a reddish component stronger than the surroundings, and eyes are black in the case of Orientals. Therefore, these can be extracted from the color / luminance information included in the image. On the other hand, examples of the shape feature point include the nose and the bottom of the nose. Since these are characterized by their shapes, they can be extracted using relative shape functions (f _x , f _y ).

FIG. 8 is a flowchart showing the operation of the feature point position search apparatus 20 according to the present invention.
Step S201:
The face image data 50 and the relative shape data 51 obtained by the relative shape analyzer 10 are input to the feature point extraction unit 22 via the input unit 21.

Step S202:
First, the color / luminance feature point extraction unit 23 uses the color / luminance information included in the face image data 50 to extract “color / luminance feature points”. Information regarding the position of the color / luminance feature point is output to the shape feature point extraction unit 24.

Step S203:
Next, extraction of the shape feature points such as nose takes place. FIG. 9 shows a contour of a face when viewed from the side, that is, a shape function z = f (y) at a certain x coordinate. Here, it is assumed that the certain x-coordinate corresponds to the center of both eyes. As shown in FIG. 9, with the shape function z = f (y) in the + y direction from the eye position a1, it is considered that the first minimum point corresponds to the position a2 of the nose apex. Also, the first maximum point is considered to correspond to the position a3 under the nose. Further, it is considered that the next minimum point corresponds to the position a4 of the lip apex. Therefore, it is only necessary to search for such a minimum point / maximum point. Here, the face shape function f (x, y) has not yet been obtained. Therefore, the shape feature point extraction unit 24 calculates the curvature H of the face surface based on the following equation.

Here, (f _x , f _y ) can be known from the relative shape data 51. Further, f _xx , f _yy , and f _xy are second-order partial derivatives of the shape function f (x, y), and can be known from the relative shape function (f _x , f _y ) indicated by the relative shape data 51. That is, the shape feature point extraction unit 24 can calculate the curvature H. Further, the position a1 of both eyes can be known from the color / luminance feature point information obtained by the color / luminance feature point extraction unit 23. Therefore, the shape feature point extraction unit 24 determines the maximum point of curvature H as the nose apex position a2 in the + y direction from the center position a1 of both eyes. Similarly, the position a3 under the nose and the position a4 of the lip apex are also determined. In this manner, the shape feature point extraction unit 24 extracts “shape feature points” based on the two-dimensional image of the face and the relative shape functions (f _x , f _y ).

Step S204:
The feature point extraction unit 22 outputs feature point position data 52 indicating the positions of the color / luminance feature points and the shape feature points obtained in steps S202 and S203. The feature point position data 52 is output to the absolute shape analyzer 30 via the output unit 26. The feature point position data 52 may be stored in the storage unit 25.

As described above, according to the feature point position device 20 according to the present invention, it is possible to automatically find a feature point of an object from a single two-dimensional image in which the object is reflected. . This is because the shape feature points can be extracted based on the relative shape function (f _x , f _y ) estimated by the relative shape analyzer 10. There is no need to manually specify feature points in the face image.

  FIG. 10 is a block diagram showing the configuration of the absolute shape analyzer 30 according to the present invention. The absolute shape analysis device 30 includes an input unit 31, a learning data creation unit 32, a storage unit 33, an absolute shape calculation unit 34, and an output unit 35. The storage unit 33 stores 3D absolute shape data 53 and 3D shape learning data 62. The three-dimensional shape learning data (second learning data) 62 indicates information related to the three-dimensional shape of a plurality of human faces (a plurality of objects of the same kind), and is accumulated by learning in advance. The learning data creation unit 32 is a unit for creating the three-dimensional shape learning data 62, and is realized by a CPU and a computer program. The absolute shape calculation unit 34 is a unit for creating the three-dimensional absolute shape data 53, and is realized by a CPU and a computer program. Examples of the storage unit 33 include a hard disk drive and a memory.

  First, the three-dimensional shape learning data 62 is created. FIG. 11 is a flowchart showing the operation of the absolute shape analysis device 30 when creating the three-dimensional shape learning data 62, that is, the operation of the learning data creating unit 32.

Step S301:
First, three-dimensional shape data (shape function z = f (x, y)) of a plurality of human faces is acquired by using a three-dimensional shape measuring device such as a range finder. In the acquired three-dimensional shape data, the front is represented by the xy plane, and is represented by the depth and the z direction (see FIG. 1). The three-dimensional shape data 60 of the faces of a plurality of people is input to the learning data creation unit 32 via the input unit 31.

Step S302:
Next, the learning data creation unit 32 finds a feature point in each of the acquired plurality of three-dimensional shape data. Specifically, the learning data creation unit 32 searches for the position of the feature point by the same method as the feature point extraction unit 22 described above. That is, the same part as the part extracted by the feature point extraction unit 22 is extracted. Here, since the shape function z = f (x, y) of each face is known, the learning data creation unit 32 may search for the position of the feature point using the shape function z. Alternatively, since the learning data is created only once in advance, the user may perform it manually. In this step, the positions of a plurality of feature points are determined for each face.

Step S303:
Next, the learning data creation unit 32 sets one point as the “base point O” among the extracted feature points (parts). This “base point O” is a reference point (origin) for fixing the coordinates of all three-dimensional face shapes, and is commonly set for all the faces. For example, the base point O is selected below the nose present near the center of the face.

Step S304:
Next, the learning data creation unit 32 calculates the relative positions of a plurality of feature points other than the base point O with respect to the base point O. For example, FIG. 12 shows a person's face. In this face, five points are extracted as feature points: the center of both eyes (P1, P2), the bottom of the nose, and the left and right ends of the mouth (P3, P4). In addition, the lower part of the nose is selected as the base point O. At this time, the relative positions of the feature points P1 to P4 with respect to the base point (origin point) O, that is, the coordinates of the feature points P1 to P4 in the coordinate system with the base point O as the origin point are calculated. This calculation is executed for all three-dimensional shape data 60. Thereafter, the learning data creation unit 32 calculates an average value (average relative position) of relative positions with respect to all the faces for each feature point (P1, P2, P3, P4). That is, the average value of the coordinates of each feature point is calculated, and four average relative positions with respect to each of the feature points P1 to P4 are calculated.

Step S305:
Next, the learning data creation unit 32 performs coordinate conversion of the shape function f (x, y) so that the plurality of feature points coincide with the average relative positions of all the faces. Specifically, the surrounding area of each feature point is enlarged / reduced. For example, in FIG. 12, the face is divided into four regions R1 to R4 by a coordinate axis uv passing through the base point O. Each of the regions R1 to R4 includes the feature points P1 to P4. At this time, each of the regions R1 to R4 is enlarged / reduced at an enlargement / reduction rate such that the plurality of feature points P1 to P4 coincide with the respective average relative positions. For example, the distance between the base point O and the feature point P1 is OP1, and the distance between the base point O and an arbitrary point X1 in the region R1 is OX1. Further, the distance from the base point O to the average relative position of the feature point P1 is OP1 ′. Further, it is assumed that an arbitrary point X1 in the region R1 is moved to the point X1 ′ by the enlargement / reduction operation, and the distance between the base point O and the point X1 ′ is OX1 ′. At this time, the following relationship is established among the distances OP1, OP1 ′, OX1, and OX1 ′.

  Conversely, the point X1 may be moved to the point X1 'so that the relationship of the expression (12) is established. In this manner, the areas R1 to R4 are enlarged / reduced. That is, coordinate conversion processing is performed on all face shape functions f (x, y). The function generated by this coordinate transformation is hereinafter referred to as “modified shape function z ′”. The corrected shape function z 'is given by the following equation.

The partial derivative of the corrected shape function z ′ = g (x, y) is referred to as “corrected relative shape function (g _x , g _y )” corresponding to the relative shape function (f _x , f _y ). The This modified relative shape function (g _x , g _y ) is given by the following equation.

尚、実際には、画像処理と同じく微分は差分に近似して計算される。

Actually, the derivative is calculated by approximating the difference as in the image processing.

Step S306:
Next, the learning data creation unit 32 performs component analysis using the corrected shape function g (x, y) of all the faces, and creates component analysis data. In all the modified shape functions g (x, y) obtained in step S305, a plurality of feature points including the base point O are coincident. Component analysis such as principal component analysis is performed by matching these. For example, it is assumed that there are n modified shape function z '. Among these, the value of the first corrected shape function g (x, y) at the position x, y of certain data (pixel) is represented by z _k ¹ . Here, k is a parameter for representing the position of a two-dimensional pixel in one dimension, and is represented by k = y × w + x (w is the number of pixels in the x direction; see FIG. 2). The value at that position k of the n-th corrected shape function g (x, y) _is expressed by _z ^{k n.} At this time, for the position (pixel) k, an n-order vector Z _k represented by the following equation is defined.

By arranging this vector _Zk by the number of pixels s (s = w × h; see FIG. 2), an s × n matrix Z expressed by the following equation is obtained.

  By performing singular value decomposition on this matrix Z, the following equation is obtained.

Here, U _z is an s × n matrix, V _z ^t is an n × n matrix, and S _z is an n × n diagonal matrix. The principal component can be obtained by using the column vector of the matrix U _z thus obtained.

Step S307:
Similarly, the learning data creation unit 32 performs component analysis using the corrected relative shape functions (g _x , g _y ) of all the faces, and creates component analysis data. A value at a certain position (pixel) k of the first modified relative shape function g _x is represented by d _k ¹ . Also, it represents the value at the position k of the n-th modified relative shape function g _x with d _k ^n. Further, the value of the first corrected relative shape function _gy at the position k is represented by e _k ¹ . Also, it represents the value at the position k of the n-th modified relative shape function g _y in e _k ^n. At this time, n-order vectors D _k and E _k represented by the following equations are defined for the position k.

By arranging these vectors D _k and E _k by the number of pixels s (s = w × h), an s × n matrix D and an s × n matrix E expressed by the following equations are obtained.

  By performing singular value decomposition on these matrices D and E, the following equation is obtained.

_Here, U d, _{U e} is the s × n _{matrix, ^V} d ^{_t, V e} t is the n × n matrix _is a S d, _{S e} is n × n diagonal matrix. The principal components can be obtained by using column vectors of the matrices U _d and U _e thus obtained.

Step S308:
The learning data creation unit 32 obtains information Uz related to the corrected shape function g (x, y), information U _d and U _e related to the corrected relative shape function (gx, gy), and the average of each feature point thus obtained. Data indicating the relative position is stored in the storage unit 33 as the three-dimensional shape learning data 62.

  Next, a method for estimating the three-dimensional absolute shape data 53 indicating the shape function f (x, y) of the person's face from the relative shape data 51 and the feature point position data 52 relating to the certain person described above is shown. FIG. 13 is a flowchart showing the operation of the absolute shape calculation unit 34 of the absolute shape analysis apparatus 30.

Step S311:
First, the relative shape data 51 created by the relative shape analysis device 10 and the feature point position data 52 created by the feature point position search device 20 are input to the absolute shape calculation unit 34 via the input unit 31.

Step S312:
Next, the absolute shape calculation unit 34 calculates feature points in the relative shape function (f _x , f _y ) indicated in the relative shape data 51 based on the positions of the plurality of feature points indicated in the feature point position data 52. Set. Further, the absolute shape calculation unit 34 sets one of the plurality of feature points as a “base point”. This base point indicates the same location as the base point set in step S303 described above. Furthermore, the absolute shape calculation unit 34 calculates the relative positions of a plurality of feature points other than the base point O with respect to the base point O. Thereby, a relative position is obtained for each of a plurality of feature points on the face of the person being processed.

Step S313:
Next, the absolute shape calculation unit 34 reads the three-dimensional learning data 62 stored in the storage unit 33 and acquires the “average relative position” of each feature point. Then, the absolute shape calculation unit 34 performs coordinate conversion of the relative shape function (f _x , f _y ) so that the plurality of feature points coincide with the respective average relative positions. Specifically, the enlargement / reduction of the peripheral area of each feature point is performed by the same method as in step S305 described above. By this coordinate conversion processing, a corrected relative shape function z ′ = (g _x , g _y ) relating to the face being processed is calculated from the relative shape function (f _x , f _y ) indicated in the relative shape data 51 (formula (14) see).

Step S314:
Next, the absolute shape calculation unit 34 obtains, from the three-dimensional learning data 62, information U _z regarding the corrected shape function g (x, y) of a plurality of persons, information U _d regarding the corrected relative shape function (g _x , g _y ) to get the U _e. Based on these learning data and the corrected relative shape function (g _x , g _y ) calculated in step S313, the absolute shape calculation unit 34 calculates a corrected shape function g (x, y) related to the face being processed. . Specifically, when the values of the modified relative shape functions (g _x , g _y ) at a certain position k (k = y × w + x) are g _x (k) and g _y (k), respectively, the absolute shape calculation unit 34 creates a column vector G and an n-th column vector c given by the following equations.

By using the column vector G, the column vector c, and the information U _d and U _e indicated by the three-dimensional shape learning data 62, a functional E2 given by the following equation is defined.

Then, the absolute shape calculation unit 34 calculates an nth-order column vector c that minimizes the functional E2. Further, using the calculated column vector c and the information U _z indicated by the three-dimensional shape learning data 62, an s-th column vector G2 given by the following equation is obtained.

  Here, s (s = w × h) is the number of pixels, and g (k) is a value of the corrected shape function g (x, y) at the position k (k = y × w + x). That is, the corrected shape function g (x, y) of the face being processed is calculated by the above equation (25).

Step S315:
Next, the absolute shape calculation unit 34 returns the corrected shape function so that the positions of the plurality of feature points in the obtained corrected shape function g (x, y) return to the original positions indicated by the feature point position data 52. g a (x, y) coordinate transformation. More specifically, the surrounding area of each feature point is enlarged / reduced by the reverse process of step S305 described above. By this coordinate conversion processing, a shape function f (x, y) relating to the face being processed is calculated from the corrected shape function g (x, y). That is, the three-dimensional shape of the face being processed is estimated.

Step S316:
The absolute shape calculation unit 34 outputs the three-dimensional absolute shape data 53 indicating the shape function f (x, y) thus obtained via the output unit 35. The absolute shape calculation unit 34 may store the three-dimensional absolute shape data 53 in the storage unit 33. As described above, the absolute shape analysis apparatus 30 refers to the three-dimensional shape learning data 62 and the feature point position data 52, so that the relative shape function (f _x , f _y ) of a certain face indicated in the relative shape data 51 is obtained. Is converted into the shape function f (x, y) of the face.

As described above, according to the three-dimensional shape estimation system 1 and the three-dimensional shape estimation method according to the present invention, the three-dimensional absolute shape data 53 can be obtained from certain face image data 50. That is, it is possible to estimate the three-dimensional shape of an object from a single two-dimensional image in which the object is reflected. Here, it is not necessary to photograph the object with a special photographing apparatus such as a stereo / multi-lens camera. Therefore, the cost is reduced. Further, it is not necessary to store information at the time of shooting, and there are no restrictions on the shooting environment of the two-dimensional image. In the process course, from the two-dimensional image, illumination basis B and the relative shape function (f _{x, f y)} of the object is estimated. Further, it is possible to automatically find the feature point of the object without manually specifying it.

(Second Embodiment)
The functions according to the first embodiment can also be realized by a computer system and a computer program. FIG. 14 is a block diagram showing a configuration of a three-dimensional shape estimation system according to the second embodiment of the present invention. The three-dimensional shape estimation system 1 ′ includes a storage device 70, an arithmetic processing device 71, an input device 72, a display device 73, a data input / output interface 74, and a three-dimensional shape estimation program 80.

  The storage device 70 is a memory such as a hard disk drive (HDD) or a RAM. The storage device 70 stores the face image data 50, the relative shape data 51, the feature point position data 52, the three-dimensional absolute shape data 53, the generalized illumination base data 61, and the three-dimensional shape learning data 62. . The arithmetic processing device 71 is connected to each device, and exchanges data and performs various operations.

  Examples of the input device 72 include a keyboard and a mouse. By using the input device 72, the user can input data and specify various commands. The display device 73 is a display, and various information is notified to the user by the display device 73. The user can input a new command or data based on the notified information. The data input / output interface 74 is used for inputting the face image data 50 and the three-dimensional shape data 60. Therefore, the data input / output interface 74 is connected to an imaging device such as a range finder or a digital camera, a CD / DVD drive, another terminal, and the like.

  The three-dimensional shape estimation program 80 is a software program executed by the arithmetic processing device 71. The three-dimensional shape estimation program 80 includes a relative shape analysis unit 81, a feature point position search unit 82, an absolute shape analysis unit 83, and a learning data creation unit 84. These program groups may be stored in the storage device 70. By executing these programs by the arithmetic processing unit 71, the same functions as the functions provided in the first embodiment are realized.

  Specifically, the arithmetic processing unit 71 creates the relative shape data 51 from the face image data 50 in accordance with the instruction of the relative shape analysis unit 81 of the three-dimensional shape estimation program 80. This is the same as the function of the relative shape calculation unit 14 in FIG. 4 (see FIG. 6). In addition, the arithmetic processing unit 71 creates feature point position data 52 from the face image data 50 and the relative shape data 51 in accordance with an instruction from the feature point position search unit 82 of the three-dimensional shape estimation program 80. This is the same as the function of the feature point extraction unit 22 in FIG. 7 (see FIG. 8). Further, the arithmetic processing unit 71 creates the three-dimensional absolute shape data 53 from the relative shape data 51 and the feature point position data 52 in accordance with the instruction of the absolute shape analysis unit 83 of the three-dimensional shape estimation program 80. This is the same as the function of the absolute shape calculation unit 34 in FIG. 10 (see FIG. 13). Further, the arithmetic processing unit 71 creates the generalized illumination base data 61 and the three-dimensional shape learning data 62 according to the instruction of the learning data creation unit 84 of the three-dimensional shape estimation program 80. This is the same as the function of the learning data creation units 12 and 32 in FIGS. 4 and 10 (see FIGS. 5 and 11).

According to the three-dimensional shape estimation system 1 ′ and the three-dimensional shape estimation program 80 described above, it is possible to estimate the three-dimensional shape of an object from a single two-dimensional image in which a certain object is reflected. . Here, since it is not necessary to use a special measuring apparatus, cost is reduced. Further, from the single two-dimensional image, illumination basis B and the relative shape function f _x of that _object, it is possible to estimate the f _y. Furthermore, the feature point of the object can be automatically found from the single two-dimensional image.

(Third embodiment)
FIG. 15 is a block diagram showing a configuration of an image generation system according to the third embodiment of the present invention. The image generation system 100 includes a relative shape analysis device 110, a feature point position search device 120, an absolute shape analysis device 130, and an image generation device 140. This image generation system 100 uses a two-dimensional image (illumination conversion image) having different illumination conditions from a face image data 50 (see FIG. 2) indicating a two-dimensional image of a certain face, or a different face orientation with different illumination conditions. The conversion image data 55 indicating the two-dimensional image (rotation conversion image) is created.

  The relative shape analyzer 110 in the present embodiment has the same configuration as the relative shape analyzer 10 in the first embodiment. That is, the relative shape analysis device 110 creates the relative shape data 51 from the face image data 50 and outputs the relative shape data 51 to the special vertex position search device 120 and the absolute shape analysis device 130. In the present embodiment, the relative shape analysis apparatus 110 outputs the illumination base data 54 indicating the illumination base B calculated in step S114 to the image generation apparatus 140.

  The feature point position search device 120 in the present embodiment has the same configuration as the feature point position search device 20 in the first embodiment. That is, the feature point position search device 120 creates feature point position data 52 from the face image data 50 and the relative shape data 51, and outputs the feature point position data 52 to the absolute shape analysis device 130.

  The absolute shape analyzer 130 in the present embodiment has the same configuration as the absolute shape analyzer 30 in the first embodiment. In other words, the absolute shape analyzer 130 creates the three-dimensional absolute shape data 53 from the relative shape data 51 and the feature point position data 52. In the present embodiment, the three-dimensional absolute shape data 53 is output to the image generation device 140.

  The image generation device 140 receives the three-dimensional absolute shape data 53 and the illumination base data 54 and creates converted image data 55. FIG. 16 is a block diagram illustrating a configuration of the image generation device 140 according to the present embodiment. The image generation apparatus 140 includes an input unit 141, an image generation unit 142, a storage unit 145, and an output unit 146. The image generation unit 142 includes a two-dimensional image conversion unit 143 and a three-dimensional image conversion unit 144. The storage unit 145 stores converted image data 55. The image generation unit 142 is a unit for creating the converted image data 55, and is realized by a CPU and a computer program. Examples of the storage unit 145 include a hard disk drive and a memory.

FIG. 17 is a flowchart showing the operation of the image generation apparatus 140 according to the present invention.
Step S401:
First, the illumination base data 54 from the relative shape analyzer 110 and the three-dimensional absolute shape data 53 from the absolute shape analyzer 130 are input to the image generation unit 142 via the input unit 141.

Step S402:
The two-dimensional image conversion unit 143 multiplies an arbitrary illumination intensity vector s by a certain person's illumination base B indicated by the illumination base data 54 as in the above equation (6). As a result, a two-dimensional face image (illumination conversion image) under an illumination condition different from that of the two-dimensional face image indicated by the face image data 50 is obtained. The two-dimensional image conversion unit 143 outputs illumination conversion image data 56 indicating the illumination conversion image.

  When the rotation conversion image is not necessary (step S403; No), the created illumination conversion image data 56 is output as the conversion image data 55 via the output unit 146 (step S406). The illumination converted image data 56 may be stored in the storage unit 145 as converted image data 55. As described above, according to the present embodiment, it is possible to create illumination conversion images with different illumination conditions using only the relative shape analysis device 110 and the image generation device 140.

Step S404:
When a rotation conversion image is necessary (step S403; Yes), the 3D image conversion unit 144 receives the illumination conversion image data 56 created in step S402 in addition to the 3D absolute shape data 53. That is, the three-dimensional image conversion unit 144 acquires “estimated three-dimensional shape (shape function f (x, y))” and “illumination conversion image with different illumination conditions” regarding the face being processed. Then, the three-dimensional image conversion unit 144 combines the illumination conversion image and the three-dimensional shape, that is, attaches the illumination conversion image to the three-dimensional shape. As a result, a “new three-dimensional shape with different illumination conditions” is created.

Step S405:
Next, the three-dimensional image conversion unit 144 rotates the new three-dimensional shape three-dimensionally and sets the face direction to a desired direction. As a result, a two-dimensional image (rotation-converted image) having illumination conditions and face orientation different from those of the original two-dimensional image is obtained. The three-dimensional image conversion unit 144 outputs rotation conversion data 57 indicating the rotation conversion image.

Step S406:
The image generation unit 142 outputs at least one of the illumination conversion image data 56 and the rotation conversion image data 57 created as described above as the conversion image data 55. The converted image data 55 is output to the outside via the output unit 146 or stored in the storage unit 145.

  As described above, according to the image generation system 100 and the image generation method according to the present invention, an image having a different orientation and illumination condition of an object is generated from a single two-dimensional image in which the object is reflected. It becomes possible to do.

(Fourth embodiment)
The function according to the third embodiment can also be realized by a computer system and a computer program. FIG. 18 is a block diagram showing a configuration of an image generation system according to the fourth embodiment of the present invention. The image generation system 100 ′ includes a storage device 170, an arithmetic processing device 171, an input device 172, a display device 173, a data input / output interface 174, and an image generation program 180.

  The storage device 170 is a memory such as a hard disk drive (HDD) or a RAM. In the storage device 170, the face image data 50, the relative shape data 51, the feature point position data 52, the three-dimensional absolute shape data 53, the illumination base data 54, the converted image data 55, the generalized illumination base data 61, and Three-dimensional shape learning data 62 is stored. The arithmetic processing device 71 is connected to each device, and exchanges data and performs various operations.

  Examples of the input device 172 include a keyboard and a mouse. The user can input data and specify various commands by using the input device 172. The display device 173 is a display, and the display device 173 notifies the user of various types of information. The user can input a new command or data based on the notified information. The data input / output interface 174 is used to input the face image data 50 and the three-dimensional shape data 60. Therefore, the data input / output interface 174 is connected to an imaging device such as a range finder or a digital camera, a CD / DVD drive, another terminal, and the like.

  The image generation program 180 is a software program executed by the arithmetic processing device 171. The image generation program 180 includes a relative shape analysis unit 181, a feature point position search unit 182, an absolute shape analysis unit 183, an image generation unit 184, and a learning data creation unit 185. These program groups may be stored in the storage device 170. By executing these programs by the arithmetic processing unit 171, the same function as the function provided in the third embodiment is realized.

  Specifically, the arithmetic processing unit 171 creates the relative shape data 51 and the illumination base data 54 from the face image data 50 in accordance with an instruction of the relative shape analysis unit 181 of the image generation program 180. This is the same as the function of the relative shape analyzer 110 in FIG. Further, the arithmetic processing unit 171 creates the feature point position data 52 from the face image data 50 and the relative shape data 51 in accordance with the command of the feature point position search unit 182 of the image generation program 180. This is the same as the function of the feature point position search device 120 in FIG. Further, the arithmetic processing unit 171 creates the three-dimensional absolute shape data 53 from the relative shape data 51 and the feature point position data 52 in accordance with the instruction of the absolute shape analysis unit 183 of the image generation program 180. This is the same as the function of the absolute shape analyzer 130 in FIG. Further, the arithmetic processing unit 171 creates the converted image data 55 from the illumination base data 54 and the three-dimensional absolute shape data 53 in accordance with an instruction of the image generation unit 184 of the image generation program 180. This is the same as the function of the image generation apparatus 140 in FIG. Further, the arithmetic processing unit 171 creates the generalized illumination base data 61 and the three-dimensional shape learning data 62 according to the instruction of the learning data creation unit 185 of the image generation program 180. This is the same function as the learning data creation units 12 and 32 in FIGS.

  As described above, according to the image generation system 100 ′ and the image generation program 180 according to the present invention, from a single two-dimensional image in which a certain object is reflected, an image in which the direction and lighting conditions of the object are different. Can be generated.

  The present invention described above is expected to be used particularly in the security field such as face personal authentication technology, the field of beauty / shaping, the field of amusement, and the like.

FIG. 1 is a diagram for explaining a shape function that expresses a three-dimensional shape of a certain face. FIG. 2 is a schematic diagram showing a two-dimensional image of a certain face. FIG. 3 is a block diagram showing the configuration of the three-dimensional shape estimation system according to the first embodiment of the present invention. FIG. 4 is a block diagram showing the configuration of the relative shape analyzer according to the present invention. FIG. 5 is a flowchart showing the operation of the relative shape analyzer according to the present invention. FIG. 6 is a flowchart showing the operation of the relative shape analyzer according to the present invention. FIG. 7 is a block diagram showing the configuration of the feature point position search apparatus according to the present invention. FIG. 8 is a flowchart showing the operation of the feature point position search apparatus according to the present invention. FIG. 9 is a diagram for explaining the operation of the feature point position search apparatus according to the present invention. FIG. 10 is a block diagram showing the configuration of the absolute shape analyzer according to the present invention. FIG. 11 is a flowchart showing the operation of the absolute shape analyzer according to the present invention. FIG. 12 is a diagram for explaining the operation of the absolute shape analyzer according to the present invention. FIG. 13 is a flowchart showing the operation of the absolute shape analyzer according to the present invention. FIG. 14 is a block diagram showing a configuration of a three-dimensional shape estimation system according to the second embodiment of the present invention. FIG. 15 is a block diagram showing a configuration of an image generation system according to the third embodiment of the present invention. FIG. 16 is a block diagram showing a configuration of an image generation apparatus according to the present invention. FIG. 17 is a flowchart showing the operation of the image generating apparatus according to the present invention. FIG. 18 is a block diagram showing a configuration of an image generation system according to the fourth embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 3D shape estimation system 10 Relative shape analysis apparatus 12 Learning data creation part 13 Storage part 14 Relative shape calculation part 20 Feature point position search apparatus 22 Feature point extraction part 23 Color / luminance feature point extraction part 24 Shape feature point extraction part 25 Storage unit 30 Absolute shape analysis device 32 Learning data creation unit 33 Storage unit 34 Absolute shape calculation unit 50 Face image data 51 Relative shape data 52 Feature point position data 53 Three-dimensional absolute shape data 54 Illumination base data 55 Conversion image data 56 Illumination conversion Image data 57 Rotational transformation image data 60 Three-dimensional shape data 61 Generalized illumination base data 62 Three-dimensional shape learning data 70 Storage device 71 Arithmetic processing device 72 Input device 73 Display device 74 Data input / output interface 80 Three-dimensional shape estimation program 81 Relative shape analysis unit 82 the feature point position Search unit 83 Absolute shape analysis unit 84 Learning data creation unit 100 Image generation system 110 Relative shape analysis device 120 Feature point position search device 130 Absolute shape analysis device 140 Image generation device 142 Image generation unit 143 Two-dimensional image conversion unit 144 Three-dimensional image Conversion unit 145 Storage unit 170 Storage device 171 Arithmetic processing device 172 Input device 173 Display device 174 Data input / output interface 180 Image generation program 181 Relative shape analysis unit 182 Feature point position search unit 183 Absolute shape analysis unit 184 Image generation unit 185 Learning data Creation department

Claims (12)

Prepare in advance illumination basis data common to a plurality of objects and three-dimensional shape information corresponding to the illumination basis data,
Obtaining a combination of the illumination base data representing the image most similar to a two-dimensional image of an object, the weight of the illumination base data, and an illumination vector;
Relative differentiation of the illumination base of the certain object determined from the weight of the illumination base data and the illumination base data to obtain relative shape information,
Obtaining a positional relationship of feature points between the two-dimensional image and the relative shape information;
A three-dimensional shape estimation method, wherein the three-dimensional shape information of the certain object is corrected by correcting the three-dimensional shape information using the positional relationship.   The three-dimensional shape estimation method according to claim 1, wherein the illumination base data common to the plurality of objects is generated from three-dimensional shape information common to the plurality of objects.   The three-dimensional shape estimation method according to claim 1, wherein the object is a human face. Prepare in advance illumination basis data common to a plurality of objects and three-dimensional shape information corresponding to the illumination basis data,
Obtaining a combination of the illumination base data representing the image most similar to a two-dimensional image of an object, the weight of the illumination base data, and an illumination vector;
Relative differentiation of the illumination base of the certain object determined from the weight of the illumination base data and the illumination base data to obtain relative shape information,
Obtaining a positional relationship of feature points between the two-dimensional image and the relative shape information;
A three-dimensional shape estimation program that causes a computer to execute correction of the three-dimensional shape information using the positional relationship to obtain three-dimensional shape information of the certain object.   The three-dimensional shape estimation program according to claim 4, wherein the illumination base data common to the plurality of objects is generated from three-dimensional shape information common to the plurality of objects.   The three-dimensional shape estimation program according to claim 4, wherein the object is a human face. A storage device storing illumination basis data common to a plurality of objects and three-dimensional shape information corresponding to the illumination basis data;
A combination of the illumination base data that represents an image most similar to a two-dimensional image of an object, a weight of the illumination base data, and an illumination vector is obtained, and the object is determined from the weight of the illumination base data and the illumination base data. A relative shape analyzer that obtains relative shape information by partial differentiation of the illumination base;
A feature point position search device for obtaining a positional relationship of feature points between the two-dimensional image and the relative shape information;
A three-dimensional shape estimation system, comprising: an absolute shape analyzer that corrects the three-dimensional shape information using the positional relationship to obtain the three-dimensional shape information of the certain object.   The three-dimensional shape estimation system according to claim 7, wherein the illumination base data common to the plurality of objects is generated from three-dimensional shape information common to the plurality of objects.   The three-dimensional shape estimation system according to claim 7, wherein the object is a human face. Storage means storing illumination basis data common to a plurality of objects and three-dimensional shape information corresponding to the illumination basis data;
A combination of the illumination base data that represents an image most similar to a two-dimensional image of an object, a weight of the illumination base data, and an illumination vector is obtained, and the object is determined from the weight of the illumination base data and the illumination base data. Relative shape analysis means for partial differentiation of the illumination base to obtain relative shape information;
Feature point position searching means for obtaining a positional relationship of feature points between the two-dimensional image and the relative shape information;
A three-dimensional shape estimation apparatus comprising: an absolute shape analysis unit that corrects the three-dimensional shape information using the positional relationship to obtain three-dimensional shape information of the certain object.   The three-dimensional shape estimation apparatus according to claim 10, wherein the illumination base data common to the plurality of objects is generated from three-dimensional shape information common to the plurality of objects.   The three-dimensional shape estimation apparatus according to claim 10, wherein the object is a human face.

Images