JP2001084362A - Method and device for estimating light source direction and three-dimensional shape and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate the direction of a light source and the three-dimensional shape of an object from an image obtained by photographing the object under a single light source. SOLUTION: In the image including the face of a human being, two-dimensional image data S0 for indicating the image of a face part and three-dimensional shape data S1 for indicating the three-dimensional shape of the face are positioned. After positioning, plural measurement points are set on the three-dimensional shape data S1. Plural virtual light sources are set, the cosine of a direction vector to the respective measurement points and a normal vector at the respective measurement points is obtained for the respective virtual light sources and the relation of it and the data value of the two-dimensional image data S0 at the respective measurement points is plotted. Then, the direction of the virtual light source for which the relation becomes a straight line is defined as a temporary light source direction at the time of photographing. The three-dimensional shape data S1 are corrected so as to position a plot on the straight line based on the temporary light source direction and the correction of the temporary light source direction and the three-dimensional shape data is repeated after the correction. Thus, the light source direction at the time of photographing and the three-dimensional shape of the object are estimated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an estimation method for estimating the direction of a light source at the time of photographing and the three-dimensional shape of the object from an image obtained by photographing the object using a single light source. The present invention relates to a computer-readable recording medium on which a program for causing a computer to execute an apparatus and an estimation method is recorded.

[0002]

2. Description of the Related Art A technique for synthesizing a real image with a virtual image such as computer graphics (CG) without discomfort is called mixed reality technology, and plays an important role in various fields such as communication, traffic, and entertainment. It is expected to fulfill. Thus, in order to combine the real image and the virtual image without discomfort, it is necessary to combine them with geometric, optical and temporal consistency. In particular, regarding the optical consistency, unless the direction and intensity of the light source in the real image and the direction and intensity of the light source in the virtual image are made to coincide with each other, the synthesized image will be very uncomfortable, so that it is extremely important. Therefore, a method for estimating the intensity distribution of a light source in an image has been proposed (Sato et al., Estimation of Real Illumination Environment Based on Soft Shadow of Real Object, Computer Vision and Image Media, 11
0-3, 1998.3.19, PP17-24).

[0003] This method is a method of estimating the intensity distribution of environmental illumination based on a soft shadow of a real object. The intensity distribution of environmental illumination is estimated by using the shadow of an object having a known three-dimensional shape in an image. Estimation. Specifically, the relationship between the brightness of the virtual light source and the data value at a certain measurement point on the object is represented by a determinant, and the intensity distribution of the light source is obtained by calculating the inverse matrix of the matrix in the determinant. It is an estimate. Then, by estimating the intensity distribution of the light source in the image by such a method, the intensity distribution of the light source in the virtual image can be matched with the intensity distribution of the light source in the real image. It can be synthesized without any.

[0004] Further, by an image paste method,
A method for generating an object image illuminated from an arbitrary light source direction has also been proposed (Katsushi Ikeuchi, Automatic Generation of Virtual Reality Models: Optical Aspects, Faculty of Science, PRMU98-122 (1998-11), P
P75-82). This method is a method of creating an image illuminated from an arbitrary light source direction as a linear sum of a plurality of sample images obtained by photographing an object by changing a light source in a plurality of directions.

[0005]

However, according to the method of Sato et al., It is necessary to set the position of the measurement point to a position that can be calculated inversely and that is mathematically linearly independent. The operation is very troublesome because the position of the point needs to be determined manually. Also, the three-dimensional shape of the object needs to be known. Further, the method in the pond described above is not a practical method because it requires a very long time to perform an operation because an image of a target object taken in advance from as many as 80 light source directions must be created.

The present invention has been made in view of the above circumstances, and has an estimation method and apparatus capable of easily estimating the direction of a light source and the three-dimensional shape of an object, and records a program for causing a computer to execute the estimation method. It is an object of the present invention to provide a computer-readable recording medium according to the present invention.

[0007]

SUMMARY OF THE INVENTION A method for estimating the direction of a light source and the three-dimensional shape of an object according to the present invention comprises: An estimation method for estimating a direction of the light source and a three-dimensional shape of the object based on an image obtained by capturing an image under a light source, the object image data representing an image of the object on the image. Values, and an approximate object represented by the three-dimensional shape data after the object image data is aligned with three-dimensional shape data representing a general three-dimensional shape of the object, and in an arbitrary plurality of directions. The direction of the light source and the three-dimensional shape of the object are estimated based on the set positional relationship with the virtual light source.

[0008] Here, "shooting with a substantially single light source" means that the presence of other light sources can be ignored, as in the case of shooting under sunlight in fine weather. This means that shooting is performed in a situation substantially equivalent to using only the light source.

Further, for example, when the object is a person's face, the shape of the face varies from person to person, but there is a shape that almost matches any shape of the face. Therefore, here, three-dimensional shape data representing a three-dimensional shape that roughly matches the shape of such an object is referred to as “general 3D shape data”.
Three-dimensional shape data representing a three-dimensional shape ".

The skin color of a person's face can be regarded as having a substantially constant surface reflectance.

The alignment between the object image data and the three-dimensional shape data is performed by adjusting the orientation of the object in the image, the orientation of the three-dimensional shape of the object, the center position, the two-dimensional direction, that is, the size in the direction in which the object image data exists. This means that the three-dimensional shape data and the object image data are matched so that the data and the like match. With such matching,
The position of the three-dimensional shape data is aligned only with the object image data in the two-dimensional direction. In the present invention, the shape of the object represented by the three-dimensional shape data aligned as described above is regarded as an approximate object. Things.

In the estimation method according to the present invention,
A plurality of measurement points are set at a portion corresponding to a portion where the surface reflectance of the object on the approximate object is substantially constant, and a direction from the virtual light source set in the arbitrary plurality of directions to the plurality of measurement points Calculating a cosine between the vector and the normal vector at the plurality of measurement points;
The relationship between the data values of the object image data corresponding to the plurality of measurement points is plotted for each direction of the plurality of virtual light sources, and the direction of the virtual light source in which the plot is substantially straight is estimated as a temporary light source direction. Correcting the cosine of the plot so that the plot obtained for the provisional light source direction is located on the straight line, and the cosine between the direction vector and the normal vector at the measurement point is corrected. Correcting the position of the measurement point so as to approach the cosine to obtain a corrected measurement point, calculating the cosine with the corrected measurement point as the new measurement point, the plot, estimating the temporary light source direction,
The correction of the cosine and the acquisition of the corrected measurement point are repeated a predetermined number of times, the temporary light source direction obtained as a result of the repetition processing is estimated as the direction of the light source, and the position of the correction measurement point obtained as a result of the repetition processing is determined. Preferably, the determined shape is estimated as a three-dimensional shape of the object.

Here, the "direction vector" can be calculated from the coordinate position of the virtual light source and the coordinate position of the measurement point, and the "normal vector" is the coordinate of the measurement point on the three-dimensional shape data after alignment. It can be obtained based on the data value of the position. The cosine can be obtained from the inner product of the direction vector and the normal vector.

"Modifying the cosine of a plot so that the plot is located on a straight line" means changing the value of the cosine at the plotted measurement point and moving the plot so that the plot is located on a straight line. It means to let.

"Correcting the position of a measurement point so as to approach the corrected cosine" means that the cosine between the direction vector and the normal vector at the measurement point before correction is completely equal to the corrected cosine. Instead of matching, the position of the measurement point is corrected to be close to the corrected cosine value without exceeding the corrected cosine value, so that the cosine between the direction vector and the normal vector at the corrected measurement point is obtained. Is to be a value close to the corrected cosine. Here, "correcting the position of the measurement point" means that the three-dimensional shape data is aligned in the two-dimensional direction in the object image data, and thus the position is not corrected in the two-dimensional direction. This refers to correcting the position in the depth direction with respect to the image data. Specifically, assuming that the plane on which the object image data exists is an xy plane, the position of the measurement point is corrected only in the z direction.

In the present invention, the three-dimensional shape is estimated only at a portion where the surface reflectance of the object is substantially constant, but the entire surface reflectance of the object having the three-dimensional shape is substantially estimated. If it is constant, the entire three-dimensional shape of the object can be estimated.

In the estimation method according to the present invention,
Based on the data values of the object image data at the plurality of measurement points, determine whether there is a specular reflection measurement point that is specularly reflected at the plurality of measurement points, when the specular reflection measurement point is present Is based on the specular reflection measurement point,
By selecting the direction of the virtual light source, which is likely to be the direction of the light source, by the virtual light source in the selected direction,
It is preferable to calculate the cosine.

Further, in the estimation method according to the present invention, the estimation of the direction of the light source is performed by judging whether or not the plot is substantially a straight line based on an evaluation function using a least square method. Is preferred.

In this case, it is preferable that the predetermined number of times is a number of times until the evaluation function becomes equal to or less than a predetermined value.

In the estimation method according to the present invention,
It is preferable to estimate the intensity of the light source based on the inclination of a straight line set by the plot used for estimating the direction of the light source.

Further, in the estimation method according to the present invention, it is preferable that the object is a human face.

Further, it is preferable that the three-dimensional shape data is polygon data representing a three-dimensional shape of the object.

The estimating apparatus according to the present invention is characterized in that the light source based on an image obtained by shooting an object having a three-dimensional shape including a portion having a substantially constant surface reflectance under a substantially single light source. An estimation device for estimating the direction of the object and the three-dimensional shape of the object, comprising: a data value of object image data representing an image of the object on the image; and a general three-dimensional image of the object image data and the object. The direction of the light source is determined based on the positional relationship between the approximate object represented by the three-dimensional shape data after the alignment with the three-dimensional shape data representing the shape and the virtual light source set in a plurality of arbitrary directions. And means for estimating the three-dimensional shape of the object.

In the estimation device according to the present invention,
The means for estimating the direction of the light source and the three-dimensional shape of the object is a measurement point for setting a plurality of measurement points on a portion of the approximate object corresponding to the portion where the surface reflectance of the object is substantially constant. Setting means, a direction vector from the virtual light source set in any of the plurality of directions to the plurality of measurement points, and a cosine calculation means for calculating a cosine formed by a normal vector at the plurality of measurement points; and Plotting means for plotting the relationship between the data values of the object image data corresponding to the plurality of measurement points for each direction of the plurality of virtual light sources, and setting the direction of the virtual light source in which the plot is substantially straight as a temporary light source Temporary light source direction estimating means for estimating the direction, cosine correcting means for correcting the cosine in the plot so that the plot for which the temporary light source direction is obtained is located on the straight line, Measuring point correcting means for correcting a position of the measuring point so as to obtain a corrected measuring point so that a cosine between the direction vector and the normal vector at a point approaches the corrected cosine; and the corrected measuring point. Calculating the cosine, setting the plot, estimating the provisional light source direction, correcting the cosine, and obtaining the corrected measurement point a predetermined number of times as a new measurement point. Control means for controlling the plotting means, the temporary light source direction estimating means, the cosine correcting means and the measuring point correcting means, and estimating the temporary light source direction obtained as a result of the repetitive processing as the light source direction Further, it is preferable to include an estimating means for estimating a shape determined by the position of the corrected measurement point obtained as a result of the repetitive processing as a three-dimensional shape of the object.

In the estimating apparatus according to the present invention,
Based on the data values of the object image data at the plurality of measurement points, determine whether there is a specular reflection measurement point for specular reflection at the plurality of measurement points, when the specular reflection measurement point is present Comprises, based on the specular reflection measurement point, selecting means for selecting the direction of the virtual light source having a high probability of being the direction of the light source, and the cosine calculating means, by the virtual light source in the selected direction, It is preferably a means for calculating a cosine.

In the estimating apparatus according to the present invention, the provisional light source direction estimating means may determine whether the plot of the light source direction is substantially a straight line based on an evaluation function using a least squares method. It is preferable to provide a determination means for determining whether or not the determination is made.

In the estimating apparatus according to the present invention,
It is preferable that the estimating means is means for estimating the intensity of the light source based on a slope of a straight line set by a plot used for estimating the direction of the light source.

In the estimating device according to the present invention,
Preferably, the object is a human face.

Further, it is preferable that the three-dimensional shape data is polygon data representing a three-dimensional shape of the object.

The estimation method according to the present invention may be provided as a program for causing a computer to execute the program on a computer-readable recording medium.

[0031]

According to the present invention, if only the data value of the object image data and the positional relationship between the approximate object and the virtual light source set in arbitrary plural directions are known, the direction of the light source and the three-dimensional shape of the object can be obtained. Can be estimated. Therefore, unlike the method of Sato et al., There is no need to find measurement points at which the inverse matrix can be calculated, and there is no need to perform a matrix operation.
The direction of the light source at the time of photographing can be estimated by relatively simple calculation, and even if the three-dimensional shape of the object is unknown, the three-dimensional shape can be estimated. Further, unlike the method in the above-mentioned pond, there is no need to capture images from many directions, so that the direction of the light source and the 3
A dimensional shape can be estimated. And this gives
When synthesizing a real image and a virtual image such as CG, it is possible to perform synthesis after correcting the virtual image so that it is illuminated from an appropriate light source based on the direction of the light source estimated in the real image. As a result, it is possible to obtain a composite image without a sense of discomfort. Further, in an encoding method such as MPEG-4 in which images are transferred for each object and synthesized at a stage of viewing a moving image, when an object includes a real image and a virtual image, the illumination direction of the real image is changed. Accordingly, the lighting direction of the virtual image can be corrected and synthesized, thereby obtaining a synthesized moving image without a sense of discomfort. Furthermore, since an object included in the real image can be represented as an appropriate three-dimensional image, a three-dimensional image without a sense of incongruity can be generated from the real image. Furthermore, in the case of an image obtained by performing shooting in sunlight, the backlight correction and the density correction can be easily performed on the image by estimating the direction of the sunlight in the image.

In particular, when the position of the virtual light source coincides with the light source position at the time of photographing the object, the direction vector from the light source to a certain measurement point on the object having a substantially constant surface reflectance and the normal line at the measurement point It is preferable to estimate the direction of the light source at the time of shooting based on the characteristic that the relationship between the cosine formed by the vector and the intensity value of the reflected light at the measurement point, that is, the data value of the object image data is located on a straight line. That is, after performing the alignment between the object image data and the three-dimensional shape data, a plurality of measurements are performed on a portion corresponding to a portion where the surface reflectance of the object on the approximate object specified by the three-dimensional shape data is substantially constant. A point is set, and a cosine between a direction vector from the measurement point toward the virtual light source and a normal vector at the measurement point is obtained. Then, the relationship between the cosine and the data value of the object image data at the measurement point is plotted for each of the plurality of virtual light sources, and the direction of the virtual light source in which these relationships are located on a straight line is the direction of the temporary light source at the time of shooting And Then, the value of the cosine in each plot is corrected so that the plot in the provisional light source direction is located on a straight line, and the cosine between the direction vector and the normal vector at the measurement point at which the plot in the provisional light source direction is obtained. Modifies the position of the measurement point so that it substantially matches the corrected cosine to obtain a corrected measurement point.

Thereafter, the calculation, plotting, estimating the temporary light source direction, correcting the cosine, and acquiring the corrected measurement point using the corrected measurement point as a new measurement point are repeated a predetermined number of times. Estimate the direction as the light source direction at the time of shooting,
It is preferable to estimate the shape determined by the position of the corrected measurement point as the three-dimensional shape of the object.

Before estimating the temporary light source direction, it is determined whether or not there is a specular reflection measurement point at a plurality of measurement points.
When there is a specular reflection measurement point, the number of virtual light sources used for estimating the light source direction is selected by selecting the direction of the virtual light source that is likely to be the light source direction at the time of shooting based on the specular reflection measurement point. Can be reduced, so that the calculation time can be shortened. In other words, the existence of the measurement point that performs specular reflection means that the light emitted from the actual light source to the measurement point is specularly reflected, and is proceeding in the direction of the photographing unit that photographs the object. Therefore,
The normal vector at the specular reflection measurement point divides the angle between the vector from the light source to the specular reflection measurement point and the vector from the specular reflection measurement point to the imaging means into two equal parts. Therefore, if there is a specular reflection measurement point,
The direction of the virtual light source used for the calculation can be narrowed down based on the normal vector at the measurement point and the vector from the specular reflection measurement point to the photographing means.

Also, by estimating the intensity of the light source, when the real image and the virtual image are combined as described above, the virtual image can be appropriately adjusted based on the intensity of the light source estimated in the real image. Combining can be performed after the lighting has been corrected so that a combined image with less discomfort can be obtained.

Further, if the plane on which the measurement point is located is determined by using the three-dimensional shape data as polygon data, the normal vector at the measurement point can be immediately calculated from the equation defining this plane. Can be more easily estimated.

[0037]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a schematic block diagram showing the configuration of an estimating device for estimating the three-dimensional shape of an object and the direction of a light source according to an embodiment of the present invention. As shown in FIG. 1, the estimation device according to the present embodiment is configured to represent an image including a face of a person.
Three-dimensional image data S0 and three-dimensional image data
Positioning means 1 for performing alignment with three-dimensional shape data S1 to obtain composite data S2, and three-dimensional shape data S1
And an estimating means 3 for estimating the direction of the light source and the three-dimensional shape of the face in the image including the person based on the combined data S2. In the present embodiment, it is assumed that there is a single light source and the surface reflectance of the flesh-colored portion of the face of the person is substantially constant.

The positioning means 1 includes two-dimensional image data S0 representing a face image of a person shown in FIG.
The position is aligned with the three-dimensional shape data S1 approximately representing the three-dimensional shape of the person's face shown in FIG. The three-dimensional shape data S1 can change its direction, size, and center position in accordance with the face image represented by the two-dimensional image data S0, and is polygon data that defines a three-dimensional shape by a plurality of surfaces. ing. In this alignment, the two-dimensional image data S0 and the three-dimensional shape data S1 are displayed on a monitor (not shown), and the three-dimensional shape represented by the three-dimensional shape data S1 is manually operated while observing the displayed image. This is performed by matching the orientation, size, and center position with the face image represented by the two-dimensional image data S0. By performing such positioning, synthesized data S2 representing a synthesized image as shown in FIG. 2C can be obtained. Here, the composite data S2 includes the two-dimensional image data S0 and the three-dimensional shape data S1 'after the alignment. In FIG. 2, the horizontal direction is the x direction, the vertical direction is the y direction, and the direction perpendicular to the plane of the drawing is the z direction. In addition, the three-dimensional shape of the person's face is approximately represented by the three-dimensional shape data S1 'after the alignment.

Here, assuming that the focal position f of the camera that captured the human image and the distance z0 from the camera to the human face are known, the three-dimensional shape represented by the three-dimensional shape data S1 'after the alignment is obtained. The coordinates (x
w, yw, zw) and the coordinates (xv, yv) at the position corresponding to this position on the human image can be described by the following perspective transformation equations (1) and (2).

Xv = α · f · xw / (f + zw + z0) (1) yv = α · f · yw / (f + zw + z0) (2) where α: scale x, y direction: direction on the plane of the human image z direction : Depth direction in human image Here, the three-dimensional shape data S1 is the depth direction (z direction)
, The position on the three-dimensional shape data S1 ′ after the alignment and the position on the two-dimensional image data S0 can be more accurately determined by using the relations of Expressions (1) and (2). Can be assigned.

If the focal position f of the camera or the like is unknown, the correspondence between the three-dimensional shape data S1 'after the alignment and the two-dimensional image data S0 is determined by the above equations (1) and (2). This cannot be done, but since the depth dimension of the person is very small compared to the distance from the camera to the person being the subject, the z-direction can be ignored in such a case. Therefore, in this case, assuming that the coordinates of an arbitrary position in the three-dimensional shape data S1 'after the positioning are (x1, y1, z1), the coordinates at the position on the two-dimensional image data S0 corresponding to this position. May be (x1, y1).

FIG. 3 is a schematic block diagram showing the configuration of the estimating means 3. As shown in FIG. 3, the estimating means 3 includes a measuring point setting means 11 for setting a plurality of measuring points on the three-dimensional shape represented by the three-dimensional shape data S1 'after the alignment,
Cosine calculating means 12 for calculating a cosine between a direction vector from a virtual light source set in arbitrary plural directions to a plurality of measurement points and a normal vector at the plurality of measurement points; a calculated cosine; Plotting means 13 for plotting the relationship between the data values of the two-dimensional image data S0 corresponding to the points for each direction of a plurality of virtual light sources, and provisional means for estimating the direction of the virtual light source whose plot is substantially straight as a temporary light source direction. Light source direction estimating means 14; cosine correcting means 15 for correcting the cosine of the plot so that the plot for which the temporary light source direction is determined is located on a straight line; and a cosine correction means for calculating the direction vector and the normal vector at the measurement point. Measuring point correcting means 16 for correcting the position of the measuring point to obtain a corrected measuring point so that the cosine approaches the corrected cosine; and calculating the cosine as a new measuring point using the corrected measuring point as a new measuring point. , Plot, estimation of the temporary light source direction,
The measuring point setting means 11, the cosine calculating means 12, the plotting means 13, the provisional light source direction estimating means 14, the cosine correcting means 15, and the measuring point correcting means 16 are controlled so as to repeat the correction of the cosine and the acquisition of the corrected measurement points a predetermined number of times. Control means 17
And a temporary light source direction obtained as a result of the repetitive processing is estimated as the direction of the light source, and a shape determined by the position of the corrected measurement point obtained as a result of the repeated processing is estimated as a three-dimensional shape of the face. .

The estimating means 3 estimates the direction of the light source and the three-dimensional shape of the face as follows. First, the estimation of the light source direction will be described. 4 and 5 are diagrams for explaining the estimation of the light source direction. In FIG. 4, for simplicity, the three-dimensional shape data S
The three-dimensional shape represented by 1 'will be described two-dimensionally with reference to a cross-sectional view taken on a plane perpendicular to the y-axis. First, the measurement point setting means 11 sets a plurality of measurement points on the three-dimensional shape represented by the three-dimensional shape data S1 '. In addition,
In FIGS. 4 and 5, two measurement points P1 and P2 are shown for simplicity.
However, actually, more measurement points are set. In addition, this measurement point is the three-dimensional shape data S
It is set to a portion corresponding to the skin color portion of the face of the person on the three-dimensional shape represented by 1 '. Then, as shown in FIG. 4, five virtual light sources (referred to as virtual light sources) L1 to L around the y-axis from −90 ° to + 90 ° at 45 ° intervals.
Assuming that 5 exists, in the cosine calculation means 12,
As shown in FIG. 5, for the plurality of measurement points P1 and P2, direction vectors LiP1 and LiP2 (i = 1 to 5) from the light sources to the measurement points P1 and P2 for each of the light sources L1 to L5.
And normal vectors n1, n2 at measurement points P1, P2
Cosine θ1 and cos θ2 are calculated. here,
Since the coordinate positions of the virtual light sources L1 to L5 and the measurement points P1 and P2 are known in advance, the direction vectors LiP1 and Li2
P2 can be calculated from the virtual light sources L1 to L5 and the coordinate values of the measurement points P1 and P2. Since the three-dimensional shape data S1 'is polygon data, the measurement point P
The plane in which 1, P2 exists can be easily obtained, and the normal vectors n1, n2 can be obtained as (a, b, c) from the equation ax + bx + cx + d = 0 that defines this plane. Therefore, the direction vectors LiP1, LiP
Cosine cos θ from the inner product of 2 and the normal vectors n1 and n2
1, cos θ2 can be calculated.

In this embodiment, since the measuring point is set at the position where the light is diffusely reflected, the intensity value of the reflected light at the measuring point, the intensity of the light, ie, the light emitted from the light source, the surface reflectance, and the cutoff And the above cosine, there is a relationship shown in the following equations (3) and (4).

I (P1) = k1 · s1 · cos θ1 · Lc + Ie (3) I (P2) = k2 · s2 · cosθ2 · Lc + Ie (4) where I (P1) and I (P2): measurement points P1 and P2 K1, k2: surface reflectance s2, s2: block value (0 or 1) Lc: illumination intensity Ie: ambient light intensity Blocking is, for example, in the direction of the light source in the face of a person. This refers to the part of the nose that cannot be reached due to the shadow of the nose. As described above, the value of the cutoff s is 0 in a portion where light does not reach, and the value of the cutoff s is 1 in other portions because light reaches and is reflected. Therefore, since the intensity value of the reflected light is 0 at the measurement point where s = 0, it is not used in the calculation of the light source direction estimation in the present embodiment. In this case, since there may be a shortage of measurement points used for calculation, it is preferable to set a new measurement point where s = 1 when there is a measurement point where s = 0.

The intensity values I (P1) and I (P2) of the reflected light are the data values of the two-dimensional image data S0 at the measurement points P1 and P2. In the present embodiment, since the two-dimensional image data S0 and the three-dimensional shape data S1 are aligned by the alignment means 1, for example, the above equation (1) is used.
According to (2), coordinate values corresponding to the measurement points P1 and P2 on the two-dimensional image data S0 are obtained from the coordinate values of the measurement points P1 and P2 in the three-dimensional shape data S1 'after the alignment. Then, the measurement point P in the two-dimensional image data S0
The data values at the coordinate positions corresponding to P1 and P2 are
1, P2, the intensity values of the reflected light I (P1), I (P
2).

Since the intensity of the reflected light becomes negative when the cosines cos θ1 and cos θ2 become negative, only the measurement points where cos θ1 and cos θ2> 0 are used in the present embodiment. In this case, since there may be a shortage of measurement points used for calculation, cos θ
When there is a measurement point satisfying 1, cos θ2 ≦ 0, c
It is preferable to set a new measurement point where osθ1 and cosθ2> 0.

In the present embodiment, since the surface reflectance at the point of diffuse reflection in the person's face is substantially constant,
In the above equations (3) and (4), k1 = k2. Further, the values of the cutoffs s1 and s2 are both 1. From the above equations (3) and (4), when the position of the virtual light source is correct, that is, when the position of the virtual light source coincides with the light source position at the time of shooting, the reflected light on the face of the face is expressed by the following equation (5). Is characterized in that the ratio of the intensity values and the ratio of the cosine coincide.

Lc · k · s = (I (P1) −Ie) / cosθ1 = (I (P2) −Ie) / cosθ2 (5) where k: k1, k2 s: s1, s2 Lc · k · s Therefore, the plotting unit 13 plots the relationship between the intensity values of the reflected light and the cosines cos θ1 and cos θ2 at the plurality of measurement points P1 and P2 for each of the virtual light sources L1 to L5 as shown in FIG. In FIG. 6, the intensity value of the reflected light is normalized so that the maximum value is 1.0.

As shown in FIG. 6, when the positions of the virtual light sources L1 to L5 coincide with the positions of the light sources at the time of shooting, the relationship between the intensity value of the reflected light and the cosine is obtained from the relationship shown in the above equation (5). The plot will be located on a substantially straight line. Therefore, the position of the virtual light source where the plot is located on a substantially straight line can be regarded as the position of the light source at the time of shooting. In FIG. 6, the light source from the direction of −45 degrees, that is, the virtual light source L2
Since the relationship between the intensity value of the reflected light and the cosine when the light source is used is located on a substantially straight line, the position of the virtual light source L2 is determined by the temporary light source direction estimating means 14 to be the position of the temporary light source at the time of shooting. Is required. Therefore, at the time of photographing, it can be considered that light is emitted from the direction of -45 degrees in FIG. 4 when viewed from the face of the person. The value at the intersection of this straight line and the vertical axis, that is, the intensity axis, is the ambient light Ie
Of strength. Further, the slope of this straight line is Lc · k · s in the above equation (5), and the magnitude of the slope can be seen from FIG. Therefore, assuming that the inclination is a, Lc = a / k ·
As s, the illumination intensity Lc can be obtained.

Here, whether or not the plot is located on a straight line is determined by, for example, an evaluation function using the least squares method. That is, if the equation of a straight line is y = ax + b,

(Equation 1) Here, N: the number of plots xn: cosine value yn: intensity value of reflected light a, b values can be obtained. Then, from the values of a and b obtained in this way, the following equation (7) is obtained.
Is calculated.

[0053]

(Equation 2) Then, the direction of the virtual light source in which the value of the evaluation function cf becomes the smallest value may be set as the temporary light source direction at the time of shooting.

Next, the estimation of the three-dimensional shape of the face will be described. FIG. 7 is a diagram showing a plot used for determining a temporary light source direction and a straight line C obtained from the plot. As shown in FIG. 7, the three-dimensional shape data S1 ′ after the alignment represents an approximate three-dimensional shape of the face.
Since the shape is different from the shape represented by 1 ', the plot does not completely lie on the straight line C but exists apart from the straight line C. Here, in the present embodiment, since the three-dimensional shape data S1 'is aligned in the x and y directions, the gap occurs because the value in the z-axis direction deviates from the actual value. Can be thought of. Therefore, in the present embodiment, the distance of each plot from the straight line C is corrected to correct the displacement of the three-dimensional shape data S1 'in the z-axis direction, and the three-dimensional shape of the face is estimated.

First, it is assumed that the deviation of the plot of the temporary light source direction obtained as described above from the straight line C is caused by the deviation of the inclination of the normal vector at the measurement point corresponding to the plot. Can be. For example, in the case of the plot corresponding to the measurement point P3 in FIG. 7, since the plot is shifted to the left from the straight line C, the slope of the normal vector at the measurement point P3 is shifted and the value of the cosine is larger than the actual value of the cosine. It can be considered that it is getting smaller.
Here, the actual cosine value is a cosine between a direction vector from the temporary light source toward a measurement point set on the three-dimensional shape of the actual face and a normal vector at this measurement point. . Therefore, the cosine correction means 15 corrects the z-value of the three-dimensional shape data S1 'at the measurement point P3 by correcting the cosine value at the measurement point P3 so as to increase the cosine value and correcting the inclination of the normal vector. To move the plot in a direction approaching the straight line C. That is, the data value of the two-dimensional image data S0 corresponding to the measurement point P3 is g
Assuming that (P3), from g (P3) = ax + b, x = cos θc = (g (P3) −b) / a (8) where cos θc is a cosine value when the measurement point P3 is located on the straight line C. Therefore, the value of θ at the measurement point P3, that is, the angle between the direction vector and the normal vector is corrected so as to approach the value of θc. When the value of the cosine is Y, the value of θ can be obtained by θ = cos ⁻¹ Y (9). When the plot is shifted to the right from the straight line C as in the plot corresponding to the measurement point P4 in FIG. 7, the slope of the normal vector at the measurement point P4 is reduced so that the value of the cosine at the measurement point P4 becomes smaller. Is corrected, the z value of the three-dimensional shape data is corrected, and the plot is moved in a direction approaching the straight line C.

FIG. 8 is a diagram for explaining the correction of the value of θ. In FIG. 8, θold is the angle between the direction vector PL and the normal vector nold at the measurement point P3 before correction, θnew is the angle between the direction vector PL and the normal vector nnew at the measurement point P3 after correction, and θc is This is the angle between the direction vector PL and the normal vector when the measurement point P3 is located on the straight line C. First, the vector nne
w is calculated by the following equation (10).

Nnew = nold + α (θold−θc) · (PL−nold) (10) where α: coefficient (0 <α <1) Here, since the value of α has a value smaller than 1, the normal vector nnew moves on the vector (PL-nold) from the end point of the normal vector nold toward the light source L according to the value of α. Here, the reason why the value of α is not set to 1 is that the straight line C represents a temporary light source direction to the last, and therefore is strictly different from the actual light source direction. If the measurement points are corrected at once based on the light source direction, the three-dimensional shape of the face cannot be obtained accurately.

When the normal vector nnew is calculated in this manner, the z value of the three-dimensional shape data S1 'at the measurement point P3 is corrected by the measurement point correction means 16 based on the normal vector nnew. . Specifically, the measurement point P
3 exists in a polygon having the vertices of points A, B, and C as shown in FIG. 9, the z value of the vertices of points A, B, and C is corrected and z of the three-dimensional shape data S1 'is changed. Modify the value. Here, assuming that the normal vector nnew is (a ′, b ′, c ′) and the coordinates of the measurement point P3 are (x1, y1, z1), the equation of a plane passing through the measurement point P3 and having the normal vector nnew is Is determined.

A′x + by ′ + c′z + d ′ = 0 (11) d ′ = − (a′x1 + b′y1 + c′z1) (12) At points A, B, and C, the values of x and y have been changed. However, since only the value of z is changed, if the changed value of z is znew, it can be obtained by the following equation: znew = − (a′x + by ′ + d) / c ′ (13) Thus, the coordinates of the corrected points A, B, and C can be expressed as (x, y, znew).

The above processing is performed for all the measurement points corresponding to the plots for which the provisional light source directions have been determined, and the coordinate values of the new measurement points are obtained, which is used as corrected three-dimensional data Sc.
Note that a measurement point on the corrected three-dimensional data Sc is a corrected measurement point.

Then, the corrected measurement point is set as a new measurement point,
Set a new virtual light source and plot the relationship between the cosine cosθnew formed by the direction vector from the new virtual light source to the corrected measurement point and the normal vector nnew of the corrected measurement point, and the intensity value of the reflected light at the corrected measurement point Then, a new straight line is determined by the above equations (7) and (8) to determine a new direction of the virtual light source, and the evaluation function cf is calculated. At this time, the new virtual light source may be set near the temporary light source. Then, it is determined whether or not the value of the evaluation function cf is larger than a predetermined value. If the value exceeds the predetermined value, the z value of the measurement point is corrected as described above to obtain a new corrected measurement point. Is calculated, and the calculation of the temporary light source direction and the value of the evaluation function cf for this new corrected measurement point is repeated until the value of the evaluation function cf becomes smaller than the predetermined value. Then, the direction of the virtual light source when the evaluation function cf becomes equal to or less than a predetermined value is estimated as the direction of the light source, and the three-dimensional shape defined by the corrected measurement point at this time is estimated as the three-dimensional shape of the face. . Here, the estimated three-dimensional shape of the face is the three-dimensional shape of the skin color portion of the face. The above processing is performed by the control unit 17 by the measurement point setting unit 11,
This is performed by controlling the cosine calculating means 12, the plotting means 13, the provisional light source direction estimating means 14, the cosine correcting means 15, and the measuring point correcting means 16.

Next, the operation of this embodiment will be described. FIG. 10 is a flowchart showing the operation of the present embodiment. First, the positioning unit 1 performs positioning between the two-dimensional image data S0 and the three-dimensional shape data S1 (step S1). Then, the aligned combined data S2 is input to the estimating means 3. The estimating means 3 sets a plurality of measurement points on the three-dimensional shape represented by the three-dimensional shape data S1 'after the alignment (step S2), and selects an arbitrary virtual light source from the virtual light sources L1 to L5. (Step S3). In the present embodiment, it is assumed that the virtual light source L1 is selected in order. Then, at each measurement point, a cosine between the normal vector and the direction vector between the measurement point and the virtual light source is calculated (step S4). 6 is plotted (step S5). Then, it is determined whether or not the plot is located on a straight line by determining whether or not the evaluation function cf has become equal to or smaller than a predetermined value in the relationship between the intensity value of the reflected light and the cosine (step). S6), Step S
If the result in No. 6 is negative, the process returns to Step S3, and a different virtual light source is selected, and the processing from Step S4 to Step S6 is repeated. When step S6 is affirmed, the direction of this virtual light source is determined as a temporary light source direction at the time of shooting (step S7).

When the temporary light source direction is determined as described above,
It is determined whether or not the evaluation function cf when the temporary light source direction straight line is determined is equal to or smaller than a predetermined value (step S).
8). If step S8 is negative, the position of the measurement point is corrected as described above (step S9), and a new virtual light source is set near the temporary light source direction (step S9).
10), the process returns to step S3, and thereafter, the processes from step S3 to step S8 are repeated. And step S
If affirmative, the tentative light source direction is determined as the direction of the light source at the time of shooting, and the three-dimensional shape of the face is determined from the corrected coordinate position of the measurement point (step S11), and the process ends.

As described above, in the present embodiment, when the position of the virtual light source is correct, that is, when the position of the virtual light source coincides with the position of the light source at the time of shooting, the intensity of the reflected light on the surface of the face is calculated as shown in Expression (5). Since the direction of the light source at the time of shooting is estimated based on the feature that the ratio of the values and the ratio of the cosine coincide with each other, only the condition that the value of the cutoff s is not 0 is satisfied for the setting of the measurement point. The direction of the light source can be estimated by a good and relatively simple calculation. Also,
Even if the three-dimensional shape of the object is unknown, the three-dimensional shape of the object can be easily estimated based on the estimated direction of the light source as compared with the above-described method in the pond. Thus, when combining the real image and the virtual image such as CG, the virtual image is corrected based on the direction of the light source estimated in the real image so as to be illuminated from an appropriate light source, and then combined. And a synthesized image without discomfort can be obtained. Further, in an encoding method such as MPEG-4 in which images are transferred for each object and synthesized at a stage of viewing a moving image, when an object includes a real image and a virtual image, the illumination direction of the real image is changed. Accordingly, the lighting direction of the virtual image can be corrected and synthesized, thereby obtaining a synthesized moving image without a sense of discomfort. Furthermore, since an object included in the real image can be represented as an appropriate three-dimensional image, a three-dimensional image without a sense of incongruity can be generated from the real image. Furthermore, in the case of an image obtained by performing shooting in sunlight, the backlight correction and the density correction can be easily performed on the image by estimating the direction of the sunlight in the image.

In the above embodiment, the virtual light sources L1 to L5 are arranged two-dimensionally for explanation.
Actually, a plurality of virtual light sources are arranged three-dimensionally with respect to the three-dimensional shape data S1, and the relationship between the intensity of the reflected light and the above-mentioned cosine is plotted at each measurement point, and the light source direction and the object This is for estimating a three-dimensional shape.

In the above embodiment, the expression (6)
As shown in (7) and (7), the temporary light source direction is estimated using the evaluation function cf by the least squares method. However, the present invention is not limited to this, and the plot shown in FIG. The operator may observe this and determine the direction of the virtual light source where the plot is located on a straight line as the direction of the temporary light source.

Further, the camera used when acquiring the two-dimensional image data S0 has a non-linear relationship between the input light and the output image data. For this reason, a gray chart is photographed by a camera to be used in advance, and a relationship between an input and an output to the camera is obtained.
It is preferable to correct the input / output characteristics of the camera so that this relationship becomes a straight line.

The brightness of the object surface in the real environment is
Specular reflection light is also included in addition to the above-described diffuse reflection light and environmental light, and can be modeled by a combination of these lights. That is, the luminance of the object surface can be represented by the following equation (14). Here, the mutual reflection is smaller than the diffuse reflection, the specular reflection, and the ambient light, and can be ignored.

I = Id + Is + Ie (14) where I: luminance of the object surface Id: diffuse reflection light Is: specular reflection light Ie: environmental light Here, it is assumed that there is one light source as in the above embodiment. The case where the specular reflection position is detected from the face image using the three-dimensional shape data S1 'will be described.

From the above equation (14), any 2 on the face surface
The difference ΔI between the intensity value at the point, that is, the data value, depends on the diffuse reflection light Id and the specular reflection light Is. First, assuming that only the diffuse reflection light Id exists, the intensity difference ΔI at the two measurement points P1 and P2 as shown in FIG.
It can be formulated as shown in 5).

ΔI = k · s1 · cosθ1 · Lc−k · s2 · cosθ2 · Lc = k (s1 · cos1-s2 · cos2) Lc (15) Since it can be assumed that the position and intensity of the light source do not change,
The intensity difference ΔI is caused by a difference in cosine between two points, that is, one of a direction difference of a normal vector and a cutoff. Here, assuming that a reference measurement point (for example, P1) exists at a position where diffuse reflection occurs, the intensity difference ΔI
Is caused by the interruption, the other measurement point P2 is shaded, so that the intensity value greatly changes even if the positions of the two points are slightly different. On the other hand, the intensity difference ΔI
Is caused by the difference in the cosine, the intensity difference ΔI is approximately proportional to the value of the cosine when the other measurement point P2 is also at the position where the diffuse reflection occurs, but the other measurement point P2 is at the position where the other surface is specularly reflected. Even if the positions of the two points are slightly different, the intensity value changes so as to greatly increase. As described above, one measurement point P1 is set at the position where diffuse reflection occurs on the three-dimensional shape data S1 'after alignment, and the data value of this measurement point P1 and an arbitrary measurement point P2 (diffuse reflection) adjacent thereto are set. (It is not limited to the position at which the data value changes.), And the point at which the data value sharply increases despite the slight difference between the two positions can be determined as the position for specular reflection.

On the other hand, the equation for specular reflection can be formulated as shown in the following equation (16).

Is = k · s · cos ⁿ α · Lc (16) That is, ^assuming that the light source is located at a position as shown in FIG. When the light is in the specular reflection direction at the point P, the point P is specularly reflected, and the intensity of the specularly reflected light becomes smaller as the position of the camera moves to the position V shifted from the R by an angle α.

Therefore, as shown in FIG. 12, it is determined whether or not the plurality of measurement points set by the measurement point setting means 11 have measurement points that reflect specularly. And selecting means 20 for selecting
Before estimating the direction of the temporary light source (before step S3 in the flowchart shown in FIG. 10), it is determined whether or not a plurality of measurement points have specularly reflected measurement points. In this case, as shown in FIG. 11, the normal vector at the position (P) divides the angle between the direction vector PL and the direction vector PR into two equal parts. The existing direction can be estimated to some extent. Therefore, even if the virtual light source is not set in every direction as in the above embodiment, if the virtual light source is set only in the direction in which the light source is likely to exist, the temporary light source direction can be estimated, so that the amount of calculation is reduced. Processing can be performed at high speed.

In the above embodiment, the shape of a person's face is estimated. However, the present invention is not limited to this, and three-dimensional shapes of various objects can be estimated.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram illustrating a configuration of an estimation device according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining alignment between two-dimensional image data and three-dimensional shape data;

FIG. 3 is a schematic block diagram illustrating a configuration of an estimation unit.

FIG. 4 is a diagram for explaining a direction of a virtual light source;

FIG. 5 is a diagram for explaining calculation of a cosine at a measurement point;

FIG. 6 is a graph showing a relationship between a cosine and an intensity value of reflected light.

FIG. 7 is a diagram showing a plot used to determine a temporary light source direction;

FIG. 8 is a diagram for explaining correction of θ.

FIG. 9 is a diagram for explaining correction of a z value.

FIG. 10 is a flowchart showing the operation of the embodiment.

FIG. 11 is a diagram for explaining specular reflection;

FIG. 12 is a schematic block diagram showing the configuration of an embodiment provided with a selection unit.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Alignment means 2 Memory 3 Estimation means 11 Measurement point setting means 12 Cosine calculation means 13 Plotting means 14 Temporary light source direction estimation means 15 Cosine correction means 16 Measurement point correction means 17 Control means 20 Selection means

Claims (24)

[Claims] An object having a three-dimensional shape including a portion having a substantially constant surface reflectivity is photographed under a substantially single light source, based on an image obtained by photographing the object, the direction of the light source and the object. A data value of object image data representing an image of the object on the image, and a three-dimensional data representing the object image data and a general three-dimensional shape of the object. Based on the positional relationship between the approximate object represented by the three-dimensional shape data after the alignment with the shape data and the virtual light source set in any of a plurality of directions, the direction of the light source and the three-dimensional shape of the object are determined. An estimation method characterized by estimating a three-dimensional shape. 2. A method according to claim 1, wherein a plurality of measurement points are set on a portion of the approximate object corresponding to a portion where the surface reflectance of the object is substantially constant, and Calculating a cosine between a direction vector toward the measurement point and a normal vector at the plurality of measurement points; and calculating a relationship between the cosine and a data value of the object image data corresponding to the plurality of measurement points. Plotting for each direction of the virtual light source, estimating the direction of the virtual light source in which the plot is substantially a straight line as a temporary light source direction, such that the plot obtained for the temporary light source direction is located on the straight line, Correcting the cosine in the plot, correcting the position of the measurement point so that the cosine between the direction vector and the normal vector at the measurement point approaches the corrected cosine, Calculating the cosine with the corrected measurement point as the new measurement point, repeating the plotting, estimating the temporary light source direction, correcting the cosine, and acquiring the corrected measurement point a predetermined number of times; And estimating the temporary light source direction obtained as a result of the processing as the direction of the light source, and estimating the shape determined by the position of the corrected measurement point obtained as a result of the iterative processing as the three-dimensional shape of the object. The estimation method according to claim 1. Determining whether or not there is a specular reflection measurement point for specular reflection at the plurality of measurement points based on a data value of the object image data at the plurality of measurement points; Is present, the direction of the virtual light source, which is likely to be the direction of the light source, is selected based on the specular reflection measurement point, and the cosine is calculated by the virtual light source in the selected direction. 3. The estimation method according to claim 2, wherein: 4. The method according to claim 1, wherein the estimation of the direction of the temporary light source is performed by determining whether or not the plot is substantially a straight line, based on an evaluation function using a least squares method. 4. The estimation method according to 2 or 3. 5. The estimation method according to claim 4, wherein the predetermined number is a number of times until the evaluation function becomes equal to or less than a predetermined value. 6. The light source according to claim 2, wherein the intensity of the light source is estimated based on a slope of a straight line set by a plot used for estimating the direction of the light source. Estimation method. 7. The estimation method according to claim 1, wherein the object is a human face. 8. The three-dimensional shape data of the object
The estimation method according to any one of claims 1 to 7, wherein the data is polygon data representing a dimensional shape. 9. The direction of the light source and the object based on an image obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectivity under a substantially single light source. A data value of object image data representing an image of the object on the image, and a three-dimensional data representing the object image data and a general three-dimensional shape of the object. Based on the positional relationship between the approximate object represented by the three-dimensional shape data after the alignment with the shape data and the virtual light source set in any of a plurality of directions, the direction of the light source and the three-dimensional shape of the object are determined. An estimating device comprising means for estimating a dimensional shape. 10. A means for estimating the direction of the light source and the three-dimensional shape of the object, comprising: a plurality of measurement points on a portion of the approximate object corresponding to the portion where the surface reflectance of the object is substantially constant. Measurement point setting means for setting a direction vector from the virtual light source set in any of the plurality of directions to the plurality of measurement points, and a cosine calculation means for calculating a cosine between a normal vector at the plurality of measurement points. Plotting means for plotting the relationship between the cosine and the data values of the object image data corresponding to the plurality of measurement points for each direction of the plurality of virtual light sources; Temporary light source direction estimating means for estimating a direction as a temporary light source direction; and a cosine for correcting the cosine in the plot so that a plot for obtaining the temporary light source direction is located on the straight line. Corrective means, measuring point correcting means for obtaining a corrected measuring point by correcting the position of the measuring point so that the cosine between the direction vector and the normal vector at the measuring point approaches the corrected cosine. Calculating the cosine, setting the plot, estimating the temporary light source direction, correcting the cosine, and obtaining the corrected measurement point a predetermined number of times, using the corrected measurement point as the new measurement point. Setting means, the cosine calculating means, the plotting means, the temporary light source direction estimating means, control means for controlling the cosine correcting means and the measuring point correcting means, and the temporary light source direction obtained as a result of the repetitive processing. The shape determined by the position of the corrected measurement point obtained as a result of the iterative process is estimated as the direction of the light source.
The estimating apparatus according to claim 9, further comprising estimating means for estimating the dimensional shape. 11. A method for determining whether or not there is a specular reflection measurement point for specular reflection at said plurality of measurement points based on a data value of said object image data at said plurality of measurement points. If there is, based on the specular reflection measurement point, comprising a selection means for selecting the direction of the virtual light source that is likely to be the direction of the light source, the cosine calculation means, in the selected direction 11. The estimating apparatus according to claim 10, wherein said cosine is calculated by a virtual light source. 12. The temporary light source direction estimating means determines whether or not the plot is a substantially straight line, based on an evaluation function using a least squares method. The estimating device according to claim 10, further comprising a determination unit configured to perform the determination according to: 13. The estimating apparatus according to claim 12, wherein the predetermined number of times is a number of times until the evaluation function becomes equal to or less than a predetermined value. 14. The apparatus according to claim 1, wherein said estimating means is means for estimating also the intensity of the light source based on a gradient of a straight line set by a plot used for estimating the direction of the light source. 14. The estimation device according to any one of 10 to 13. 15. The estimating apparatus according to claim 9, wherein the object is a human face. 16. The estimating apparatus according to claim 9, wherein the three-dimensional shape data is polygon data representing a three-dimensional shape of the object. 17. The direction of the light source and the object based on an image obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectivity under a substantially single light source. A computer-readable recording medium storing a program for causing a computer to execute an estimation method for estimating a three-dimensional shape of the object, wherein the program includes data of object image data representing an image of the object on the image. Values, and an approximate object represented by the three-dimensional shape data after the object image data is aligned with three-dimensional shape data representing a general three-dimensional shape of the object, and in an arbitrary plurality of directions. A computer-readable method for estimating a direction of the light source and a three-dimensional shape of the object based on a set positional relationship with the virtual light source. Ri capable of recording medium. 18. The direction of the light source and 3 of the object
The procedure of estimating the dimensional shape is a procedure of setting a plurality of measurement points on a portion corresponding to a portion where the surface reflectance of the object on the approximate object is substantially constant, and setting the measurement points in the arbitrary plural directions. A direction vector from the virtual light source toward the plurality of measurement points, and a procedure for calculating a cosine formed by a normal vector at the plurality of measurement points; and a method of calculating the cosine and the object image data corresponding to the plurality of measurement points. A procedure of plotting a relationship with a data value for each direction of the plurality of virtual light sources, a procedure of estimating a direction of the virtual light source in which the plot is substantially a straight line as a temporary light source direction, and obtaining the temporary light source direction. Correcting the cosine of the plot so that the plot is located on the straight line; and the cosine of the direction vector and the normal vector at the measurement point is corrected. Correcting the position of the measurement point so as to approach the cosine to obtain a corrected measurement point; calculating the cosine with the corrected measurement point as the new measurement point; the plotting; estimating the temporary light source direction A procedure of repeating the correction of the cosine and the acquisition of the corrected measurement point a predetermined number of times; and estimating the temporary light source direction obtained as a result of the repetition processing as the direction of the light source, and a corrected measurement point obtained as a result of the repetition processing. The shape determined by the position of
18. The computer-readable recording medium according to claim 17, further comprising a step of estimating a dimensional shape. 19. A step of judging whether or not there is a specular reflection measurement point for specular reflection at the plurality of measurement points based on a data value of the object image data at the plurality of measurement points; If there is a measurement point, the method further comprises: selecting a direction of the virtual light source having a high probability of being the direction of the light source, based on the specular reflection measurement point, and calculating the cosine. 19. The computer-readable recording medium according to claim 18, wherein the cosine is calculated using a virtual light source in the selected direction. 20. The procedure for estimating the direction of the temporary light source includes a procedure for determining whether or not the plot is substantially a straight line based on an evaluation function using a least squares method. The computer-readable recording medium according to claim 18. 21. The computer-readable recording medium according to claim 20, wherein said predetermined number of times is a number of times until said evaluation function becomes equal to or less than a predetermined value. 22. The method according to claim 18, further comprising a step of estimating the intensity of the light source based on a slope of a straight line set by a plot used for estimating the direction of the light source. The computer-readable recording medium according to claim 1. 23. The computer-readable recording medium according to claim 17, wherein the object is a human face. 24. The computer-readable recording medium according to claim 17, wherein said three-dimensional shape data is polygon data representing a three-dimensional shape of said object.