JP4764963B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that more accurately expresses reflection characteristics of a subject.

  In recent years, in the field of computer graphics (CG), the pursuit of photorealistic technology that faithfully reproduces physical and optical phenomena in the real world has attracted attention. This technology can be applied to various fields such as television, movies and games. As an application method of this technique, there is a method of faithfully modeling an actual object by CG. For example, it may be possible to measure historically valuable objects precisely and store them semi-permanently as digital archives, or digitize exhibits stored in museums and museums and publish them on the WEB as virtual museums. ing.

  When a real object is represented by CG, the reflection characteristics of the object surface play an important role. A reflection model is a model in which this reflection characteristic is modeled based on physical and geometrical laws. Various reflection models have been developed depending on the application, from models that faithfully model physical phenomena to models that emphasize low calculation costs. The expressive power by the reflection model plays an important role for reproducing a real object by CG.

  Furthermore, since the texture of the object varies greatly depending on the parameter setting of the reflection model, it is necessary to set the parameter of the reflection model in order to represent the real object in CG. A method for estimating this parameter from a small number of actually captured images has been proposed (see Non-Patent Document 1 below).

Further, instead of estimating the reflection characteristic parameter from the image, another estimation method has been proposed. For example, a method has been proposed in which an image obtained by photographing a sample of a material is used as data representing direct reflection characteristics to faithfully represent the object surface (see Non-Patent Document 2 below).
"Estimation of light source condition and reflection characteristics from dense image sequence" CVIM, vol. 44 NO. SIG5 PP. 1-10, 2003 W.Matusik, H.Pfister. M.Brand, and L.Mcmillan, "A Data-Driven Reflectance Model" ACM TOG, Vol.22 No3, PP.759-769, 2003

  However, in the technique described in Non-Patent Document 1, fine unevenness on the surface of the real object is not taken into consideration. For this reason, it has been difficult to faithfully express the texture of the surface of the real object expressed in CG.

  Furthermore, in the above-described invention described in Non-Patent Document 2, it is necessary to prepare a sample according to the material of the subject and measure the parameter of the reflection characteristic. Therefore, there has been a problem that it takes cost and time to obtain parameters. Furthermore, there is a problem that it is not possible to deal with a subject made of a material for which a sample has not been acquired.

  Furthermore, other models capable of highly expressing the texture of the object surface have been proposed. However, these models have various problems such as too many parameters to be determined and high calculation cost.

  The present invention has been made in view of the above problems. One object of the present invention is to provide an image processing apparatus capable of faithfully expressing fine irregularities on the surface of a subject.

The image processing apparatus of the present invention includes a separating unit that separates an actual value of a specular reflection component from a real image obtained by photographing an object, an estimation unit that calculates an estimated value of the specular reflection component, and an actual value of the specular reflection component. A calculation unit that calculates a parameter of the specular reflection model based on a difference from the estimated value; and a generation unit that generates an image including a specular reflection component by the specular reflection model using the parameter , the generation The means is characterized in that bump mapping is performed based on a difference between the measured value and the estimated value .

  According to the present invention, since the model for estimating the specular reflection component has a plurality of roughness parameters, an image having a gloss point closer to an actual object can be obtained. Furthermore, since the diffuse reflection component also has a parameter that raises the power of the trigonometric function, a generation result close to a real image can be obtained.

  Further, according to the present invention, an image including a gloss point reflecting fine irregularities on the object surface is generated. Therefore, high-quality computer graphics can be obtained. Further, parameters of the specular reflection component model and the diffuse reflection component model are calculated from the inputted real image. Therefore, it is possible to perform image processing in which the actual surface shape of the subject is reflected more accurately.

<First embodiment>
In this embodiment, a basic configuration of the image processing apparatus 10 of this embodiment will be described with reference to FIG. The image processing apparatus 10 according to the present invention is composed of parts connected to each other by a bus 17. Then, image processing can be performed by the operation of each unit centering on the calculation unit 11.

  The calculation unit 11 is a semiconductor integrated circuit, and a circuit is formed so that calculation can be performed according to a predetermined program. In this embodiment, image processing is performed by the calculation unit 11 performing a predetermined calculation.

  The storage unit 12 is a storage medium such as a semiconductor memory or a hard disk. The storage unit 12 stores information such as a program for operating the calculation unit 11, three-dimensional shape data of the subject, temporarily stored image data, and various parameters.

  The input unit 13 is a device for inputting information to the image processing apparatus 10. For example, devices such as a keyboard and a mouse constitute the input unit 13.

  The external interface 14 is a USB (Universal Serial Bus), for example, and has a function of capturing image data from the camera 15. The captured image data is stored in the storage unit 12.

  The camera 15 is a digital camera, for example, and obtains image data digitized by an image pickup device such as a CCD.

  The display unit 16 is, for example, a liquid crystal display, and has a function of displaying image data captured by the camera 15 and image data generated by the calculation unit 11.

  The image processing apparatus 10 configured as described above captures image data with the camera 15, performs an operation with the calculation unit 11 based on the image data, and causes the display unit 16 to display the generated new image data. ing. Furthermore, the image processing apparatus 10 estimates a parameter related to the roughness of the object surface from the input captured image. Rendering is performed using the obtained parameters. These detailed processing methods will be described later.

<Second Embodiment>
In this embodiment, an outline of modeling of reflection characteristics will be described with reference to FIGS. In the present invention, it is assumed that the reflection of the object surface can be expressed by diffuse reflection and specular reflection. Here, the specular reflection component is a highlight that appears on the surface of the object, and is a region having a high luminance value caused by the reflection of the light source. Further, this component has a property of reflecting from the surface toward a specific direction. The diffuse reflection component is a reflection in which light is scattered in all directions due to very fine irregularities on the surface and internal reflection, and represents a color component on the surface. Further, it is assumed that the object surface has fine irregularities.

  First, the diffuse reflection model will be described with reference to FIG. FIG. 2A is an external view of gypsum, and FIG. 2B is a graph showing the distribution of diffuse reflection components.

  In the present invention, a Lambert model as shown in the following equation 1 is considered as a diffuse reflection model. The Lambert model has been used for many years in the fields of computer graphics and computer vision as a model describing diffuse reflection on the surface of an object. However, it has been found that an object such as gypsum, which is considered to be completely diffuse reflection, is not actually a Lambert model.

上記の式で、Ｉｄは拡散反射成分であり、Ｋｄは物体表面に於ける拡散反射色を示し、θは物体表面に於ける法線と光源方向との成す角を示している。

In the above equation, Id is a diffuse reflection component, Kd indicates a diffuse reflection color on the object surface, and θ indicates an angle formed between a normal line on the object surface and the light source direction.

  The graph in FIG. 2B is a plot of the luminance values of the scan lines indicated by the horizontal lines in FIG. In general, gypsum is considered to be a material that matches the Lambert model, but it can be confirmed that there is a difference between the diffuse reflection actually taken and the Lambert model. Actual diffuse reflection attenuates more slowly than the Lambert model in a region where the angle between the light source direction and the surface normal is small. In the region where the angle formed by the light source direction and the surface normal is large, the actual diffuse reflection is rapidly attenuated. That is, in the above formula 1, it is difficult to faithfully express diffuse reflection.

  Therefore, in the present invention, as shown in the following Equation 2, a model has been developed in which the attenuation ratio is changed by adding a parameter that raises the term of cos (trigonometric function) in the Lambert model. In the following description, this model may be referred to as an extended Lambert model.

上記数２にてｎの範囲は0＜ｎ＜＝１である。そして、ｎ＝１のときは、数１に示したモデルが数２に示したものと同一となる。この数２に示すモデルは、多くの物体における拡散反射成分の表現に適用することができる。更にこのモデルは、他の拡散反射モデルと比較しても、計算コストが安い。

In the above formula 2, the range of n is 0 <n <= 1. When n = 1, the model shown in Equation 1 is the same as that shown in Equation 2. The model shown in Equation 2 can be applied to the expression of diffuse reflection components in many objects. Furthermore, this model has a low calculation cost even when compared with other diffuse reflection models.

  The specular reflection model will be described with reference to FIGS.

  The Torrance-Sparrow model will be described with reference to FIG. 3A shows the direction of each vector such as the light source direction, FIG. 3B is a real image of a spherical object, and FIG. 3C is a graph showing the distribution of specular reflection components.

  The Torrance-Sparrow model is often used because it can express reflections on the surface of real objects such as metal. However, since there are many parameters in this model, it is difficult to estimate all the parameters from only the actual image. As shown in FIG. 3A, when the intermediate point vector indicating the intermediate between the light source direction vector L and the line-of-sight direction vector V is H, the specular reflection component Is in the Torrance-Sparrow model is expressed by the following equation (3).

数３に於いて、Ｌは光源色、Ｋｓは鏡面反射色、Ｄはマイクロファセット分布関数、Ｇは幾何滅衰係数、Ｆはフレネル係数を表している。幾何減衰係数に関しては、面の法線と光源方向があまり離れていない部分を撮影するものとし、Ｇ＝１とする。フレネル係数に関しては、Ｆは一定であるとして物体の持つ鏡面反射色Ｋｓに含めるものとする。また、撮影時の光源に対してカメラのホワイトバランスが適切に設定されているものとし、光源色LはＲ：Ｇ：Ｂ＝１：１：１とする。マイクロファセット分布関数Ｄは、物体表面における鏡面反射光の広がりを表す項と考えることができる。鏡のように物体が完全に滑らかな表面を持つ場合、単一の点光源によって照射された鏡面反射成分は1点に収まるが、粗い表面を持つ物体の場合には鏡面反射成分はある程度の広がりを持つ。Torrance-Sparrowモデルでは、この分布関数をガウス分布であると仮定している。ガウス分布を用いるとマイクロファセット分布関数は数４で表される

In Equation 3, L is a light source color, Ks is a specular reflection color, D is a microfacet distribution function, G is a geometric extinction coefficient, and F is a Fresnel coefficient. As for the geometric attenuation coefficient, it is assumed that a portion where the normal of the surface and the light source direction are not so far apart is photographed, and G = 1. Regarding the Fresnel coefficient, F is assumed to be constant and is included in the specular reflection color Ks of the object. In addition, it is assumed that the white balance of the camera is appropriately set with respect to the light source at the time of shooting, and the light source color L is R: G: B = 1: 1: 1. The micro facet distribution function D can be considered as a term representing the spread of specular reflection light on the object surface. If the object has a perfectly smooth surface, such as a mirror, the specular reflection component irradiated by a single point light source can be reduced to one point, but in the case of an object with a rough surface, the specular reflection component will spread to some extent. have. In the Torrance-Sparrow model, this distribution function is assumed to be Gaussian. Using a Gaussian distribution, the microfacet distribution function is expressed as

ここで、αは物体表面での法線ベクトルＮと中間ベクトルＨとのなす角、σ^２はマイクロファセットの傾きを表している。σ^２が小さいと物体の法線ベクトルＮとマイクロファセットの法線との変化が少なく、鋭い鏡面反射となる。逆に、σ^２が大きいと、法線からの変化が大きくなり、粗い物体表面によって広がりを持った鏡面反射光と成る。

Here, α represents an angle formed by the normal vector N and the intermediate vector H on the object surface, and σ ² represents the inclination of the microfacet. When σ ² is small, there is little change between the normal vector N of the object and the normal of the micro facet, and sharp specular reflection occurs. On the other hand, when σ ² is large, the change from the normal line becomes large, resulting in specular reflected light having a spread by a rough object surface.

  When the reflection on the real object is observed, the distribution of the specular reflection component on the surface of the object painted with a metal color as shown in FIG. 3B is as shown in FIG. In FIG. 3C, the vertical axis represents the luminance value of the specular reflection component, and the horizontal axis represents the angle formed by the intermediate vector and the normal of the object surface. As can be seen from this graph, in the region where the angle formed by the intermediate vector and the normal of the object surface is large, the attenuation of the specular reflection component is gently distributed compared to a simple normal distribution function. Therefore, it is difficult to faithfully reproduce the reflection of an object having a special surface material such as metal by a reflection model using a simple normal distribution function.

  Therefore, in the present invention, by adding a parameter to the micro facet distribution function. Expand to a more expressive model. The distribution of the specular reflection component as shown in FIG. 3C is considered to be constituted by the sum of a plurality of normal distribution functions having different parameters, and the microfacet distribution function is changed as shown in Equation 5.

ここで、σ１、σ２は表面の粗さを表すパラメータである。より多くの粗さパラメータを用いることによって精密な表現が可能となる。しかしながら、多くのパラメータを設定した場合には、観測データからのパラメータ推定を安定に行うことが困難と考え、粗さパラメータは２パラメータのみとした。ここで、３つ以上のパラメータを用いることも可能であり、この場合は精度を向上させることができる。σ１＝σ２のとき、数５は数４と同等のものとして扱うことができる。以上のことから、本手法で用いる鏡面反射成分のモデル式は数６のように定義される。

Here, σ1 and σ2 are parameters representing the surface roughness. By using more roughness parameters, precise representation is possible. However, when many parameters are set, it is difficult to stably estimate the parameters from the observation data, and the roughness parameter is only two parameters. Here, it is also possible to use three or more parameters. In this case, the accuracy can be improved. When σ1 = σ2, Equation 5 can be treated as equivalent to Equation 4. From the above, the model expression of the specular reflection component used in the present method is defined as in Equation 6.

次に、図４を参照して、物体表面における微細な凹凸のモデル化を説明する。図４（Ａ）は鏡面反射成分の観測値とモデル関数を示すグラフである。図４（Ｂ）はモデル関数曲線と観測値との距離を示すグラフである。

Next, modeling of fine irregularities on the object surface will be described with reference to FIG. FIG. 4A is a graph showing observed values and model functions of the specular reflection component. FIG. 4B is a graph showing the distance between the model function curve and the observed value.

  In the present invention, the unevenness at the pixel level is defined as a fine unevenness, and the bump model is defined based on the characteristics. Here, the difference between the specular reflection component estimation curve and the observation data is evaluated by paying attention to the fact that fine unevenness on the object surface appears prominently in the specular reflection component.

  Referring to FIG. 4 (A), the estimated curve shown in this figure is obtained by the above-described equation (6). This estimated curve does not completely match the specular reflection component and appears with some variation. If there are no fine irregularities on the object surface, the specular reflection component should be distributed without deviating from the model function. That is, this variation can be considered to occur because the specular reflection component is scattered by the influence of fine irregularities on the object surface.

  Referring to FIG. 4B, this graph is obtained by measuring the distance from the model function curve to the observed value in the direction of the angle axis as shown in FIG. 4A and examining the distribution. The horizontal axis of this graph is the angle difference β (radian) between the estimated curve and the observed value, and the vertical axis is the appearance frequency. In the obtained graph, the appearance frequency gradually decreases with a certain angle as a peak. In the present invention, this distribution is assumed to be a normal distribution, and a model formula shown in Equation 7 is defined.

ここで、μは法線の振れ幅の平均、υは法線のふれ幅の分散である。多くの場合、平均は極めて０に近い値となる。

Here, μ is the average of the fluctuation width of the normal line, and υ is the variance of the fluctuation width of the normal line. In many cases, the average is very close to zero.

<Third Embodiment>
In the present embodiment, a specific image processing method will be described. That is, the details of acquiring various parameters from the input real image and performing rendering using these parameters will be described.

  First, a specific process will be described with reference to FIG. In step S11, the camera position / orientation is specified. In step S12, a diffuse reflection component image is generated. In step S13, the specular reflection component is separated. In step S14, specular reflection color Ks is estimated. In step S15, a light source hemisphere is generated. In step S16, surface roughness parameters σ1 and σ2 and bump model parameters μ and υ in specular reflection are estimated. In step S17, the diffuse reflection index n is estimated. In step S18, a diffuse reflection image is generated. In step S19, the diffuse reflection component image is integrated and rendered. By these steps, various parameters of the reflection model can be estimated from a plurality of inputted real images, and an image in which the texture of the surface is expressed faithfully can be generated. Each step is described in detail below.

Step S11: See FIG. 6 In this step, the camera position and orientation are specified. FIG. 6A is a diagram illustrating a state in which three-dimensional coordinates are projected on an image. FIG. 6B is a diagram illustrating a state in which an actual image is obtained by photographing an object. FIG. 6C is an example of a photographed photographed image. FIG. 6D is a diagram showing three-dimensional data of a photographed image.

In the present invention, the camera position at the time of shooting is not fixed in order to realize a more flexible shooting environment. Therefore, it is necessary to specify the position and orientation in advance. The three-dimensional shape of the target object and the focal length of the camera at the time of shooting are known. Therefore, if the captured image matches the three-dimensional projected image, the projection position / posture can be handled as the position / posture of the camera at the time of photographing. Here, the three-dimensional shape and the captured image are aligned by rotating and translating the three-dimensional shape in the virtual space. As parameters necessary for alignment, there are six parameters including a translation component (tx, ty, tz) and a rotation component (roll, pitch, yaw). If the position / orientation of the camera can be specified in this way, three-dimensional coordinates corresponding to each pixel in the captured image can be obtained. The three-dimensional coordinates p _i = [X _i Y _i Z _i ] ^T corresponding to the point U _i = [x _i y _i ] ^T on the image are obtained by the following equation (8).

ここで、Ｒ、Ｔはそれぞれ回転行列、並進ベクトルであり、カメラ位置姿勢の特定において得られたものである、Ｐは透視投影を表している。回転行列ＲはＺ軸まわりの回転φ(roll)、Ｙ軸まわりの回転ψ(yaw)、Ｘ軸まわりの回転θ(pitch)を用いて以下の数９で表すことができる。

Here, R and T are a rotation matrix and a translation vector, respectively, which are obtained in specifying the camera position and orientation, and P represents perspective projection. The rotation matrix R can be expressed by the following equation 9 using the rotation φ (roll) around the Z axis, the rotation ψ (yaw) around the Y axis, and the rotation θ (pitch) around the X axis.

また、３次元座標ｐｉが画像上の点Ｕｉに投影されるとき、この関係はカメラの焦点距離ｆを用いて下記数１０で表される。

Further, when the three-dimensional coordinate pi is projected onto the point Ui on the image, this relationship is expressed by the following formula 10 using the focal length f of the camera.

また、本ステップでは、図６（Ｂ）に示すように、カメラの位置を移動させて物体の実写画像を複数撮影している。具体的には、異なるカメラの位置から３枚の画像を撮影している。この画像の例を図６（Ｃ）に示す。物体が球形である場合の３次元の形状データを図６（Ｄ）に示す。本発明では、実写画像および物体の形状データが主たる入力情報である。これらの情報から、各種パラメータを算出してレンダリングを行っている。

Further, in this step, as shown in FIG. 6B, the position of the camera is moved to capture a plurality of actual images of the object. Specifically, three images are taken from different camera positions. An example of this image is shown in FIG. FIG. 6D shows three-dimensional shape data when the object is spherical. In the present invention, the photographed image and the shape data of the object are the main input information. Rendering is performed by calculating various parameters from these pieces of information.

Step S12: See FIG. 7A In this step, the reflection component is separated from the photographed image. In the present invention, since the specular and diffuse reflection components are separately analyzed, it is necessary to separate both components. Therefore, an image composed only of the diffuse reflection component is generated. If the reflection of light on the object surface follows the diffuse reflection model described above, the diffuse reflection component at any one point on the object surface will always be the same luminance value in the image regardless of changes in the viewpoint position under the same light source environment. Appear in By utilizing this fact, an image composed of only the diffuse reflection component is generated from a plurality of images taken while moving the camera position under the same light source environment.

  First, one reference image is determined from the captured images, and three-dimensional coordinates on the object surface in the image are acquired. The three-dimensional coordinates can be obtained by perspective projection conversion because the three-dimensional shape of the target object, the shooting position of the camera, and the internal parameters are known. Next, image coordinates in the remaining image corresponding to the obtained three-dimensional coordinates are obtained, and the luminance value I is calculated by the following equation (11).

このようにして求められた数枚の画像中の対応するピクセルにおける輝度値の中でもっとも低い輝度値を画素における拡散反射成分として採用する。この処理を各画素に対して行うことで、拡散反射成分のみで構成される画像を生成することができる。ここで、得られた拡散反射成分画像中には光源からの光が到達しない領域も含まれるが、この段階では光源方向は未知であり、影かどうかの判定はできないため、この時点では影は考慮せず、拡散反射係数の推定時に考慮するものとする。

The lowest luminance value among the luminance values of the corresponding pixels in the several images thus obtained is adopted as the diffuse reflection component in the pixels. By performing this process on each pixel, an image composed of only the diffuse reflection component can be generated. Here, the obtained diffuse reflection component image includes a region where the light from the light source does not reach, but at this stage the direction of the light source is unknown and it cannot be determined whether it is a shadow. It is not taken into account, but is taken into account when estimating the diffuse reflection coefficient.

  FIG. 7A is a flowchart showing the above process. In step S51, it is determined whether or not the processing of each image in the reference pixel has been completed. If it has been completed (YES in step S51), the process ends. If not completed (NO in step S51), the process proceeds to step S52.

Then, the three-dimensional coordinates (X _att , Y _att , Z _att ) of the target image are acquired (step S52), and the luminance value of the target image is calculated (step S53). Thereafter, in step S54, it is determined whether there is an image that is not referred to. If it does not exist (NO in step S54), the process returns to step S51. If it exists (YES in step S54), the process proceeds to step S55.

In step S55, image coordinates (u, v) in three-dimensional coordinates (X _att , Y _att , Z _att ) are acquired. In step S56, the luminance value I _att at the image coordinates (u, v) is calculated.

In step S57, luminance values are compared. Specifically, it is determined whether or not I to which the lowest value is substituted in the steps so far is larger than Iatt. If it is smaller (YES in step S57), Iatt is substituted for I. If larger (NO in step S57), the process returns to step S54. The above is the description of the flowchart.

Step S13: See FIG. 7B In this step, the specular reflection component is separated. Specifically, the specular reflection component is separated by taking the difference between the diffuse reflection component image obtained as shown in FIG. 7B and the captured image. The specular reflection component image obtained in this way includes, in addition to the specular reflection component, the influence of the reflection of the surrounding environment and mutual reflection. However, the reflection component on the object surface is assumed to consist of only specular reflection and diffuse reflection, and the influence of reflection and mutual reflection is relatively small compared to the specular reflection component. Treat as.

Step S14
In this step, the specular reflection color Ks is estimated. Specifically, in the Torrance-Sparrow model, the specular reflection component on the object surface is proportional to the luminance value of the light source. Therefore, in this step, the average value of the RGB color vectors in each pixel in the specular reflection component image is obtained to obtain the specular reflection color Ks in the object.

Step S15: See FIG. 8 In this step, a light source hemisphere is generated and a light source direction vector is estimated. FIG. 8A is a schematic diagram showing a light source hemisphere, and FIG. 8B is an image of the light source hemisphere.

  With reference to FIG. 8A, in order to estimate the reflection intensity and the light source direction from the obtained specular reflection component image, a hemisphere covering an object is considered, and this is set as a light source hemisphere. First, a regular reflection vector is calculated from the normal vector and the line-of-sight direction vector at the point S on the object using the following equation (12).

次に、得られた正反射ベクトルと光源半球との交点に点Ｓの持つＲＧＢ値を与える。この操作を鏡面反射成分画像中の各ピクセルに適用することで、図８（Ｂ）のような光源半球を生成する。上記の作業により、光源方向ベクトルが算出される。

Next, the RGB value of the point S is given to the intersection of the obtained regular reflection vector and the light source hemisphere. By applying this operation to each pixel in the specular reflection component image, a light source hemisphere as shown in FIG. 8B is generated. The light source direction vector is calculated by the above operation.

  In the light source hemisphere thus obtained, it can be seen that the corresponding specular reflection component spreads on a concentric circle centered on the light source. The spread of the specular reflection component is unique to the object, and an object with a rough surface is wide, and an object with a smooth surface provides a light source hemisphere concentrated at one point.

Step S16: See FIG. 4 In this step, parameters in specular reflection are estimated. Specifically, first, roughness parameters σ1 and σ2 that are parameters related to the spread of the gloss point on the surface are estimated. Next, the bump model parameters are estimated.

Specifically, with reference to the graph of FIG. 4A, σ1 and σ2 are obtained by a statistical method so that the difference between the measured value and the estimated value of the luminance value is minimized. The above equation 6 is used to calculate the estimated value. Therefore, σ1 and σ2 are estimated by solving the least square problem shown in Equation 13 below.

ここで、Ｎは物体の法線ベクトル、Ｌは光源方向ベクトル、αは中間ペクトルと法線ベクトルとのなす角である。また、~Ｉｓ(ｘ、ｙ)は画像座標(ｘ、ｙ)における鏡面反射成分の輝度値を示している。

Here, N is a normal vector of the object, L is a light source direction vector, and α is an angle formed by the intermediate vector and the normal vector. Further, ~ Is (x, y) indicates the luminance value of the specular reflection component at the image coordinates (x, y).

  Subsequently, the bump model parameters are estimated. Specifically, a parameter that minimizes the estimated value based on the equation shown in Equation 7 and the actually measured value shown in FIG. That is, σ and μ were estimated by solving the following least squares problem of Equation 14. This makes it possible to obtain bump model parameters that reflect actual variations in luminance values.

上記数１４にて、Ｓｉは角度ｉにおける鏡面反射成分の観測値の出現頻度である。また、ｍは出現頻度のスケールを表している。ここではμ、υ、ｍの３つのパラメータを同時に最適化するが、ｍはμ、υを適切に推定するために与えるものであり、バンプパラメータとしては定義していない。また、υは０＜υ＜９０度の範囲内となるよう制約を加えた。それぞれのパラメータの初期値は以下のように設定する。
υ：出現頻度が最大となる角度＋（観測される最大の角度−出現頻度が最大となる角度）／２
μ：出現頻度が最大となる角度
ｍ：最大の出現頻度
上記パラメータを得ることにより、被写体となる物体の質感を忠実に再現することが可能となる。例えば、表面に凹凸を有するミカンが被写体である場合を想定すると、上記パラメータの設定により、このミカンの表面のざらつきが忠実に反映されたＣＧを得ることができる。

In the above equation 14, Si is the appearance frequency of the observed value of the specular reflection component at the angle i. M represents the scale of appearance frequency. Here, the three parameters μ, υ, and m are simultaneously optimized. However, m is given to appropriately estimate μ and υ, and is not defined as a bump parameter. In addition, a constraint was added so that υ is in the range of 0 <υ <90 degrees. The initial value of each parameter is set as follows.
υ: angle at which appearance frequency is maximum + (maximum observed angle−angle at which appearance frequency is maximum) / 2
μ: angle at which appearance frequency is maximum m: maximum appearance frequency By obtaining the above parameters, it is possible to faithfully reproduce the texture of an object that is a subject. For example, assuming that a mandarin orange having an uneven surface is an object, a CG that faithfully reflects the roughness of the mandarin orange surface can be obtained by setting the above parameters.

Step S17: See FIG. 9A In this step, the diffuse reflection parameter is estimated. The specular reflection component can be observed only in a relatively limited region, and it is difficult to estimate the specular reflection component at all points on the object surface. For this reason, in the present invention, the target object is treated as having a uniform specular reflection parameter. On the other hand, the diffuse reflection component represents the color of the object surface itself, and since the attenuation ratio is lower than that of the specular reflection component, it can be observed in a wide area. Therefore, in the present invention, the specular reflection component is assumed to be uniform on the object surface, but the diffuse reflection component is estimated as having different values at each vertex on the object. Therefore, in this step, after estimating the diffuse reflection index n, the diffuse reflection component at each vertex is estimated individually.

  First, parameter estimation using the extended Lambert model is performed. When the luminance value in each pixel of the diffuse reflection component image is plotted on the vertical axis and the angle between the normal vector and the light source direction vector in the three-dimensional coordinates corresponding to that pixel is plotted on the horizontal axis, the distribution is as shown in FIG. Thus, the distribution shows regular attenuation. The diffuse reflection index n can be estimated by fitting Equation 2 to this distribution. The diffuse reflection index n on the object surface can be obtained by solving the following least squares problem of Equation 15.

ここで、~Ｉｄ(ｘ、ｙ)は画像座標（ｘ、ｙ）におけるピクセルの輝度値を示している。また、Ｎは画像座標（ｘ、ｙ）に対応する３次元座標おける法線ベクトルを、Ｌは光源方向ベクトルを表している。

Here, ~ Id (x, y) indicates the luminance value of the pixel at the image coordinates (x, y). N represents a normal vector in a three-dimensional coordinate corresponding to the image coordinates (x, y), and L represents a light source direction vector.

  Next, the diffuse reflection color Kd is estimated. The diffuse reflection component image generated in step S12 described above is considered to be obtained by attenuating the color of the object itself, that is, the diffuse reflection color by the defined extended Lambert model. From this, the diffuse reflection color Kd in each pixel of the diffuse reflection component image can be obtained by the following equation (16).

法線ベクトルＮと光源方向ベクトルＬとのなす角θが９０度以上の場合、その頂点は光源からの光が到達していないため、拡散反射係数は〔０、０、０〕（ＲＧＢ）となる。上記の計算を拡散反射成分画像中の各ピクセルに対して行う。このことにより、画像中の各画素における拡散反射係数により構成される拡散反射係数画像を生成することができる。

When the angle θ formed by the normal vector N and the light source direction vector L is 90 degrees or more, the light from the light source does not reach the vertex, and the diffuse reflection coefficient is [0, 0, 0] (RGB). Become. The above calculation is performed for each pixel in the diffuse reflection component image. This makes it possible to generate a diffuse reflection coefficient image composed of the diffuse reflection coefficient at each pixel in the image.

Step S18: See FIG. 9B In this step, a diffuse reflection component image is generated. That is, the diffuse reflection coefficient image is mapped onto the object shape. Specifically, the diffuse reflection component is given to the entire object surface by mapping the obtained diffuse reflection coefficient image to the three-dimensional shape of the target object. Here, a different diffuse reflection component is given to each vertex of the object surface. The diffuse reflection component at the apex on the target object surface can be given by taking the correspondence with the diffuse reflection coefficient image estimated in the previous section. However, with only one image, it is not possible to observe diffuse reflection components at all vertices on all object surfaces due to the influence of self-hiding and shadows. Therefore, the diffuse reflection components at all vertices on the object surface are acquired by integrating a plurality of diffuse reflection component images estimated by rotating the object. Since there are a plurality of target diffuse reflection coefficient images, there may be a case where a plurality of images are given to one vertex. As processing at this time, an evaluation value shown in Expression 17 below is given to each bixel in the image based on the normal vector of each vertex and the viewing direction of the camera at the time of capturing the image.

法線ベクトルＮと視線方向ベクトルＶとのなす角θが９０度以上の場合、その頂点はカメラの視点位置からは見えないため、評価値には０を与える。それぞれの頂点においてこの評価値が最大となる画像が最も適切な画像である。しかしながら、カメラで撮影された実写画像には、カメラ位置推定の際に生じる誤差が含まれる。このことから、形状モデル上の同じ位置であっても、画像のずれによって隣接する頂点で適用する画像が異なる境界付近では明度差が生じ不自然な結果となる可能性がある。そこで、各画像における最大の評価値を用いるのではなく、図９（Ｂ）のように評価値に基づいてブレンディング処理を行うことで、画像の境界付近における明度を滑らかに変化させる。画像ｔにおける混合の重みＥｔは、下記する数１８によって決定する。ここでｎは画像枚数とする。

When the angle θ formed by the normal vector N and the line-of-sight direction vector V is 90 degrees or more, the vertex is not visible from the viewpoint position of the camera, so 0 is given to the evaluation value. The image having the maximum evaluation value at each vertex is the most appropriate image. However, an actual image captured by the camera includes an error that occurs when the camera position is estimated. For this reason, even at the same position on the shape model, there is a possibility that a difference in brightness occurs in the vicinity of a boundary where different images are applied at adjacent vertices due to image shift, resulting in an unnatural result. Therefore, instead of using the maximum evaluation value in each image, the brightness near the boundary of the image is smoothly changed by performing blending processing based on the evaluation value as shown in FIG. 9B. The blending weight Et in the image t is determined by the following Expression 18. Here, n is the number of images.

ステップＳ１９：図１０および図１１参照
このステップでは、推定された各種パラメータを用いて鏡面反射成分のレンダリングを行う。

Step S19: See FIG. 10 and FIG. 11 In this step, the specular reflection component is rendered using the various estimated parameters.

  First, the details of rendering will be described with reference to FIG. 10A is an image to which bump mapping is not applied, FIG. 10B is a mapping image, and FIG. 10C is an image to which bump mapping is applied.

  Here, bump mapping is used to render the specular reflection component using the bump model parameters estimated in the above step. Bump mapping is a technique for expressing unevenness on the object surface without changing the geometric shape of the object, and expresses unevenness by perturbing the normal vector of the vertex. Since this mapping does not directly perturb the vertex coordinates, there is no need to recalculate the coordinates, and the amount of calculation is reduced. Bump mapping is basically an application of texture mapping technology, and perturbs the normal vector according to a separately prepared map image as shown in FIG. In this method, since each pixel is perturbed during rendering, Phong shading that performs vertex normal interpolation is used as the shading algorithm.

  Specifically, first, in the generation of the bump map image, a normal deflection angle (β, φ) in each pixel is given as a luminance value. The perturbation to be applied to the normal line at each pixel is determined by β and φ.

  FIG. 10D is a schematic diagram showing perturbation imparted to the normal. Referring to the figure, the bump parameter estimated by Equation 7 described above affects only β and does not participate in φ. Therefore, β is given a value generated by a normally distributed random number having the average μ and variance υ estimated in the above step. Further, a value generated with a uniform random number is given to φ. As a result, the bump mapping shown in FIG. 10B is obtained. When the direction vector shown in FIG. 10D is expressed as a unit vector having a length of 1, this unit vector can be expressed as a vector having three elements with respect to the X direction, the Y direction, and the Z direction. In FIG. 10B, the bump image is expressed as a black and white image, but in reality, the image is a color image in which these three components are visualized. The bump image described above reflects unevenness on the surface of the object that is the subject.

  Next, the specular reflection component is rendered using the bump image. Specifically, the specular reflection component of each pixel is calculated using Equation 6, which is a model developed according to the present invention. In this case, σ1 and σ2 calculated in the above steps are applied. By applying these parameters, the “spread” of the gloss point, which is a specular reflection component, can be faithfully reproduced with respect to the object. Further, by applying the bump mapping, the “roughness” of the object surface is faithfully reproduced to obtain an image.

  FIG. 10A shows an image obtained as a result of rendering without applying bump mapping. Referring to this figure, the gloss points are uniformly distributed from one center to the periphery. Therefore, when an object having irregularities on the surface is a subject, the image shown in this figure is an unnatural image in which the irregularities on the object surface are not reproduced.

  FIG. 10C shows an image rendered by applying bump mapping. Compared with the image of FIG. 10A, the unevenness of the object surface is well expressed. That is, the image shown in this figure expresses the unevenness of the surface of the object with higher quality.

  Next, a case where the viewpoint position is moved or the focal length is changed will be described with reference to FIG. It is difficult to estimate the size of fine irregularities on the object surface by the model of the present invention. Therefore, here, it is assumed that the size of the unevenness at the time of shooting corresponds to one pixel at the time of image generation. However, when rendering a scene in which the viewpoint position is moved or the focal length is changed, the size of the unevenness should be changed as shown in FIG. Therefore, the size of the unevenness is changed by enlarging / reducing the bump map indicating the amount of change in the normal line in each pixel in accordance with the enlargement ratio based on the time of shooting. Pixel interpolation at the time of enlargement / reduction is realized by simple linear interpolation.

  FIG. 11 shows an image generated by changing the enlargement ratio and a bump map image used therefor.

  FIG. 11A shows an image with an enlargement ratio of 1.0. When rendering this image, a bump map image with an enlargement ratio of 1.0 shown in FIG. 11D is used.

  The image illustrated in FIG. 11B is an image obtained by enlarging the image illustrated in FIG. As the image is enlarged, enlarged unevenness is expressed. At the time of rendering of this image, a bump map image enlarged twice as shown in FIG. 11E is applied.

  The image shown in FIG. 11C is obtained by enlarging the image shown in FIG. 11A four times. Also in this case, the bump map image shown in FIG. 11F enlarged to fit the image is applied.

  The above is the specific image processing method of the present invention. The above-described embodiment can be applied to both still images and moving images. Specific examples of the scope of application of the present invention include electronic catalogs, digital archives, virtual museums, and the like.

<Fourth embodiment>
In the present embodiment, specific effects of the image processing apparatus of the present invention will be described with reference to the graph of FIG.

  The present invention is superior to the related art in terms of three points of estimation of the diffuse reflection component, specular reflection component and bump estimation. These details will be described with reference to the respective drawings in FIG.

  With reference to FIG. 12A, the effect of the present invention relating to the estimation of the diffuse reflection component will be described. In the graph shown in this figure, the vertical axis indicates the luminance value of the diffuse reflection component, and the horizontal axis indicates the angle formed by the light source direction and the normal line. Here, the object to be the subject is a blue sphere.

  In this graph, the set of points indicates the observed value of the luminance value of the diffuse reflection component. Moreover, the solid line has shown the estimation curve of this invention shown by Numerical formula 2. The dotted line shows the estimated curve of the conventional body represented by Equation 1.

  As is clear from this graph, the estimated curve of the present invention indicated by the solid line is smaller in difference from the observed value than the estimated curve of the conventional example indicated by the dotted line. Here, consider a case where the difference between the diffuse reflection component image and the generated image is compared. Then, in the conventional model, the average error and the maximum error were 0.0232 and 0.0849. In contrast, in the model of the present invention, the average error was 0.0149 and the maximum error was 0.0732. That is, the difference between the model of the present invention and the diffuse reflection image is smaller. Therefore, the diffuse reflection component model of the present invention more accurately models the actual diffuse reflection component.

  With reference to FIG. 12B, an effect relating to specular reflection component estimation will be described. The vertical axis and horizontal axis of this graph indicate the same as in FIG. Here, a silver sphere is used. Referring to this figure, it can be seen that the reflection model of the present invention indicated by the solid line (shown in Equation 6) has a smaller difference from the measured value than the conventional reflection model indicated by the broken line (shown in Equation 3). Here, consider a case where the difference between the specular reflection component image and the generated image is compared. Then, in the conventional model, the average error and the maximum error were 0.076 and 0.377. In contrast, in the model of the present invention, the average error was 0.064 and the maximum error was 0.264. That is, the model of the present invention has a smaller difference. Therefore, even in the specular reflection component, the model of the present invention more accurately represents actual reflection.

  The bump estimation will be described with reference to FIG. In this graph, the vertical axis represents the appearance frequency, and the horizontal axis represents the angle difference between the estimated curve and the observed value.

  In this graph, the light-colored bar graph shows the result using the conventional model (shown in Equation 3), and the dark-colored bar graph shows the result using the model of the present invention. The graph distribution of the conventional example is more widely distributed in the angular direction (horizontal axis direction) than the graph distribution of the present invention. From this, it can be seen that the model of the present invention has a smaller difference from the estimated value than the conventional model. Therefore, it can be understood that the specular reflection component model of the present invention is a model closer to an actual phenomenon.

Furthermore, in this graph, the solid curve indicates the estimated curve of the bump of the present invention. The wavy curve shows the bump estimation curve of the conventional example. From the graph, it can be understood that the curve of the present invention indicated by the solid line is more approximate to the distribution of the bar graph than the conventional example. From this, it can be understood that the bump estimation curve of the present invention is superior to the conventional one.

It is a figure which shows the electrical constitution of the image processing apparatus of this invention. It is a figure (A) which shows gypsum, and a figure (B) which shows distribution of a diffuse reflection component. FIG. 4A is a diagram showing a vector direction such as a light source direction, FIG. 4B is a diagram showing an example of an object as a subject, and FIG. 4C is a diagram showing a distribution of specular reflection components. It is a figure (A) which shows a luminance value of a specular reflection component, and a figure (B) which shows a specular reflection component which deviated from an estimated value. It is a flowchart which shows an image processing method. The figure (A) showing the alignment of the three-dimensional shape and the photographed image, the figure (B) showing the photography method, the figure (C) showing the photographed image, and the figure (D) showing the inputted three-dimensional coordinate data. is there. It is a flowchart (A) which shows the production | generation method of a diffuse reflection component image, and a figure (B) which shows isolation | separation of a specular reflection component. It is a figure (A) which shows a light source hemisphere, and a figure (B) which shows a light source hemisphere image. It is a figure (A) which shows distribution of a diffuse reflection component image, and a figure (B) which shows mixing of a diffuse reflection component image. A diagram showing an image to which bump mapping is not applied (A), a diagram showing a mapping image (B), a diagram showing an image to which bump mapping is applied (C), a diagram showing perturbation to normal (D), a mapping image It is a figure (E) which shows scaling of. FIGS. 4A to 4C show images to which bump mapping is applied, and FIGS. 4D to 4F show mapping images. FIGS. FIG. 4A is a diagram showing a luminance value of a diffuse reflection component, FIG. 4B is a diagram showing a luminance value of a specular reflection component, and FIG.

Claims (5)

Separation means for separating the actual value of the specular reflection component from the actual image obtained by photographing the object;
Estimating means for calculating an estimated value of the specular reflection component;
Calculation means for calculating a parameter of the specular reflection model based on a difference between the actual value and the estimated value of the specular reflection component;
Generating means for generating an image including a specular reflection component by the specular reflection model using the parameter;
Equipped with,
The image processing apparatus according to claim 1, wherein the generation unit performs bump mapping based on a difference between the measured value and the estimated value .   The specular reflection model is a Torrance-Sparrow model,
  The specular reflection component is calculated based on at least a sum of a first normal distribution function calculated from the first roughness parameter and a second normal distribution function calculated from the second roughness parameter. The image processing apparatus according to claim 1. The image processing apparatus according to claim 2, wherein the first roughness parameter and the second roughness parameter are obtained from measured values of the specular reflection component. The generation unit generates an image including a diffuse reflection component calculated using a Lambert model,
The image processing apparatus according to claim 1, wherein the Lambert model has a parameter greater than 0 and less than or equal to 1 that raises a term of a trigonometric function. The separating means separates a diffuse reflection component from the photographed image;
The image processing apparatus according to claim 4, wherein the parameter of the Lambert model is obtained from the diffuse reflection component.

Images