JP2019082955A - Image processing apparatus, image processing method and program - Google Patents

Abstract

To provide a process for acquiring reflection characteristics of an object with high accuracy.SOLUTION: The image processing apparatus includes: first acquisition means for acquiring, from an image, pixel values of a plurality of pixels whose geometric conditions determined by positions of a light source, an imaging device, and a target region in an object are different from one another; second acquisition means for acquiring normal-line information representing a normal direction on a surface of the target region; and determination means for determining reflection characteristics in the target region by processing using at least two or more pixel values among the pixel values of the plurality of pixels, and a reflectance calculated based on the geometric conditions and the normal directions corresponding to the two or more pixel values, wherein the processing accords to a distance between a position of the target region to which the plurality of pixels correspond, and a position in the object at which light from the light source is regularly reflected to the imaging device.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing technique for acquiring reflection characteristics of an object.

  It is generally known how to express the change (reflection characteristic) of the reflectance of an object according to the illumination direction and the observation (imaging) direction using a bi-directional reflectance distribution function (BRDF). There is. By using information about BRDF, it becomes possible to reproduce natural shadows and highlights in any lighting condition and gaze direction. Patent Document 1 discloses a technique for estimating a BRDF of an object by using a plurality of image data obtained by imaging while changing the light source direction or the observation direction.

JP 2003-203220 A

  In a general digital camera, the range of obtainable brightness is limited according to the setting of the exposure time and the aperture value. When the subject is brighter than a predetermined luminance value with respect to a certain exposure setting, the pixel value of the captured image is saturated and the luminance can not be identified. On the other hand, when the subject is too dark with respect to a certain exposure setting, the S / N ratio of the pixel value is reduced, which makes it difficult to identify the luminance.

  When the method of Patent Document 1 is used, how light from a light source hits in a captured image differs from region to region. For this reason, the BRDF is determined based on an image including pixels that are too bright and saturated or pixels that are too dark and having a low S / N ratio. In this case, it is difficult to obtain BRDF with high accuracy.

  The present invention has been made in view of the above problems, and an object thereof is to provide a process for acquiring the reflection characteristic of an object with high accuracy.

  In order to solve the above-mentioned subject, the image processing device concerning the present invention determines the reflective characteristic in the object field of the object using the picture obtained by an imaging device picturizing the object illuminated using a light source An image processing apparatus for acquiring, from the image, pixel values of a plurality of pixels having different geometric conditions determined by the positions of the light source, the imaging device, and the target area; The second acquisition means for acquiring normal information representing the normal direction on the surface of the region, at least two or more pixel values among the pixel values of the plurality of pixels, and the two or more pixel values respectively Determining means for determining reflection characteristics in the target area by processing using the geometric condition and the reflectance calculated based on the normal direction, and the processing includes Light from the light source at the position and the object of the target area, characterized in that responsive to the distance between the position of specular reflection relative to the imaging device of the pixel corresponds.

  According to the present invention, the reflection characteristic of an object can be obtained with high accuracy.

A diagram showing an example of the configuration of a reflection characteristic acquisition system Block diagram showing an example of the logical configuration of the image processing apparatus Diagram for explaining the geometric conditions θ, ρ Flow chart showing an example of processing executed by the image processing apparatus Flow chart showing an example of processing to determine BRDF A diagram showing an example of the positional relationship between a light source, an imaging device, an imaging region, and a regular reflection point A diagram showing an example of the distribution of luminance in an imaging region FIG. 7 is a view showing an example of the positional relationship between an imaging region and measurement points when performing multi-viewpoint imaging. Diagram showing the relationship between luminance value, angle ρ, and BRDF coefficients A diagram showing an example of a BRDF obtained at a measurement point including an end of a captured image A diagram showing an example of a BRDF obtained at a measurement point not including the end of a captured image Flow chart showing an example of processing to determine BRDF A diagram showing an example of the relationship between the pixel value of the reference point and the angle ρ Figure showing an example of the relationship between the angle ρ and the weighting factor

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments do not necessarily limit the present invention. Moreover, not all combinations of features described in the following embodiments are essential to the solution of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

First Embodiment
<Configuration of Reflection Characteristic Acquisition System>
FIG. 1 is a diagram showing an example of the configuration of a reflection characteristic acquisition system according to the present embodiment. The reflection characteristic acquisition system includes a multi-viewpoint imaging system 101 and an image processing apparatus 102. The multi-viewpoint imaging system 101 includes a light source 104, an imaging device 105, and a movable stage 106. The light source 104 in the present embodiment is a projector. Further, the imaging device 105 in the present embodiment is a color digital camera. The imaging device 105 captures an image of the subject 103 to which light from the light source 104 is applied while moving the imaging position by the movable stage 106 a plurality of times. A plurality of image data can be obtained by the plurality of imagings. The imaging device 105 generates image data having three channels of R, G, and B in each pixel. In the present embodiment, R, G, and B values represented by 16 bits are recorded in each channel as pixel values of image data, but the number of bits for representing pixel values is not limited to 16. It may be a pixel value represented by 8 bits or the like.

  The subject 103 has a shape close to a plane such as an oil painting. The movable stage 106 is a stage capable of moving the XY plane in a three-dimensional space whose axes are the X axis, the Y axis, and the Z axis shown in FIG. The light source 104 and the imaging device 105 are fixed to the movable stage 106, and move on a plane substantially parallel to the subject 103 while maintaining their relative positions. The subject 103 is placed approximately parallel to the XY plane, and the surface to be measured is directed to the movable stage 106 side. The light source 104 is installed in a direction facing the subject 103, and the imaging device 105 is installed in a direction slightly inclined to the light source 104 in order to image the front of the light source 104.

  The shape of the subject 103 and the configuration of the multi-viewpoint imaging system 101 are not limited to the above example. The subject does not have to be a plane, and may be, for example, a three-dimensional shape such as a sphere. In this case, a robot arm movable in all directions in a three-dimensional space may be used instead of the movable stage 106 in order to image the entire surface of the subject. Alternatively, the entire surface of the subject 103 may be imaged by fixing the imaging device 105 and placing the subject on the rotation table and rotating it. Alternatively, both the imaging device 105 and the subject 103 may move. Furthermore, by using a plurality of light sources or a plurality of imaging devices, imaging may be performed a plurality of times under different geometric conditions. The light source 104 is not limited to a projector, but may be an LED point light source as long as the light source 104 can be imaged under an illumination condition that can be approximated as a single point light source.

  The image processing apparatus 102 is, for example, a computer, and includes a CPU 107, a ROM 108, and a RAM 109. The CPU 107 executes an OS (Operating System) and various programs stored in the ROM 108, the HDD (Hard Disk Drive) 110, etc., using the RAM 109 as a work memory. The CPU 107 also controls each component via the system bus 111. Note that program code stored in the ROM 108 or the HDD 110 is expanded on the RAM 109 and executed by the CPU 107 according to a flowchart to be described later. An input device 113 such as a mouse and a keyboard and a multi-viewpoint imaging system 101 are connected to the general-purpose I / F (interface) 112 via a serial bus. The image processing apparatus 102 is connected to the light source 104, the imaging apparatus 105, and the movable stage 106 through a serial bus, and thus can control illumination conditions, imaging, a viewpoint position, and the like. The SATA (serial ATA) I / F 114 is connected to the HDD 110 and a general-purpose drive 115 that reads and writes various recording media via a serial bus. The CPU 107 uses various recording media mounted on the HDD 110 or the general-purpose drive 115 as a storage location of various data. A display 117 is connected to the video I / F 116. The CPU 107 displays a UI (user interface) provided by the program on the display 117, and receives an input such as a user instruction received via the input device 113.

<Issues in determining BRDF>
The problems in determining the BRDF of an object will be described with reference to FIGS. FIG. 6 is a view showing the positional relationship between the subject 103, the light source 104, and the imaging device 105 as viewed from behind the movable stage 106. As shown in FIG. The trapezoidal imaging region indicates a range (that is, a range shown in a captured image) within the angle of view of the imaging device 105 on the subject 103. The specular reflection point indicates the approximate position at which the specularly reflected light from the light source 104 reaches the imaging device 105 when the normal to the surface of the subject 103 coincides with the negative direction of the Z axis. . The regular reflection point is in the positive direction of the Z axis relative to the light source 104 and the imaging device 105. FIG. 7 is a diagram showing the distribution of luminance in a captured image. The intensity of the reflected light obtained by actual imaging is affected by the material and unevenness (distribution of height and normal) of the subject 103, but the reflected light tends to be acquired closer to the regular reflection point. In the example of FIG. 7, since the specular reflection point is located outside the right side of the imaging area, the area A near the center of the right end of the imaging area is the brightest, and the area J at the upper and lower ends of the left end of the imaging area is the most Get dark.

  FIG. 8 is a view showing an imaging range of each captured image when imaging is performed eight times while moving the movable stage 106 in the X-axis direction. When the reflection characteristic (BRDF) of the subject 103 is actually acquired, imaging is performed with many viewpoints such that the imaging region covers the entire subject 103 by movement including movement in the Y-axis direction. However, in order to simplify the explanation here, captured images of eight viewpoints obtained by imaging with movement in the X-axis direction are dealt with. Measurement points a to d in FIG. 8 indicate measurement points on the subject 103 for which a BRDF is to be determined. Measurement points a and c include areas near the left and right ends of the imaging area, and measurement points b and d do not include areas near the left and right ends of the imaging area. In the following description, the unevenness is not considered in the vicinity of each measurement point of the measurement points a to d, and the normal is assumed to be equal to the negative direction of the Z axis.

FIG. 9 is a diagram showing the relationship between the obtained BRDF coefficients (Cd _R , Cs _R , σ _R ) and the result of fitting the BRDF to a Gaussian curve. Here, a wavelength band corresponding to the R channel is taken as an example. The vertical axis of the graph is the luminance of R, and the horizontal axis of the graph is the angle ρ between the normal vector and the vector (half vector) between the observation direction vector and the light source vector. FIG. 10 is a diagram showing the relationship between the combination of the pixel value obtained from each pixel of the captured image and the angle ρ at the measurement point a or c in FIG. 8 and the Gaussian curve indicating the obtained BRDF. is there. The solid curve is a Gaussian curve showing a BRDF determined by the method of the present embodiment described later. The dotted curve is a Gaussian curve indicating the BRDF obtained when the weighting factor is not set in the optimization process for determining the BRDF. Not setting the weighting factor can be reworded as setting the weighting factor to a constant value. Plots A to I show combinations of the acquired R pixel values and the angle ρ, and each alphabet corresponds to the region in the image shown in FIG.

Here, the plot A indicates that the pixel value is saturated because the region A is close to the regular reflection point, and the pixel value is lower than the true luminance value. On the other hand, plot I is the darkest region in the image and may contain relatively large noise relative to the pixel values. In particular, when the light amount (Cd _R cos θ) of the diffuse reflection light at the measurement point is small, the S / N ratio is largely reduced. Under these conditions, when BRDF coefficients are optimized without setting weighting factors, plot A and plot I are strongly affected, and the resulting BRDF has a large error relative to the true value. . Therefore, in the present embodiment, the weighting factor in the optimization process is set according to the position in the image. That is, the weight coefficient of the plot E is maximized, and the weight coefficients of the plots A and I are set to 0 or a value close to 0. As a result, the influence of pixel value saturation and noise can be reduced, and BRDF closer to the true value can be obtained.

  FIG. 11 is a diagram showing the relationship between the combination of the pixel value of R obtained from each pixel and the geometric condition ρ at the measurement point b or d and the Gaussian curve indicating the obtained BRDF. The measurement points b and d do not include saturated plots of pixel values and plots that are largely affected by noise, so even when no weighting factor is set, a BRDF closer to the true value is obtained compared to the measurement points a and c. Be That is, when the BRDF coefficient optimization process is performed, if the weighting coefficient is not set, a difference occurs between the BRDF obtained at the measurement points a and c and the BRDF obtained at the measurement points b and d. This difference is perceived by the observer as a step along the imaging region of each captured image when the BRDF of the subject 103 is reproduced by computer graphics (CG) or the like.

  On the other hand, when the weighting factor is set by the method of the present embodiment, the weight of the plots D and F becomes relatively high, so that the BRDF close to the true value can be obtained similarly to the measurement points a and c. Therefore, there is no difference in BRDF obtained at each measurement point. That is, according to the method of the present embodiment, the reflection characteristic (BRDF) of the object (subject) can be obtained with high accuracy, and the step along the imaging region when reproducing the BRDF of the object by CG, printing, etc. Can generate an image that is less likely to occur.

<Logical Configuration of Image Processing Device>
The logical configuration of the image processing apparatus 102 will be described below. The processing of each unit described below is implemented as software by the CPU 107 executing a computer program read onto the RAM 109 from the ROM 108 or the like. FIG. 2A is a block diagram showing the logical configuration of the image processing apparatus 102. As shown in FIG. The image processing apparatus 102 includes a first acquisition unit 201, a second acquisition unit 202, a calculation unit 203, a third acquisition unit 204, a determination unit 205, and an output unit 206.

  The first acquisition unit 201 acquires a plurality of image data acquired by the imaging device 105 and imaging parameters corresponding to each imaging for obtaining each image data. The imaging parameters are the position and orientation of the light source 104, the position and orientation of the imaging device 105, and internal parameters of the imaging device 105 in imaging. The internal parameters are the focal length of the lens of the imaging device 105, the principal point position, and the distortion parameter of the image caused by the lens or the like. The first acquisition unit 201 in the present embodiment performs imaging by driving each part of the multi-viewpoint imaging system 101 to acquire image data. The first acquisition unit 201 may acquire image data and imaging parameters stored in advance in the HDD 110 or the like.

  The second acquisition unit 202 acquires shape data representing the three-dimensional shape of the subject 103. In the present embodiment, shape data is generated based on a plurality of image data acquired by the first acquisition unit 201 and imaging parameters. Specifically, first, point cloud data representing a point cloud (a set of points having coordinate information in a three-dimensional space) indicating a three-dimensional shape of the subject 103 is generated using a known stereo method. Based on the generated point cloud data, the point cloud is converted into a polygon mesh model using a known PSR (Poisson Surface Reconstruction) method. Thus, shape data representing the three-dimensional shape of the subject 103 is generated using the polygon mesh model. Here, the polygon mesh model is a model that represents the surface shape of an object using a set of triangles. The shape data is not limited to data representing the three-dimensional shape of the subject 103 using a polygon mesh model, and for example, if it is an object close to a plane like the subject 103 in the present embodiment, the height distribution of the object is represented. It may be height data (height field data).

  In the present embodiment, shape data is generated based on a plurality of image data obtained by imaging from multiple viewpoints, but the method of acquiring shape data is not limited to this. For example, a method may be used in which shape data is acquired by projecting specific pattern light onto the subject 103 using the light source 104 and capturing the reflected light using the imaging device 105. Alternatively, a method of acquiring shape data of the subject 103 by controlling a laser range scanner or the like connected to the image processing apparatus 102 may be used. Alternatively, shape data stored in advance in the HDD 110 or the like may be acquired.

  The calculation unit 203 calculates a normal vector (normal information), a light source vector, and an observation direction vector for a combination of each point in the three-dimensional shape represented by the shape data and an image including an area corresponding to each point. . Here, each point in the three-dimensional shape is a point located on the surface of the polygon mesh model. In the present embodiment, since imaging is performed so as to be an imaging region as shown in FIG. 8, a plurality of images including regions corresponding to each point exist. The normal vector is calculated as a unit vector indicating the outward normal direction of the triangle in which the target point is included in the shape data. The calculation unit 203 calculates a normal vector by the outer product of two vectors in a triangle. The light source vector is calculated as a unit vector pointing from the point of interest to the coordinates of the light source 104. The observation direction vector is calculated as a unit vector pointing from the point of interest to the coordinates of the imaging device 105. The calculation unit 203 calculates the light source vector by vector operation based on the coordinates of the target point in the shape data, the position (coordinates) of the light source 104 and the position (coordinates) of the imaging device 105 as in the normal vector described above. Calculate the observation direction vector. The relative positional relationship between the coordinates of the target point in the shape data, the position (coordinates) of the light source 104 and the position (coordinates) of the imaging device 105 may be calculated using the position information of the movable stage 106 Good. Also, the relative positional relationship may be calculated using a known calibration method. When height field data is used as shape data, a normal vector may be calculated based on a differential value of height information.

  The third acquisition unit 204 acquires the pixel values (R value, G value, B value) of the corresponding image data for each point in the three-dimensional shape represented by the shape data. In the present embodiment, as described above, there are a plurality of images including the area corresponding to each point, and therefore, a plurality of pixel values are acquired for each image.

For each point in the three-dimensional shape represented by the shape data, the determination unit 205 determines a BRDF coefficient (R, G, B) based on a plurality of combinations of geometric conditions (θ, 組 み 合 わ せ). Cd _R , Cs _R , σ _R, Cd _G , Cs _G , σ _G , Cd _B , Cs _B , σ _B ) are determined. The coefficients of BRDF are reflection characteristic parameters for determining the reflection characteristic of an object. As shown in FIG. 3, the geometric condition represents an angle θ formed by the normal vector and the light source vector, and an angle ρ formed by the half vector and the normal vector of the observation direction vector and the light source vector. In the present embodiment, a Gaussian model shown in Expression (1) is used as the BRDF of the subject 103.

R = Cd _R cos θ + Cs _R exp (−ρ ² / 2σ _R ² ) cos θ
G = Cd _G cos θ + Cs _G exp (−ρ ² / 2σ _G ² ) cos θ formula (1)
B = Cd _B cos θ + Cs _B exp (−ρ ² / 2σ _B ² ) cos θ
Cd _R is a diffuse reflection coefficient indicating the reflectance (reflection intensity) of the diffuse reflection light in the wavelength band corresponding to the R channel. Cs _R is a specular reflection coefficient indicating the reflectance (reflection intensity) of specular reflected light in a wavelength band corresponding to R channel, and σ _R is a specular reflection coefficient indicating the spread of specular reflected light in a wavelength band corresponding to R channel . Similarly, Cd _G , Cs _G and σ _G are the diffuse reflection coefficient and the specular reflection coefficient in the wavelength band corresponding to the G channel. Further, Cd _B , Cs _B and σ _B are the diffuse reflection coefficient and the specular reflection coefficient in the wavelength band corresponding to the B channel.

  The determination unit 205 determines the difference (error) between the left side and the right side using Equation (1) for each of a plurality of combinations of pixel values (R value, G value, B value) and geometric conditions (θ,)). Calculate and optimize the BRDF coefficients so as to minimize the sum of differences. In addition, when optimizing the BRDF coefficients, the determination unit 205 performs weighting in accordance with the position of the pixel in the image for each combination. Details of optimization processing and weighting of the BRDF coefficients will be described later.

  The output unit 206 generates output data by associating the BRDF coefficients determined at each point in the three-dimensional shape represented by the shape data with the three-dimensional shape represented by the shape data, and outputs the output data to the HDD 110.

<Process Performed by Image Processing Device>
Next, processing executed by the image processing apparatus 102 will be described with reference to FIGS. 4A and 5A. FIG. 4A is a flowchart of processing performed by the image processing apparatus 102. In the following, each step (step) is represented by adding S to the front of its reference numeral.

  In S401, the first acquisition unit 201 acquires a plurality of image data and imaging parameters corresponding to each image data. In step S402, the second acquisition unit 202 generates shape data representing the three-dimensional shape of the subject 103 based on the image data acquired in step S401 and the imaging parameters.

  In S403, the calculation unit 203, the third acquisition unit 204, and the determination unit 205 perform BRDF determination processing on each surface (each triangle) of the polygon mesh model represented by the shape data. Details of the BRDF determination process will be described later. Note that S403 loops until each triangle is processed. In S404, the output unit 206 generates output data to which the information of the BRDF coefficients determined in S403 is added for each surface of the polygon mesh model, and outputs the output data to the HDD 110.

<BRDF decision processing>
Hereinafter, the BRDF determination process executed in S403 will be described in detail. FIG. 5A is a flowchart showing BRD determination processing in the present embodiment.

In step S501, the calculation unit 203 acquires coordinates in a three-dimensional space of a point (measurement point) which is a target for which a BRDF is to be determined. Specifically, the coordinates of the center of gravity of the triangle to be processed are calculated using the coordinates of each vertex of the triangle as the coordinates of the measurement point. Next, S502 to S504 are looped until all image data corresponding to the measurement point to be processed are processed. In step S502, the calculating unit 203 acquires, for the n-th image data, a normal vector, a light source vector, an observation direction vector, and the coordinates of the light source 104 and the imaging device 105. Furthermore, the calculation unit 203 calculates an angle [rho _n the angle theta _n formed by the normal vector and the light source _vector, and a half vector and the normal vector of the viewing direction vector and the light source vector eggplant.

In S503, the third acquisition unit 204 acquires, in the n-th image data, a pixel value corresponding to the measurement point whose coordinates are acquired in S501. Specifically, based on the coordinates of the imaging device 105 acquired in step S502 and the imaging parameters acquired in step S401, which pixel position (x, the image point represented by the n-th image data in the three-dimensional space represents) Calculate whether it is reflected in y). Since the value of the pixel position (x, y) calculated here includes fractions, the pixel value corresponding to the pixel position (x, y) is calculated by performing interpolation processing using the pixel values of the nearby four pixels. calculate. A list of combinations of the acquired pixel values and the geometric conditions (θ _n , _{n n} ) calculated in S502 is generated. In the image, when the triangle to be processed is the back surface, the measurement point is behind another part, or the measurement point is outside the image, it is assumed that the pixel value can not be acquired.

In S504, the determining unit 205 determines the weighting factor used in the optimization process of S505 by referring to the list of combinations generated in S503. In this embodiment, on the basis of the coordinates in each image from which the pixel values are acquired, the coefficient is calculated so as to take a value of 0 or a value close to 0 at the edge of the image and a large value at the center of the image. As an example, the pixel position (x, y) at which the pixel value is acquired in the n-th image data is normalized by dividing it by the width and height (W, H) of the image and (u _n , v _n And). In this case, to calculate the weight coefficient k _n to be applied to the combination of the pixel values and the geometric conditions for the n-th image data by equation (2).

k _n = ((1−cos (2πu _n )) * (1−cos (2πv _n ))) ^0.5 (2)
According to the above determination method of the weighting factor, the optimization processing in the present embodiment is performed according to the distance between the position on the object corresponding to each of the plurality of pixels and the position (specular reflection point) at which light from the light source is specularly reflected Optimization process. That is, the weight is small for the pixel whose distance is too large (pixel too dark) and the pixel whose distance above is too small (pixel too bright). Note that for image data for which the pixel value can not be acquired in S503, the weight coefficient is set to 0 because the combination of the pixel value and the geometric condition is not used in the optimization process of S505.

In step S505, the determination unit 205 determines the BRDF coefficients (Cd _R , Cs _R , σ _{R, C} d _G , Cs _G , σ _G , etc.) at the measurement point based on the combination of the pixel value obtained from each image data and the geometric condition. Determine Cd _B , Cs _B , σ _B ). In the present embodiment, BRDF coefficients are determined independently in each of the R, G, and B wavelength bands.

The determination of the BRDF in each wavelength band is performed by a known optimization algorithm such as the Newton method using the least squares method as an evaluation function. For example, in the wavelength band of _R , a combination of parameters such as BRDF coefficients (Cd _R , Cs _R , σ _R ) as optimization target parameters and minimizing the evaluation function E _R represented by equation (3) Derive Here, N is the number of image data used for the optimization process.

Similarly, evaluation functions E _G and E _B are defined for G and B wavelength bands, and BRDF coefficients (Cd _G , Cs _G , σ _G ) and (Cd _B , Cs _B , σ _B ) are optimized. .

<Effect of First Embodiment>
As described above, the image processing apparatus according to the present embodiment acquires, from an image, pixel values of a plurality of pixels having different geometric conditions determined by the positions of the light source, the imaging device, and the object. Furthermore, normal information representing a normal direction on the surface of the object is acquired. Object by processing using at least two or more pixel values among pixel values of a plurality of pixels, and a reflectance calculated based on a geometric condition and a normal direction corresponding to each of the two or more pixel values Determine the reflection characteristics in the target area of This processing corresponds to the distance between the position of the target area corresponding to the plurality of pixels and the position at which light from the light source at the object is specularly reflected to the imaging device. As a result, it is possible to prioritize and use pixel values other than the pixel values of pixels that are too bright and saturated and pixels that are too dark and that have a low S / N ratio for optimization. Therefore, the reflection characteristic of the object can be obtained with high accuracy.

Second Embodiment
In the first embodiment, a method has been described in which an object having a shape close to a flat surface such as an oil painting is used as a subject, and the weight coefficient is determined using the correlation between the position in the image and the geometric condition. However, when the surface of the subject has large irregularities, or when the reflection characteristic is obtained for a subject having a three-dimensional shape such as a sphere, the geometric conditions at the time of imaging are strongly affected by the subject's shape. The degree of correlation between the position and the geometric condition decreases. In the present embodiment, an example will be described in which a weighting factor is determined based on geometric conditions for combinations of acquired pixel values and geometric conditions. The configuration of the reflection characteristic acquisition system and the logical configuration of the image processing apparatus 102 in the present embodiment are the same as those in the first embodiment, and thus the description thereof is omitted. Since the details of the BRDF determination process in S403 are different between the present embodiment and the first embodiment, the details of the BRDF determination process in the present embodiment will be described using FIGS. 12 to 14.

<BRDF decision processing>
FIG. 12 is a flowchart showing BRDF determination processing in the present embodiment. The processes in S501 to S503 and S505 are the same as in the first embodiment, and thus the description thereof is omitted.

  In step S <b> 1201, the calculation unit 203 determines a set of points (reference points) in the area around the measurement point of the subject 103. Specifically, triangles at positions closer than a predetermined distance from the measurement point in the three-dimensional space are extracted, and a list of barycentric points of each triangle is generated. The reference point is not limited to the center of gravity of the triangle, and all points within the predetermined distance (the vertices of the triangle) may be extracted. In S1202, the calculation unit 203 acquires, for each reference point, the normal vector, the light source vector, the observation direction vector, and the coordinates of the light source 104 and the imaging device 105, as in S502. In S1203, the third acquisition unit 204 acquires pixel values for each reference point, as in S503.

In S1204, the determination unit 205 generates, for each reference point, a function for determining a weighting factor based on the combination of the acquired pixel value and the geometric condition ρ. FIG. 13 shows the relationship between the pixel value of each reference point and the geometric condition ρ. The vertical axis 65535 is the maximum value that the pixel value represented by 16 bits can take. The vertical axis t is a threshold for setting the lower limit to the pixel value used for the optimization process. This threshold can reduce the influence of pixels having a low S / N ratio on optimization processing. In the processing in this step, first, ρ _low which is the maximum value of す_る at which a pixel saturated with any one of R, G and B appears, and the pixel value of any one of R, G and B have threshold t. Get 最小_high which is the smallest value of 下回_る below. Next, based on the obtained _{low low} and ρ _high , a function for determining a weighting factor is generated. FIG. 14 is a diagram showing the relationship between the geometric condition と and the weighting factor in the generated function. ρ _mid is an intermediate value between _{low low} and ρ _high , and the weighting factor is 0 at 以下_low and ρ _high and at ρ _mid . Also, the weighting factor increases linearly from _{low low} to _{mid mid} and decreases linearly from _{mid mid} to _high . The weighting factor may increase non-linearly from _{low low} to _{mid mid} or may decrease non-linearly from ρ _mid to ρ _high .

  In S1205, based on the function generated in S1204, a weighting factor to be applied in the optimization process is determined for the combination of each pixel value at the measurement point and the geometric condition ρ. As in the first embodiment, the optimization process according to the present embodiment determines the position of the object corresponding to each of the plurality of pixels and the position at which the light from the light source is specularly reflected by the object. Optimization processing according to the distance to the reflection point).

<Effect of Second Embodiment>
As described above, the image processing apparatus according to the present embodiment determines the weighting factor used for the optimization process for determining the BRDF based on the geometric condition. Therefore, even when the surface asperity of the object is large or when the object has a three-dimensional shape, it is possible to obtain the reflection characteristic of the object with high accuracy.

Third Embodiment
In the embodiment described above, an example in which one normal vector is calculated for one polygon as the normal vector has been described. However, since the normal direction of an object with irregularities on the surface changes finely even in one polygon that constitutes the object in the data, the optimization process acquires the detailed normal direction change in the polygon There is a need to. However, as a result of the optimization process, issues as described in the issue of determination of BRDF also occur in the normal direction. That is, when the optimization process is performed without weighting when acquiring the normal vector, a decrease in acquisition accuracy and a step along the imaging region occur. In this embodiment, an example will be described in which normal vectors corresponding to each polygon are determined by optimization processing weighted similarly to the determination processing of BRDF in the above-described embodiment. The configuration of the reflection characteristic acquisition system according to this embodiment is the same as that of the first embodiment, and thus the description thereof is omitted.

<Logical Configuration of Image Processing Device>
FIG. 2B is a block diagram showing the logical configuration of the image processing apparatus 102. The image processing apparatus 102 includes a first acquisition unit 201, a second acquisition unit 202, a calculation unit 203, a third acquisition unit 204, a determination unit 205, an output unit 206, and a setting unit 207. The first acquisition unit 201, the second acquisition unit 202, the calculation unit 203, the third acquisition unit 204, and the output unit 206 perform the same processing as the above-described embodiment, and thus the description thereof is omitted. The setting unit 207 associates the coordinates in the image (hereinafter referred to as a texture image) with the coordinates in the shape data representing the texture to be attached to the surface of the three-dimensional shape represented by the shape data based on known texture mapping. decide. Specifically, it is set which coordinate on the texture image corresponds to each vertex of a triangle forming a three-dimensional shape represented by the shape data. Here, the coordinates on the texture image are set as values of 0.0 to 1.0 normalized by the width and height of the image, and the number of vertical and horizontal pixels of the texture image may take any value. The coordinates inside the triangle are similarly mapped according to the distance from each vertex as the coordinates inside the corresponding triangle projected onto the texture image. When height field data is used as shape data, the height and width of the height field data may be made to correspond to those of the texture image as it is. In addition to the coefficients of the BRDF, the determination unit 205 determines a normal vector by optimization processing. Optimization of the normal vector is performed using the normal vector calculated by the calculation unit 203 as an initial value.

<Process Performed by Image Processing Device>
Next, processing executed by the image processing apparatus 102 will be described with reference to FIGS. 4 (b) and 5 (b). FIG. 4B is a flowchart of processing executed by the image processing apparatus 102. In addition, since S401 and S402 are the processes equivalent to Embodiment 1, description is abbreviate | omitted.

  In step S601, the setting unit 207 determines the correspondence between the coordinates on the texture image corresponding to the polygon mesh model and the coordinates of the vertex of each triangle. Although the resolution of the texture image in the present embodiment is 4096 pixels × 4096 pixels, the resolution may be different depending on information of BRDF to be acquired. In step S602, the calculation unit 203, the third acquisition unit 204, and the determination unit 205 perform BRDF and normal vector determination processing on each pixel of the texture image. Details of the BRDF and normal vector determination processing will be described later. Note that S602 loops until each pixel of the texture image is processed. In step S603, the output unit 206 generates output data obtained by adding information of BRDF coefficients and normal vectors as texture images to the polygon mesh model, and outputs the output data to the HDD 110.

<BRDF and Normal Vector Determination Processing>
The BRDF and normal vector determination process executed in S602 will be described in detail below. FIG. 5B is a flowchart showing BRDF and normal vector determination processing in the present embodiment. In addition, since S502 to S505 are the same processes as the first embodiment, the description will be omitted.

  In step S701, the calculation unit 203 acquires coordinates in a three-dimensional space of a point (measurement point) which is a target for which a BRDF is to be determined. Specifically, first, triangles of the polygon mesh model corresponding to the pixels on the texture image to be processed are extracted. Coordinates on shape data corresponding to pixels on the texture image to be processed are calculated by interpolation calculation based on the coordinates of each vertex of the extracted triangle and the coordinates on the texture image.

In S702, the determination unit 205 determines the BRDF coefficients (Cd _R , Cs _R , σ _{R, C} d _G , Cs _G , σ _G , etc.) at the measurement point based on the combination of the pixel value obtained from each image data and the geometric condition. Determine Cd _B , Cs _B , σ _B ) and normal vectors.

The determination of the BRDF and the normal vector is performed by a known optimization algorithm such as the Newton method using the least squares method for the evaluation function. The optimization process takes BRDF coefficients and a normal vector (two degrees of freedom between the rotation angle and the elevation angle because it is a unit vector) as parameters to be optimized, and minimizes the evaluation function E _RGB represented by equation (4) It is a process which derives the combination of such parameters. Here, N is the number of image data used for the optimization process. Further, θ _n is an angle formed by the normal vector and the light source vector, and _{n n} is an angle formed by the half vector of the observation direction vector and the light source vector and the normal vector. The θ _n and _{n n} also change with the change of the normal vector as a parameter.

<Effect of Third Embodiment>
As described above, the image processing apparatus according to the present embodiment determines the reflection characteristic and the normal direction by the optimization process using the weighting factor. As a result, the optimization processing can be performed in a state where the weight of a pixel which is too bright and saturated and a pixel which is too dark and which is largely affected by noise can be reduced, so that the reflection characteristic of the object can be obtained with high accuracy. In addition, since similar optimization processing is performed also in the normal direction, the detailed surface shape (concave and convex and inclined) of the object can be acquired with high accuracy.

[Modification]
In the first embodiment, the weight coefficient used for the optimization process is determined based on the position in the captured image, and in the second embodiment, the weight coefficient used for the optimization process based on the distribution of geometric conditions including the peripheral region. It was determined. However, the method of determining the weighting factor is not limited to the above example, and for example, the methods of determining the weighting factor in the two embodiments described above may be combined. For example, weighting factors determined by respective methods in the two embodiments described above may be multiplied, and values obtained by the multiplication may be used as weighting factors.

  In the first embodiment, an example in which the regular reflection point is outside the imaging area has been described, but the regular reflection point may be inside the imaging area. In this case, the coefficient is calculated so that the position of the regular reflection point in the image takes a value of 0 or a value close to 0, and the position in the image takes a larger value as it goes away from the regular reflection point. Specifically, the coefficients may be calculated by the method of the first embodiment or the method of the second embodiment, or a combination of the two methods as described above. The position of the specular reflection point in the image may be determined based on, for example, geometric conditions, or may be determined based on the luminance value (brightness) of the image.

  Also, in the third embodiment, both the BRDF and the normal vector are determined by the optimization process, but either one may be determined by the optimization process. For example, coefficients of BRDF may be determined in advance to predetermined values, and only normal vectors may be optimized.

  In the embodiment described above, an object is imaged using one imaging device fixed to a movable stage, but the number and arrangement of imaging devices and light sources are not limited to the above-described example. For example, a camera, an infrared camera, or the like capable of imaging a wavelength band of visible light other than R, G, and B may be added.

  Further, in the above-described embodiment, BRDFs are determined independently for each pixel of each triangle or texture image that constitutes an object. On the other hand, when it can be regarded as having the same BRDF, for example, a certain area on the object is made of the same material, the pixel value obtained from each image data corresponding to a plurality of measurement points of that area and the geometric condition The BRDF may be determined on an area basis based on the combination. In this case, N in Equation (3) and Equation (4) is the total number of combinations of pixel values and geometric conditions spanning a plurality of measurement points. In such a case, the BRDF can be determined based on a smaller number of image data or one image data than the method of the above-described embodiment. For example, when all regions of an object are made of a uniform material, pixel values and geometric conditions are acquired for all pixels including the object in image data obtained by performing imaging from only one viewpoint, and The BRDF coefficients may be optimized based on the evaluation function of 3). In this case, N in equation (3) is the number of pixels including an object.

  In the embodiment described above, optimization processing is performed using all pixel values acquired from image data, but it is not necessary to use all pixel values for optimization processing, and at least two or more pixel values may be used. It may be used for optimization processing. For example, among the acquired pixel values, pixel values of pixels with high reliability may be used for the optimization process. A pixel that is too bright and saturated or a pixel that is too dark and that has a low S / N ratio is regarded as a pixel with low reliability, and the other pixels are regarded as pixels with high reliability. Specifically, for example, the above-described 65535 is set as the upper limit threshold, the value input by the user is set as the lower limit threshold, and determination of the reliability of the pixel value using the threshold is performed. Pixel values larger than the lower limit threshold value are set as pixel values with high reliability.

  Although the specular reflection coefficient and the diffuse reflection coefficient are optimized in the above-described embodiment, the specular reflection coefficient may be optimized without optimizing the diffuse reflection coefficient. In this case, for example, the diffuse reflection component may be removed in advance using image data obtained by imaging using a polarizing plate or the like.

  In the above-described embodiment, the BRDF coefficients are determined for each of the R, G, and B wavelength bands, but based on the pixel values (R value, G value, B value) of the image data, the following formula (5) The luminance value Y may be calculated by using the above equation to determine the BRDF coefficient for the luminance. In this case, since it is not necessary to determine the BRDF coefficients in each wavelength band, it is possible to reduce the processing time and data amount.

Y = 0.2126 × R + 0.7152 × G + 0.0722 × B (5)
Further, in the above-described embodiment, in the shape data, the surface shape of the object is expressed using a set of triangles, but polygons other than triangles such as quadrilateral may be used to represent the surface shape.

In the above embodiment, the pixel values of the image data are R value, G value, and B value, but L ^* value, a ^* value, b ^* value defined in L ^* a ^* b ^* space It may be another type of value such as.

  Further, although the output unit 206 in the above-described embodiment outputs the output data to the HDD 110, the output destination of the output data is not limited to the HDD 110. For example, data for output may be output to the display to display an object having the determined BRDF on the display. Further, in order to reproduce the object having the determined BRDF using a printer, the output data may be output to an inkjet printer, a 3D printer or the like capable of forming an object having a three-dimensional shape.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

102 image processing apparatus 202 second acquisition unit 204 third acquisition unit 205 determination unit

Claims (21)

An image processing apparatus for determining a reflection characteristic of a target region of an object using an image obtained by an imaging device imaging an object illuminated using a light source,
A first acquisition unit configured to acquire pixel values of a plurality of pixels having different geometric conditions determined by the positions of the light source, the imaging device, and the target area from the image;
A second acquisition unit configured to acquire normal information representing a normal direction on the surface of the target area;
At least two or more pixel values among the pixel values of the plurality of pixels, and the reflectance calculated based on the geometric condition and the normal direction corresponding to each of the two or more pixel values Determining means for determining reflection characteristics in the target area by processing;
The image processing apparatus according to claim 1, wherein the processing is performed according to a distance between a position of the target area corresponding to the plurality of pixels and a position where light from the light source specularly reflects the imaging device in the object. The determination unit may calculate the reflectance based on the geometric condition and the normal direction corresponding to at least two or more pixel values among the pixel values of the plurality of pixels and the two or more pixel values. The reflection characteristic in the target region is determined by optimizing the reflection characteristic parameter for determining the reflection characteristic in the target region so as to reduce the sum of the differences of and.
The optimization is performed according to a distance between a position of the target area corresponding to the plurality of pixels and a position where light from the light source is specularly reflected on the imaging device at the object. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein the sum is a weighted sum calculated by weighting each of the differences using a weighting factor.   The image processing apparatus according to claim 3, wherein the determination unit determines the weighting factor based on a position in the image corresponding to the target area.   The determination means is configured such that the weighting factor at the end of the image is lower than the weighting factor at the central part of the image when imaging is performed under a geometric condition in which the position where the specular reflection occurs is outside the image. The image processing apparatus according to claim 4, wherein the weight coefficient is determined.   The determination means includes a first area having the highest luminance in the image, a second area having the lowest luminance in the image, and a third area having a lower luminance than the first area and a luminance higher than the second area. 4. The image processing apparatus according to claim 3, wherein the weight coefficient is determined such that a weight corresponding to the third region is larger than the first region and the second region in the image including .   The image processing apparatus according to claim 3, wherein the determination unit determines the weight coefficient based on the geometric condition. A first angle formed by a half vector of a first vector representing the direction from the object to the light source, and a half vector of a second vector representing the direction from the object to the imaging device, and a third vector representing the normal direction And calculating means for calculating
The image processing apparatus according to claim 7, wherein the determination unit determines the weighting factor based on the first angle of imaging corresponding to the target area.   Based on the smallest second angle among the first angles corresponding to pixel values less than or equal to a first threshold and the largest third angle among the first angles corresponding to the largest pixel value in the image The image processing apparatus according to claim 8, wherein the weight coefficient is determined.   In the first angle, the weight of the pixel corresponding to the fifth angle between the second angle and the third angle is larger than the weight of the pixels corresponding to the second angle and the third angle. 10. The image processing apparatus according to claim 9, wherein the weight coefficient is determined.   The image processing apparatus according to any one of claims 3 to 10, wherein the weighting factor is such that the lowest value is 0 or a value close to 0.   The determination means uses the pixel values of the plurality of pixels acquired by the first acquisition means, based on the reliability of each pixel value of the plurality of pixels, for the at least two or more pixel values used in the process. The image processing apparatus according to any one of claims 1 to 11, wherein the image processing apparatus determines the image.   The determination means determines, as the at least two or more pixel values, pixel values which are equal to or less than the first threshold and greater than a second threshold smaller than the first threshold among pixel values of the plurality of pixels. The image processing apparatus according to claim 12, characterized in that:   14. The image processing apparatus according to claim 1, wherein the first acquisition unit acquires pixel values of the plurality of pixels from a plurality of images acquired by imaging by the imaging device. The image processing apparatus according to claim 1.   The image processing apparatus according to any one of claims 1 to 13, wherein pixel values of the plurality of pixels are acquired from one image obtained by imaging by the imaging device.   The second acquisition means acquires shape data representing the shape of the object, and calculates the normal line information based on the shape data. Image processing apparatus as described.   The image processing apparatus according to claim 16, wherein the shape data is data representing a three-dimensional shape of the object by a model using a polygon.   The image processing apparatus according to claim 16, wherein the shape data is data representing a three-dimensional shape of the object by a height distribution of a surface of the object.   The determining means determines the normal direction on the surface of the target area by the optimization with the normal direction represented by the normal information as an initial value, in addition to the reflection characteristic in the target area. An image processing apparatus according to any one of claims 2 to 18.   The program for functioning a computer as each means of the image processing apparatus as described in any one of Claims 1-19. An image processing method for determining a reflection characteristic of a target area of an object using an image obtained by an imaging device imaging an object illuminated using a light source,
A first acquisition step of acquiring, from the image, pixel values of a plurality of pixels whose geometric conditions determined by the positions of the light source, the imaging device, and the target area are different from each other;
A second acquisition step of acquiring normal information representing a normal direction on the surface of the target area;
At least two or more pixel values among the pixel values of the plurality of pixels, and the reflectance calculated based on the geometric condition and the normal direction corresponding to each of the two or more pixel values Determining, by processing, a reflection characteristic in the region of interest.
The image processing method according to claim 1, wherein the processing is performed according to a distance between a position of the target area corresponding to the plurality of pixels and a position where light from the light source specularly reflects the imaging device in the object.

Images