JP2016011874A - Image processor and method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate a reflection characteristic with high accuracy about a plurality of areas having different reflection characteristics of an object.SOLUTION: An imaging data input part 303 inputs a plurality of pieces of image data obtained by lighting a plurality of light sources arranged around the object by one at a time to image the object. A reflection characteristic acquisition part 304 acquires reflection characteristic data of the object from the plurality of pieces of image data obtained by imaging the object. A reliability calculation part 306 calculates the reliability of each of a plurality of pieces of reference data associated with the reflection characteristic of the object. A reflection characteristic interpolation processing part 307 performs interpolation processing of the reflection characteristic data on the basis of the plurality of pieces of reference data and their reliabilities.

Description

  The present invention relates to image processing for obtaining reflection characteristics of an object.

  As information that can be acquired from an object as a subject, there are various types of information such as the distance to the object, the shape of the object, and the reflection characteristics of the object in addition to the color. When acquiring the reflection characteristics, if the surface of the object is uneven, a shadow area may be generated, and acquisition of the reflection characteristics may fail. In order to prevent failure in obtaining the reflection characteristics, the number and arrangement of illuminations are adjusted so as to eliminate shadows on the surface of the object. However, when the object has a complicated uneven shape, adjustment work is difficult in practice.

  In view of this, various image processing techniques for interpolating the reflection characteristics of an area where acquisition failed have been proposed in preparation for the case where acquisition of reflection characteristics fails. For example, Patent Document 1 proposes a method of performing interpolation processing on reflection characteristics when estimating reflection characteristics that match pixel values of a captured image from an object shape estimated from the captured image and a known illumination field.

  According to the method of Patent Document 1, the reflection characteristics are estimated from the object shape and the illumination field according to the reliability of the estimated shape value, or the reflection characteristics are interpolated using the reflection characteristics estimated in the surrounding area where the shape reliability is high. Switch what to do. Therefore, by increasing the shape reliability of the peripheral area having the same shape as the shadow area, the reflection characteristic of the shadow area can be accurately calculated from the estimated value of the reflection characteristic of the peripheral area.

  However, according to the estimation process of Patent Document 1 corresponding to the reliability of the shape, the shadow area of the object surface has a uniform surface unevenness, and a plurality of paints having different colors are applied unevenly. In this case, it is not possible to detect a change in reflection characteristics due to a difference in paint. As a result, the same reflection characteristics are calculated for a plurality of areas having different reflection characteristics depending on the paint in the shadow area, and the estimation accuracy of the reflection characteristics decreases.

JP 2013-025762

G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, K. Toyama “Digital Photography with Flash and No-Flash Image Pairs” ACM Trans. On Graphics, vol. 23, August 2004

  An object of the present invention is to estimate the reflection characteristics of a plurality of regions having different reflection characteristics of an object with high accuracy.

  The present invention has the following configuration as one means for achieving the above object.

  In the image processing according to the present invention, a plurality of light sources arranged around the object are turned on one by one and a plurality of image data obtained by imaging the object are input, and the reflection characteristic data of the object is obtained from the plurality of image data. , The reliability of each of the plurality of reference data related to the reflection characteristic of the object is calculated, and the reflection characteristic data is interpolated based on the plurality of reference data and their reliability.

  According to the present invention, reflection characteristics can be estimated with high accuracy for a plurality of regions having different reflection characteristics of an object.

The figure explaining the acquisition method of the reflective characteristic of a to-be-photographed object. FIG. 11 is a block diagram illustrating a configuration example of an imaging device. The block diagram which shows the structural example of the image process part at the time of acquiring the reflective characteristic of a to-be-photographed object. 5 is a flowchart for explaining reflection characteristic acquisition processing by an image processing unit. The figure explaining reference data. The flowchart explaining the calculation process of reflection characteristics. The flowchart explaining the calculation process of reliability. The figure explaining the reliability obtained by the calculation process of reliability. The flowchart explaining the interpolation process of reflection characteristic data. The figure explaining the interpolation process of the reflection characteristic data which used color data for reference data. 9 is a flowchart for explaining reliability calculation processing according to the second embodiment. FIG. 10 is a diagram for explaining reliability obtained by reliability calculation processing according to the second embodiment. 10 is a flowchart for explaining reliability calculation processing according to the third embodiment. FIG. 10 is a diagram for explaining reliability obtained by reliability calculation processing according to the third embodiment.

  Hereinafter, an image processing apparatus and an image processing method according to embodiments of the present invention will be described in detail with reference to the drawings. In addition, an Example does not limit this invention concerning a claim, and all the combinations of the structure demonstrated in an Example are not necessarily essential for the solution means of this invention.

  As will be described in detail later, according to the present invention, a plurality of physical characteristic data such as a shape and a color that change the reflection characteristic of an object as a subject are acquired as reference data related to the reflection characteristic of the object. Then, the reliability of each reference data is calculated for each position in the shadow area where the reflection characteristics are interpolated, and the reference data used for the interpolation processing is adaptively switched at each position according to the reliability. Thereby, it is possible to calculate the reflection characteristics of an object having various surface irregularities and colors with high accuracy. Hereinafter, an object for which reflection characteristics are acquired is referred to as a “subject”.

[Obtaining reflection characteristics]
A method for obtaining the reflection characteristics of a subject will be described with reference to FIG. As shown in FIG. 1A, the imaging device 102 is arranged in the zenith direction as viewed from the subject 101. And at the polar angle θ of 30 degrees and 60 degrees with respect to the zenith direction, and at the position corresponding to the azimuth divided by 30 degrees with the right direction of the imaging device 102 as the reference azimuth (azimuth angle φ = 0 degree) A total of 24 lights (indicated by the symbol ●) will be placed.

  That is, a plurality of light sources for illumination are arranged around the subject 101, the light sources are sequentially turned on, and the subject 101 with unevenness is photographed by the imaging device 102 arranged at the zenith. FIG. 1 (b) shows a total of 24 images of the subject 101 that are shot with only one of the 24 lights on, and the reflection characteristics of the subject 101 are acquired using these 24 images. The

  The subject 101 is, for example, an oil painting of a person image. When observing the oil painting from the side, it is assumed that the body of the person rises from the background and the face of the person is drawn higher than the body. Further, it is assumed that a change in color or pattern in the subject 101 is small.

[Configuration of imaging device]
A configuration example of the imaging apparatus 102 will be described with reference to the block diagram of FIG. The CPU 201 uses the RAM 202 as a work memory, executes an OS and various programs stored in the ROM 203, and controls each component to be described later via the system bus 200.

  The imaging unit 220 includes an imaging optical system (lens, aperture, shutter), an optical low-pass filter, a color filter, and an imaging device such as a CMOS or CCD, and receives light from a subject by the imaging device. Then, imaging data (hereinafter referred to as RAW data) obtained by analog-digital (A / D) conversion of the analog signal output from the imaging device is output. The RAW data is stored in a predetermined area of the RAM 202 via the system bus 200 under the control of the CPU 201.

  The optical system control unit 221 controls the imaging optical system according to the control of the CPU 201, such as focusing, adjusting the aperture, and opening and closing the shutter. The flash control unit 222 controls the lighting of the 24 lights shown in FIG. The operation unit 223 corresponds to a cross key, a button, a mode dial, or the like, and is a user instruction input unit.

  The CPU 201 controls the computer graphics (CG) generation unit 204 and the display control unit 205 to display a user interface (UI) on the monitor 206 and receives a user instruction from the operation unit 223 or the like. Note that a touch screen may be superimposed on the monitor 206, and in this case, user input via the touch screen can be handled in the same way as input to the operation unit 223.

  A digital signal processing unit (DSP) 207 performs image processing by performing demosaicing processing, development processing, white balance processing, gamma processing, noise reduction processing, and the like on the RAW data stored in the RAM 202 under the control of the CPU 201. The image data is stored in a predetermined area of the RAM 202 under the control of the CPU 201.

  The compression / decompression unit 208 performs an encoding process for converting the image data stored in the RAM 202 into a format such as JPEG or MPEG according to the control of the CPU 201. Further, if necessary, processing for lossless compression of the RAW data stored in the RAM 202 is performed.

  The interface 209 is, for example, a memory card reader / writer, and stores image data output from the compression / decompression unit 208 in the medium 210 or image data read from the medium 210 in the RAM 202 under the control of the CPU 201.

  The interface 209 may be a general-purpose interface such as USB, and the medium 210 may be a recording medium such as a hard disk drive (HDD), a solid state drive (SDD), or a USB memory. Further, a wireless network interface can be used as the interface 209, and examples of data transmission / reception destinations include computer devices such as personal computers, smartphones, tablet computers, and server devices. Further, the CPU 201 can receive a user instruction via the interface 209.

  The image processing unit 211 generates new image data using RAW data or image data stored in the RAM 202 according to the control of the CPU 201. The generated image data is stored in the RAM 202 or the medium 206, or displayed on the monitor 206 via the display control unit 204. The image displayed on the monitor 206 is a live view image or a captured image.

[Image processing unit]
An example of the configuration of the image processing unit 211 when acquiring the reflection characteristics of the subject 101 is shown in the block diagram of FIG. The imaging data input unit 303 inputs n pieces of image data (24 pieces in the example of FIG. 1B) imaged by the imaging unit 220 from the RAM 202, for example. The reflection characteristic acquisition unit 304 acquires reflection characteristic data of the subject 101 from the input image data.

  The reference data acquisition unit 305 acquires reference data used when interpolating reflection characteristic data from the medium 210, for example. Reference data will be described later. The reliability calculation unit 306 calculates the reliability of the acquired reference data.

  The reflection characteristic interpolation processing unit 307 receives the reflection characteristic data acquired by the reflection characteristic acquisition unit 304 and the reliability of the reference data calculated by the reliability calculation unit 306, and interpolates the reflection characteristic data based on the reliability of the reference data. To process. The reflection characteristic data subjected to the interpolation processing is stored in the RAM 202 or the medium 210, for example.

Reflection Property Acquisition Processing Reflection property acquisition processing by the image processing unit 211 will be described with reference to the flowchart of FIG. The reflection characteristic acquisition process is executed by the CPU 201 controlling the image processing unit 211 in accordance with a user instruction. Note that it is also possible to execute reflection characteristic acquisition processing by supplying a computer that implements the function of the image processing unit 211 shown in FIG. 3 to a computer device.

  The imaging data input unit 303 inputs n (24 in the example of FIG. 1B) image data captured by the imaging unit 220 (S401). In the following, for the sake of simplicity, the input image data will be described as grayscale image data output from the RGB image data output by the imaging unit 220. Subsequently, the reflection characteristic acquisition unit 304 acquires reflection characteristic data of the subject 101 from the n pieces of image data (S402). Details of the reflection characteristic data acquisition process will be described later.

  On the other hand, the reference data acquisition unit 305 acquires reference data according to the user instruction (S403). Subsequently, the reliability calculation unit 306 calculates the reliability of the reference data (S404). Details of the reference data reliability calculation processing will be described later. FIG. 4 shows that the acquisition of the reference data and the calculation of the reliability of the reference data are performed following the acquisition of the image data and the calculation of the reflection characteristic data, but this order may be reversed or in parallel. It may be done.

  Next, the reflection characteristic interpolation processing unit 307 interpolates the reflection characteristic data based on the reliability of the reference data (S405), and outputs the reflection characteristic data subjected to the interpolation processing (S406). Details of the reflection characteristic data interpolation processing will be described later.

Reference data Reference data will be described with reference to FIG. FIG. 5A shows the shape data of the subject 101 as one reference data, and FIG. 5B shows the color data of the subject 101 as the other reference data. These reference data are measured in advance and stored in the medium 210, for example. The shape data is measured by irradiating the subject 101 with slit light by a light cutting type shape measuring device. The pixel position of the shape data is aligned with the pixel position of the image data imaged by the imaging unit 220, and each pixel of the shape data is a gray scale value indicating the height from the floor surface in the imaging system shown in FIG. Have

  Also, the color data is image data captured by the imaging unit 220 with all the illuminations turned on, with all the illuminations 1, illuminations 2,... It is. Of course, the pixel position of the color data as the reference data and the pixel position of the image data for calculating the reflection characteristics are aligned. For the sake of simplicity, the color data is described as image data obtained by converting the RGB image data output from the imaging unit 220 into a gray scale, similarly to the image data for calculating the reflection characteristics.

Reflection Characteristic Calculation Unit The reflection characteristic calculation process (S402) will be described with reference to the flowchart of FIG. The reflection characteristic is calculated by comparing n images and considering the maximum pixel value at each pixel position (u, v) as the reflected light intensity or the reflected light intensity in the direction close to the regular reflection direction. . Further, the pixel value at the pixel position (u, v) of the image j is expressed as Ij (u, v), and the size of each image is U × V pixels.

  The reflection characteristic acquisition unit 304 first performs initial setting. That is, the pixel value I1 (u, v) of the image 1 is stored in the bitmap memory Imax that stores the maximum pixel value (S501), the counter j is set to j = 2 (S502), and the pixel position (u, v ) Is set to an initial value (0, 0) (S503).

  Next, the reflection characteristic acquisition unit 304 acquires the pixel value Ij (u, v) from the image data of the image j (S504). Then, the pixel value Ij (u, v) is compared with the value Imax (u, v) stored at the position of the bitmap memory corresponding to the pixel position (u, v) (S505). When Ij (u, v)> Imax (u, v), the reflection characteristic acquisition unit 304 determines that Ij (u, v) is a regular reflection light intensity or a reflection light intensity in a direction close to the regular reflection direction, Imax (u, v) is updated by Ij (u, v) (S506).

  Next, the reflection characteristic acquisition unit 304 increments u of the pixel position (S507), and compares u after the increment with U of the image size (S508). If u ≦ U, the process returns to step S504 to make a determination regarding the next pixel position. If u> U, u is reset (u = 0), v at the pixel position is incremented (S509), and v after the increment is compared with V at the image size (S510). If v ≦ V, the process returns to step S504 to make a determination regarding the next pixel position.

  When v> V, the reflection characteristic acquisition unit 304 resets v (v = 0), increments the counter j (S511), and compares the incremented count value j with the number of images n (S512). The reflection characteristic acquisition unit 304 returns the process to step S504 if j ≦ n and starts the determination on the next image j, and ends the acquisition of the reflection characteristic if j> n.

  Through the above processing, reflection characteristic data indicating the intensity of specular reflection light of the subject 101 extracted from n images is stored in the bitmap memory Imax. However, when the subject 101 has irregularities such as an oil painting, for example, there is a possibility that there is a region (for example, a shaded region shown in FIG. 5C) where acquisition of reflection characteristics has failed.

Reliability calculation unit The reliability calculation processing (S404) will be described with reference to the flowchart of FIG. The reliability calculation unit 306 sets an initial value (0, 0) to the pixel position (u, v) (S701), and calculates the reliability Wk (u, v) of each reference data at the pixel position (u, v). Calculate (S702).

The reliability Wk (u, v) is calculated by the following formula, at the pixel position (u, v), the standard deviation σref () of the pixel value in the filter region of P × P pixels centered at (u, v) This is done by calculating u, v). In addition, when the pixel value does not exist in a part of the filter area, for example, when the pixel position is (0, 0), the part is excluded from the calculation target of the standard deviation σref (u, v).
I ′ = Σ _{(x ′, y ′) ∈T} {I (x ′, y ′) / P ² };
σref (u, v) = √ [Σ _{(x, y) ∈T} {I (x, y)-I '} ² / P ² ];… (1)
Where T is the filter region,
I (x, y) and I (x ′, y ′) are pixel values in the filter area.

Next, the reliability calculation unit 306 normalizes the reliability Wk (u, v) (= σref (u, v)) of each reference data at the pixel position (u, v) by the following equation (S703). In other words, the normalization is performed by dividing the reliability Wk (u, v) at the pixel position (u, v) of each reference data by the sum of the reliability of the two reference data according to the equation (2).
W'k (u, v) = Wk (u, v) / Σ _k∈D Wk (u, v)… (2)
Here, D is structure data having a subscript representing each reference data type as an element,
k is the type of reference data (shape s, color c).

  Next, the reliability calculation unit 306 increments u of the pixel position (S704), and compares u after the increment with U of the image size (S705). If u ≦ U, the process returns to step S702 to calculate the reliability of the next pixel position. If u> U, u is reset (u = 0) (S706), v at the pixel position is incremented (S707), and v after the increment is compared with V at the image size (S708). If v ≦ V, the process returns to step S702 to calculate the reliability of the next pixel position. If v> V, the calculation of the reliability is terminated.

  The reliability obtained by the reliability calculation process will be described with reference to FIG. FIG. 8 (a) shows the reliability of the shape data in the oil painting having a large undulation obtained by the above processing, and the reliability increases in the order of the human face region, body region, and background region (in FIG. 8). The darker the area, the higher the reliability.) FIG. 8B shows the reliability of the color data obtained by the above processing. The reliability increases in the order of the background region, the human body region, and the face region where the color and pattern change is large. FIG. 8C shows the magnitude relationship between the reliability Ws (u, v) of the shape data due to the undulation of the shape and the reliability Wc (u, v) of the color data due to the change in color and pattern.

Reflection Characteristic Interpolation Processing Unit The reflection characteristic data interpolation process (S405) will be described with reference to the flowchart of FIG. Hereinafter, an example in which interpolation processing is performed using a joint bilateral filter will be described, but details of the joint bilateral filter are described in Non-Patent Document 1, and the like.

The reflection characteristic interpolation processing unit 307 sets an initial value (0, 0) to the pixel position (u, v) (S801), and interpolates the reflection characteristic data I (u, v) at the pixel position (u, v). The reflected characteristic data I _JBF (u, v) is generated (S802). The following equation shows a joint bilateral filter, and reflection characteristic data I (u, v) is interpolated using each reference data. That is, Is _JBF (u, v) interpolated using shape data as reference data and Ic _JBF (u, v) interpolated using color data as reference data are generated.
Ik _JBF (u, v) = [Σ _{(x, y) ∈S} G _σs {|| (u, v)-(x, y) ||} G _σr {| Ik (u, v) -Ik (x , y) |} I (x, y)] / N (u, v)… (3)

  With reference to FIG. 10, the reflection characteristic data interpolation process (the process of equation (3)) using color data as reference data will be described. FIG. 10 (a) shows the reflection characteristic data I (x, y). The pixel value of the reflection characteristic data I (x, y) corresponding to the upper half where the regular reflection light intensity of the subject is large is, for example, 255, and the pixel of the reflection characteristic data I (x, y) corresponding to the lower half where the regular reflection intensity is small The value is 128, for example. However, the area indicated by hatching in the center is a missing area where reflection characteristics have failed to be acquired due to the unevenness of the subject, and the pixel value of the missing area is 0, for example.

  FIG. 10B shows the color data Ic (x, y) of the reference data. A gray paint is applied to the upper half of the subject, and a black paint is applied to the lower half. The color data Ic (x, y) follows the paint distribution, the pixel value of the color data Ic (x, y) corresponding to the upper half is, for example, 128, and the pixel of the color data Ic (x, y) corresponding to the lower half The value is 0, for example.

  The pixel value I (u, v) at the pixel position (u, v) in the missing region is subjected to interpolation processing based on the color data Ic (x, y) and the filter operation shown in Expression (3). In Expression (3), S is a window region of the filter, and is an F × F pixel region centered on the pixel position (u, v) (region surrounded by a broken line in FIG. 10 (a)).

G _σs is a weighting coefficient in the spatial direction, and is represented by a Gaussian with a maximum value of 1 and a half-value half-width σs centered on the pixel position (u, v) (FIG. 10 (c)). G _σr is a weighting coefficient in the luminance direction, and follows a Gaussian with a maximum value of 1 and a half-value half-width σr centered on the pixel value Ic (u, v) at the pixel position (u, v) of the color data.

Therefore, the pixel value in the upper half of the color data is equal to Ic (u, v) and the weighting coefficient is 1, and the pixel value in the lower half of the color data is different from Ic (u, v) and the weighting coefficient is smaller than 1. (Fig. 10 (d)). For each pixel location in the window region S of the filter, using the weighting coefficients obtained by multiplying the G _{[sigma] s} and G _.sigma.r (Fig 10 (e)), an interpolation process reflection characteristic data. N (u, v) is a normalization coefficient that makes the sum of the weighting coefficients in the window region S of the filter 1.

  By performing the filter operation shown in Equation (3) for each pixel position (u, v), the reflection characteristic data of the missing region is changed to the pixel value 255 on the upper side and the pixel value 128 on the lower side according to the paint distribution of the color data. Interpolation is performed (FIG. 10 (f)).

Next, the reflection characteristic interpolation processing unit 307 performs reflection characteristic data Is _JBF (u, v) and Ic _JBF (u, v) generated by the interpolation process based on the reliability W′k (u, v) of each reference data. ) Is synthesized (S803). In other words, the reflection characteristic data Ik _JBF (u, v) is synthesized by the following formula using the reliability W'k (u, v) as a weight, and the combined reflection characteristic data is reflected at the pixel position (u, v). Output as I (u, v).
I (u, v) = W's (u, v) × Is _JBF (u, v) + W'c (u, v) × Ic _JBF (u, v)… (4)

  Next, the reflection characteristic interpolation processing unit 307 increments u of the pixel position (S804), and compares u after the increment with U of the image size (S805). If u ≦ U, the process returns to step S802 to interpolate the reflection characteristic data at the next pixel position. If u> U, u is reset (u = 0) (S806), v at the pixel position is incremented (S807), and v after the increment is compared with V of the image size (S808). If v ≦ V, the process returns to step S802 to interpolate the reflection characteristic data at the next pixel position. If v> V, the interpolation process ends.

  By interpolating the reflection characteristic data based on the reliability of each reference data, as shown in Fig. 5 (d), the reflection characteristic data of a human face with a large shape undulation and a small change in color and pattern is the shape data. Are interpolated. On the other hand, the reflection characteristic data of the background having a small shape undulation and a large change in color and pattern is subjected to interpolation processing based on the color data, and the missing area is eliminated.

  Thus, different reflections such as a human face area with a large shape undulation and a small color and pattern change, a background area with a small shape undulation and a large color and pattern change, and a person body area in between. The reflection characteristics can be estimated with high accuracy for a plurality of regions having characteristics.

[Modification]
In the above description, the reflection characteristic data is acquired from 24 images captured by lighting only one of the 24 lights. However, the illumination method, the number of lights, and the number of images are not limited. The number of illuminations and the number of images can be increased or decreased according to the shape and size of the subject and the reflection characteristics of the subject.

  In addition, the example in which the maximum pixel value in the 24 images is determined as the reflected light intensity data from the regular reflection direction or the direction close to the regular reflection direction and the reflection characteristic data is obtained has been described. The acquisition method is not limited to this. For example, the variable angle characteristic data with respect to the reflection direction of the reflected light may be acquired as the reflection characteristic data.

  For simplicity, gray scale image data and reflection characteristic data have been described. However, RGB image data or multiband image data may be used, and the number of components and the number of bands of image data are not limited.

  In addition, the example in which the image data obtained by converting the RGB image data to gray scale is used as the color data of the reference data has been described. However, any image data captured using an optical filter having wavelength selectivity may be used. There is no limit to the number of bands. That is, RGB image data or multiband image data can also be used as color data.

  Moreover, although the example which uses a joint bilateral filter for the interpolation process was demonstrated, it should just be the interpolation process using several reference data, and there is no limitation in the method of an interpolation process. Moreover, although the example which uses height data as shape data was demonstrated, a normal map may be used and there is no limitation in the format of shape data.

  Moreover, although the example using the shape data measured by the light measuring method shape measuring device has been described, the shape data may be calculated from the image data. For example, the reference data acquisition unit 305 estimates the surface normal of the surface of the subject by illuminance difference stereo method from the image data used to acquire the reflection characteristic data, and integrates the surface normal in the spatial direction, thereby Shape data can be calculated.

  In addition, although an example has been described in which image data obtained by attaching all the illuminations and attaching a polarization filter to each illumination and camera in order to remove specularly reflected light is used as color data, color data is calculated from the image data. Also good. For example, the reference data acquisition unit 305 inputs image data used for acquisition of reflection characteristic data, compares the image data of image 1, image 2,..., Image 24, and compares each pixel position (u, v). The minimum pixel value can be used as subject color data.

  Moreover, although the example which calculates the reliability of reference data by the filter calculation process of PxP pixel which calculates a standard deviation was demonstrated, the area | region which calculates reliability is not limited. For example, the reliability of the entire reference data may be calculated using all the pixels of each reference data, and the switching process using only the reference data having a high reliability may be performed. Further, although an example in which the standard deviation is used as the reliability has been described, the reliability may be calculated from the spatial frequency or noise amount of the reference data, and the reliability calculation method is not limited.

  Further, although the weighting coefficient is calculated according to the reliability of the plurality of reference data, if all the reliability of the plurality of reference data deviates from the predetermined range, interpolation processing may be performed using preset reference data. Good. In addition, when the reliability of a plurality of reference data deviates from a predetermined range, interpolation processing may be performed using surrounding reflection characteristic data without using reference data, or through without performing interpolation processing. Also good.

  Hereinafter, an image processing apparatus and an image processing method according to a second embodiment of the present invention will be described. Note that the same reference numerals in the second embodiment denote the same parts as in the first embodiment, and a detailed description thereof will be omitted.

  In the second embodiment, a method for highly accurately interpolating the reflection characteristic data will be described even when the undulation of the shape of the subject is large and occlusion occurs to reduce the measurement accuracy of the shape data. The image processing other than the reference data reliability calculation processing in the second embodiment is the same as the image processing in the first embodiment, and only the reference data reliability calculation processing in the second embodiment will be described below.

  The reliability calculation process (S404) of the second embodiment will be described with reference to the flowchart of FIG. In FIG. 11, steps denoted by the same reference numerals as in FIG. 7 are steps for performing the same processing as in FIG.

After calculating (S702) the reliability Wk (u, v) of each reference data at the pixel position (u, v), the reliability calculation unit 306 determines the reliability Ws (u of the shape data by the determination in step S1301. , v), the following processing is executed. First, an occlusion map O (u, v) is generated using the shape data for which the reliability Ws (u, v) is calculated (S1302), and the occlusion map is obtained from Ws (u, v) and O (u, v). The reliability Wos (u, v) is corrected with a weight according to (S1303). The reliability calculation unit 306 uses an SSAO (screen space ambient occlusion) algorithm to generate an occlusion map, and corrects the reliability Wos (u, v) using the following equation.
if (O (u, v) <Oth)
Wos (u, v) = Ws (u, v);
else
Wos (u, v) = 0 ；… (5)
Here, Oth is a predetermined threshold value.

  Next, the reliability calculation unit 306 performs the normalization process in step S703 using the reliability Wos (u, v) and the reliability Wc (u, v) instead of the reliability Ws (u, v). The subsequent processing is the same as in FIG.

  The reliability obtained by the reliability calculation process according to the second embodiment will be described with reference to FIG. FIG. 12 (a) shows the reliability W's (u, v) of the shape data, and the reliability of the area where the occlusion is large in the subject (for example, human eyes, mouth, body contour) is 0 (minimum value, figure (Indicated in white in 12).

  On the other hand, as the pixel position where Wos (u, v) = 0 is increased by the processing of Equation (5), the denominator value of Equation (2) becomes smaller, and the colors of the eyes, mouth, and body of the person are relatively The data reliability W′c (u, v) is increased (FIG. 12 (b)). Therefore, by correcting the reliability according to the occlusion of the shape data, even when the undulation of the shape of the subject is large and the occlusion occurs, the reflection characteristic data can be interpolated with high accuracy using the color data.

  Hereinafter, an image processing apparatus and an image processing method of Example 3 according to the present invention will be described. Note that the same reference numerals in the third embodiment denote the same parts as in the first and second embodiments, and a detailed description thereof will be omitted.

  In the third embodiment, a method for highly accurately interpolating the reflection characteristic data will be described even when the color of the subject is close to the illumination color and the separation accuracy between the color data of the reference data and the reflection characteristic data is reduced. In the first embodiment, for simplicity, the reflection characteristic data and the color data are described as gray scale image data. In the third embodiment, description will be made assuming that reflection characteristic data and color data are RGB image data I (u, v) and Irc (u, v) (I is R, G, or B). Depending on the subject, a bronze phenomenon may occur in which the color of reflected light on the subject surface looks different from the illumination color. In the third embodiment, for simplicity, it is assumed that the bronze phenomenon does not occur and the hue of the reflection characteristic data matches the hue of the illumination color at the time of imaging.

  The image processing other than the reference data reliability calculation processing in the third embodiment is the same as the image processing in the first embodiment, and only the reference data reliability calculation processing in the third embodiment will be described below. The reliability calculation process (S404) of the third embodiment will be described with reference to the flowchart of FIG. In FIG. 13, steps denoted by the same reference numerals as in FIG. 7 are steps for performing the same processing as in FIG.

After calculating (S702) the reliability Wk (u, v) of each reference data at the pixel position (u, v), the reliability calculation unit 306 determines the reliability Wc (u of the color data based on the determination in step S1501. , v), the following processing is executed. First, the hue map Hp (u, v) is calculated using the reflection characteristic data I (u, v) acquired in step S402 and the color data Irc (u, v) for which the reliability Wc (u, v) is to be calculated. (S1502).
Hp (u, v) = tan ^-1 [√3 {Gp (u, v)-Bp (u, v)} / {2Rp (u, v)-Gp (u, v)-Bp (u, v) }]… (6)
Where (Rp, Gp, Bp) (u, v) constitutes I (u, v) or Irc (u, v)
p is a subscript indicating reflection characteristic data or color data.

Next, the reliability calculation unit 306 calculates a hue difference map ΔH (u, v) from the hue map by the following equation (S1503).
ΔH (u, v) = | H (u, v)-Hc (u, v) |… (7)
Here, H (u, v) is a hue map calculated from reflection characteristic data I (u, v),
Hc (u, v) is a hue map calculated from the color data Irc (u, v).

The hue map H (u, v) calculated from the reflection characteristic data I (u, v) corresponds to the illumination color that illuminates the subject when the image data used to acquire the reflection characteristic data is captured. Next, the reliability calculation unit 306 determines the reliability of the color data from the reliability Wc (u, v) of the color data and the hue difference map ΔH (u, v) according to the difference between the hues of the subject color and the illumination color. Is corrected by the following equation (S1504).
if (ΔH (u, v) <Hth)
Whc (u, v) = 0;
else
Whc (u, v) = Wc (u, v);… (8)
Here, Hth is a predetermined threshold value.

  Next, the reliability calculation unit 306 performs normalization processing in step S703 using the reliability Whc (u, v) and the reliability Ws (u, v) instead of the reliability Wc (u, v). The subsequent processing is the same as in FIG.

  The reliability obtained by the reliability calculation process according to the third embodiment will be described with reference to FIG. FIG. 14 (b) shows the reliability W′c (u, v) of the color data, and the reliability of the background area other than the foliate pattern of the background with a small hue difference and the human face area is 0 (minimum value, (Shown in white in FIG. 14).

  On the other hand, as the pixel position where Whc (u, v) = 0 is increased by the processing of Equation (8), the value of the denominator of Equation (2) becomes smaller, and the background region other than the background pattern is relatively And the reliability W's (u, v) of the shape data of the human face area is increased (FIG. 14 (a)). Therefore, by correcting the reliability according to the hue difference between the color data and the illumination color, even if the subject color is close to the illumination color, the reflection characteristic data can be interpolated with high accuracy using the shape data. become.

[Modification]
In the second embodiment, as an example of the reliability correction, when the occlusion map exceeds a predetermined threshold, the reliability of the shape data is set to the lowest value. In the third embodiment, as an example of the reliability correction, when the hue difference is less than a predetermined threshold, the reliability of the color data is set to the lowest value. Not limited to this, the weight w (0 ≦ w ≦ 1) is obtained using an occlusion map or a table or calculation formula showing the correspondence between the hue difference and the weight w, and the weight w and the reliability Wk (u, v) are obtained. Correction for multiplication may be performed. Needless to say, the combinations of Embodiments 2 and 3 are possible.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  303 ... Imaging data input unit, 304 ... Reflection characteristic acquisition unit, 306 ... Reliability calculation unit, 307 ... Reflection characteristic interpolation processing unit

Claims (13)

An input means for inputting a plurality of image data obtained by imaging the object by turning on a plurality of light sources arranged around the object one by one;
Obtaining means for obtaining reflection characteristic data of the object from the plurality of image data;
Calculating means for calculating the reliability of each of a plurality of reference data related to the reflection characteristics of the object;
An image processing apparatus comprising: interpolation processing means for interpolating the reflection characteristic data based on the plurality of reference data and their reliability.   2. The image processing apparatus according to claim 1, wherein the acquisition unit generates, as the reflection characteristic data, image data obtained by extracting a maximum pixel value from the plurality of image data for each pixel position.   3. The image processing apparatus according to claim 1, wherein the calculation unit calculates, as the reliability, a standard deviation of a pixel value of a predetermined area centered on the pixel position for each pixel position. The interpolation processing means includes
First reflection characteristic data obtained by interpolating the reflection characteristic data using one of the plurality of reference data, and second reflection characteristic data obtained by interpolating the reflection characteristic data using the other of the plurality of reference data. Produces
4. The reflection characteristic data subjected to the interpolation process is generated by combining the first and second reflection characteristic data based on the reliability of each of the plurality of reference data. An image processing apparatus according to one item.   5. The image processing apparatus according to claim 1, wherein the plurality of reference data are shape data indicating a shape of the object and color data indicating a color of the object.   6. The image processing apparatus according to claim 5, further comprising means for acquiring the shape data and / or the color data from the plurality of image data.   7. The image processing apparatus according to claim 5, wherein the calculation unit corrects the reliability of the shape data based on an occlusion map generated from the shape data.   8. The image processing apparatus according to claim 7, wherein the calculation unit corrects the reliability of shape data in which the occlusion map exceeds a predetermined first threshold value to a minimum value.   9. The calculation unit according to claim 5, wherein the calculation unit corrects the reliability of the color data based on the illumination color of the object and the color data when the plurality of image data is captured. Image processing device.   10. The image processing apparatus according to claim 9, wherein the calculation unit corrects the reliability of color data in which a difference between the hue of the illumination color and the hue of the color data is less than a predetermined second threshold value to a minimum value. A plurality of light sources arranged around the object are turned on one by one to input a plurality of image data obtained by imaging the object,
Obtaining reflection characteristic data of the object from the plurality of image data;
Calculating the reliability of each of a plurality of reference data related to the reflection characteristics of the object;
An image processing method for interpolating the reflection characteristic data based on the plurality of reference data and their reliability.   11. A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.   A computer-readable recording medium on which the program according to claim 12 is recorded.