JP2014039126A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate appropriate texture data even under a light source not sufficiently including a high-frequency component.SOLUTION: When light source information includes a high-frequency component, it is determined to estimate texture data of a subject from image data captured using a flash. Then, texture data indicating reflectance distribution of the subject is estimated in accordance with the determination. The texture data is estimated using image data acquired by image data acquisition means, shape data acquired by shape data acquisition means, and light source data acquired by light source data acquisition means.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for estimating texture data.

  In order to give an effect according to the photographer's intention to the captured image, there is a case where processing by image processing software or the like is performed. For example, processing such as changing the direction and brightness of the light source or synthesizing a CG subject in the scene. Here, in the case where information on the shape, texture, and light source of the subject is unknown, in order to create a natural processed image by the above processing, it is necessary for an expert to perform processing over time. However, if the information on the shape, texture, and light source is known, the processing can be automated by the CG rendering technique (Patent Document 1).

  Many techniques for estimating information on the shape, texture, and light source necessary for the automation of the processing have been proposed. For example, Patent Document 2 discloses a technique for estimating a subject shape by generating a patch on a subject surface so as to be consistent with an image captured from multiple viewpoints. Non-Patent Document 1 discloses a technique for simultaneously estimating the shape, texture, and light source by performing optimization calculation so that the pixel values of the multi-viewpoint image and the reflection luminance calculated from the rendering equation are consistent. It is disclosed.

Japanese Patent No. 4078926 US Patent Application Publication No. 2009/0052796

"Incorporating the Torrance and Sparrow Model of Reflectance in Uncalibrated Photometric Stereo", A. Georghiades, ICCV, pp.816-823 (2003) "signalprocessing framework for inverse rendering", Ravi Ramamoorthi, et al, Proc. ACM SIGGRAPH 2001, pp.117-128 (2001)

  However, it is known that the texture of a subject cannot be estimated under a light source that does not sufficiently contain high-frequency components (for example, an environment where there is no locally high light source such as a cloudy sky or a shadow) (non-patent document). Reference 2). If the above automatic processing is performed based on the wrong texture, an unnatural processed image that does not reflect the actual texture of the subject is generated.

  Therefore, an object of the present invention is to estimate appropriate texture data even under a light source that does not sufficiently contain high-frequency components.

  An image processing apparatus according to the present invention includes: a determination unit that determines based on light source information whether to estimate subject texture data from image data captured using a flash; and the subject texture according to the determination by the determination unit It is characterized by comprising texture data estimation means for estimating data.

  The present invention can obtain appropriate texture data even under a light source that does not sufficiently contain high-frequency components.

FIG. 3 is a block diagram illustrating an example of each processing unit of the image processing apparatus according to the first embodiment. 3 is a flowchart illustrating an example of an operation of the image processing apparatus according to the first exemplary embodiment. FIG. 3 is a diagram illustrating an example of a “direction” that can be expressed by a polar angle and an azimuth angle in the first embodiment. FIG. 3 is a diagram schematically illustrating an example of an imaging unit according to the first embodiment. 6 is a block diagram illustrating an example of each processing unit of a scene element data acquisition unit in Embodiment 1. FIG. It is a figure which shows an example of the relationship between the incident polar angle in Example 1, and incident luminance. It is a figure which shows an example of the relationship between the frequency order in Example 1, and an incident luminance frequency coefficient. FIG. 3 is a block diagram illustrating an example of each processing unit of a flash necessity determination unit according to the first embodiment. 6 is a flowchart illustrating an example of an operation of a flash necessity determination unit 112 according to the first embodiment. It is a figure which shows an example of the relationship between the frequency order in Example 1, and the threshold value of a frequency order. It is a figure which shows an example of the relationship between the imaging mode in Example 1, and a light source frequency. 10 is a flowchart illustrating an example of an operation of the image processing apparatus according to the second exemplary embodiment. FIG. 10 is a block diagram illustrating an example of each processing unit of a scene element data acquisition unit according to a second embodiment. 10 is a flowchart illustrating an example of an operation of a texture / light source data acquisition unit according to the second embodiment. 12 is a flowchart illustrating an example of a flow of texture / light source data estimation processing using a difference image in the second embodiment.

  Embodiments will be described below with reference to the drawings.

<Device configuration>
FIG. 1 is a block diagram illustrating an example of an image processing apparatus according to the first embodiment. The imaging units 101 and 102 include a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, a color filter, a sensor such as a CMOS or CCD, a flash, and the like. Detect. Here, FIG. 1 shows an example in which there are two imaging units, but the present invention is not limited to the case where there are two imaging units, and can be applied to an image processing apparatus having one or more arbitrary number of imaging units. The A / D conversion unit 103 converts the light amount of the subject into a digital value. The signal processing unit 104 performs white balance processing, gamma processing, noise reduction processing, and the like on the digital value to generate digital image data. The D / A conversion unit 105 performs analog conversion on the digital image data. The scene element data acquisition unit 106 acquires and estimates scene element data such as shape, light source, and texture. The encoder unit 107 performs processing for converting the digital image into a file format such as Jpeg or Mpeg. The media interface 108 is an interface for connecting to a PC or other media (for example, hard disk, memory card, CF card, SD card, USB memory). The media interface 108 is connected to a communication network such as the Internet, and transmits and receives data as necessary. The CPU 109 is involved in all the processes of each configuration, reads the instructions stored in the ROM 110 and the RAM 111 in order, interprets them, and executes the processes according to the results. The ROM 110 and the RAM 111 provide the CPU 109 with programs, data, work areas, and the like necessary for the processing. The flash necessity determination unit 112 determines whether it is necessary to throw a flash at the time of imaging in order for the scene element data acquisition unit 106 to estimate the texture data. The imaging system control unit 113 performs control of the imaging system instructed by the CPU 109 such as focusing, opening a shutter, adjusting an aperture, and blowing a flash. The operation unit 114 corresponds to a button, a mode dial, or the like, and receives a user instruction input via these buttons. The character generation 115 generates characters and graphics. Generally, a liquid crystal display is widely used as the display unit 116, and displays a captured image and characters received from the D / A conversion unit 105 and the character generation 115. Further, it may have a touch screen function, and in that case, a user instruction can be handled as an input of the operation unit 114. Although there are other components of the apparatus than the above, the description thereof is omitted because it is not the main point of the present embodiment.

<Flowchart of Example 1>
FIG. 2 is a flowchart showing an operation in the first embodiment of the image processing apparatus shown in FIG. The flowchart shown in FIG. 2 is executed when the CPU 109 reads and interprets a command stored in the ROM 110 or the RAM 111, for example. Hereinafter, the operation of the image processing apparatus according to the first embodiment will be described with reference to FIG.

  Step S201 is light source information acquisition processing, in which the scene element data acquisition unit 106 acquires light source information of an imaging scene. The light source information is information used to determine whether or not it is necessary to throw a flash as will be described later. The light source information includes light source data. As an example of the light source data, there is data relating to illumination information such as the position, direction, and luminance of the light source. In the following description, it is assumed that the light source data is represented by a combination of a “direction” that can be expressed by a polar angle and an azimuth angle as shown in FIG. 3 and an “incident luminance” that enters the subject from each direction. It should be noted that there are various methods for expressing the light source data, and it is needless to say that the method is not limited to the example shown in FIG. In the first embodiment, any method may be used for obtaining light source data. Since the light source data acquisition method is not the main focus of this embodiment, detailed description is omitted, but the light source data is acquired from image data obtained by imaging a specular sphere or image data obtained by imaging the surroundings of a scene using a fisheye lens. The method is known.

  In step S202, the flash necessity determination unit 112 determines whether or not it is necessary to throw a flash during imaging based on the light source information acquired in step S201. Details of the operation of the flash necessity determination unit 112 will be described later.

  In step S203, the imaging units 101 and 102 perform imaging with flash or imaging without flash according to the determination result of the flash necessity determination unit 112. Here, in order to improve the accuracy of the estimated texture data, it is desirable that the imaging units 101 and 102 capture the subject from multiple viewpoints. As such an image pickup unit, a multi-lens camera as shown in FIG. 4A, a plurality of cameras as shown in FIG. 4B, or a video that can be picked up while changing the image pickup position as shown in FIG. 4C. There are cameras. In the case of FIG. 4C, an image obtained by extracting a plurality of frames of a moving image captured by a video camera is multi-viewpoint image data. In addition, since a rough texture can be acquired even in a captured image from one viewpoint, the present embodiment is not limited to an imaging unit that can perform multi-viewpoint imaging.

  In step S204, the scene element data acquisition unit 106 acquires subject shape data. For shape data, external data input may be used.For example, shape data is calculated and acquired by generating patches on the subject surface so as to be consistent with images captured from multiple viewpoints, for example. May be.

  Step S205 is a texture data estimation process, in which the scene element data acquisition unit 106 estimates the texture data of the subject. Here, the texture data refers to the reflectance distribution of the subject. As a model representing the reflectance distribution, for example, there is a BRDF (Bidirectional Reflectance Distribution Function) represented by the following equation.

Here, θ _i is the incident polar angle, φ _i is the incident azimuth angle, θ _r is the reflected polar angle, φ _r is the reflected azimuth angle, L is the incident luminance, and R is the reflected luminance. It is an angle formed by the “normal direction” included. In the following description, it is assumed that the texture data is represented by the BRDF model, but the present embodiment is not limited to this. It can be applied to any reflection model such as BSSRDF (Bidirectional Scattering Surface Reflectance Distribution Function).

<Details of Scene Element Data Acquisition Unit 106>
FIG. 5 is a block diagram illustrating each processing unit of the scene element data acquisition unit 106. The texture data estimation process in step S205 of FIG. 4 will be described with reference to FIG. In the first embodiment, the texture data acquisition unit 504 uses the captured image data acquired by the captured image data acquisition unit 501, the shape data acquired by the shape data acquisition unit 502, and the light source data acquired through the light source data acquisition unit 503. Estimate the texture data. Details of the texture data estimation process will be described below.

First, the relationship between the shape, texture, light source, and pixel value of the captured image will be described. The pixel value I of the pixel that receives the reflected light with the reflection angle (θ _r , φ _r ) is calculated from the following equation.

Here, α is a conversion coefficient from luminance to pixel value. This conversion coefficient α can be obtained by measurement or the like. Specifically, a light source that has been measured with a luminance meter is imaged, and α may be calculated from the relationship between the pixel value of the captured image and the luminance measured with the luminance meter.

Next, a texture estimation method will be described. In this embodiment, since the light source data is acquired by the light source data acquisition unit 503 through the media interface 108, the incident luminance L (light source data) in the expressions 2 and 3 is the incident polar angle θ _i and the incident angle. The azimuth angle φ _i is known. Further, the pixel value I is known by the number of pixels of the imaging units 101 and 102. The basic method for estimating the texture data is to calculate BRDF that optimizes the matching between the incident luminance L and the pixel value I by optimization calculation. However, it is practically difficult to calculate BRDF with all combinations of incident polar angle θ _i , incident azimuth angle φ _i , reflected polar angle θ _r , and reflected azimuth angle φ _r . Therefore, as in the technique described in Non-Patent Document 1, it is generally assumed that BRDF can be expressed by a parametric model, and the texture is estimated by a method of obtaining parameters of the model. Since this embodiment does not depend on the texture estimation method, any texture estimation method other than the above may be used. The initial value of the optimization calculation is arbitrary, but it is desirable to set an initial value close to actual texture data.

<Light source frequency>
The main point of this embodiment is that when the light source does not contain sufficient high-frequency components, it is determined that “texture estimation is difficult and it is necessary to apply a flash”. Hereinafter, the frequency of the light source will be described.

FIG. 6 is a diagram showing the relationship between the incident polar angle θ _i and the incident luminance L for sunlight (FIG. 6A) and cloudy sky (FIG. 6B) (for simplification of explanation). The incident azimuth angle φ _i is fixed). FIG. 7 is a diagram showing the relationship between the frequency order f and the incident luminance frequency coefficient L ′ for sunlight (FIG. 7A) and cloudy sky (FIG. 7B). Here, the frequency order f represents the frequency level, and the incident luminance frequency coefficient L ′ represents the result of frequency conversion of the incident luminance L. The method of frequency-converting the incident luminance L is arbitrary, but for example, spherical harmonic expansion and wavelet expansion described in Non-Patent Document 2 may be used. Hereinafter, the frequency of the light source will be described with reference to FIGS. 6 and 7.

  As shown in FIG. 6A, the incident luminance L is locally increased. This represents that the incident luminance L is extremely high only in a certain direction of the sun. In this way, when there is a direction in which the incident luminance L increases locally, the light source includes a large amount of high frequency components. Conversely, FIG. 6B is cloudy and there is no direction in which the incident luminance L increases locally. In such a case, the light source does not contain sufficient high frequency components. This difference appears in FIG. 7, and it can be seen that FIG. 7A contains more high-frequency components than FIG. 7B.

<Relationship between light source frequency, texture estimation success, and flash necessity>
Hereinafter, the relationship between the light source frequency, the success or failure of the texture estimation, and the necessity of the flash will be described. First, the reason why it is difficult to estimate the texture if the light source does not contain sufficient high-frequency components will be described using a high gloss texture such as metal and a low gloss texture such as plastic as examples. Comparing images obtained by imaging a high gloss subject and a low gloss subject, the way in which the light source is reflected is greatly different. Specifically, a light source is clearly reflected in an image obtained by imaging a high-gloss subject, and a light source is blurred in an image obtained by imaging a low-gloss subject. Whether the subject is high gloss or low gloss can be determined from the difference in how the light source is reflected. However, if the light source does not contain sufficient high-frequency components, that is, if the light source itself is blurred, the image is blurred regardless of whether a high-gloss subject or a low-gloss subject is imaged. As described above, since there is no difference in how the light source of the captured image is reflected, it is difficult to estimate the texture data. The above is the reason why it is difficult to estimate the texture data when the light source does not contain sufficient high frequency components.

  In the following, the above reason will be theoretically explained by converting the frequency of Equation (2). When Formula 2 is frequency-converted, it can be transformed into the following formula (details of formula transformation are described in Non-Patent Document 2).

Here, B ′ is a result of frequency conversion of BRDF (hereinafter referred to as BRDF frequency coefficient), and β is a conversion coefficient that varies depending on which frequency conversion is performed.

  Next, the reason why it is difficult to estimate the texture if the light source does not contain sufficient high-frequency components will be theoretically described with reference to Equation 4. When the light source does not contain sufficient high-frequency components, the incident luminance frequency coefficient L ′ is zero for large f. Then, it can be seen from Equation 4 that even if the BRDF frequency coefficient B ′ has a non-zero value for large f, the influence is not reflected on the pixel value I. Since the texture estimation is performed with reference to the pixel value I, it is difficult to estimate the texture when the influence of the BRDF frequency coefficient B ′ is not reflected on the pixel value I as described above. The above is the reason why it is difficult to estimate the texture if the light source does not contain sufficient high-frequency components.

  Note that if the BRDF frequency coefficient B ′ has a large frequency order f and does not have a non-zero value, that is, if the subject has low gloss, texture estimation is possible regardless of the value of the incident luminance frequency coefficient L ′. is there. That is, if the BRDF frequency coefficient B ′ that is the object of texture estimation is a large frequency order f and does not have a non-zero value, even if the incident luminance frequency coefficient L ′ is zero for the large frequency order f, the BRDF frequency B ′ Is reflected in the pixel value I. Therefore, texture estimation is possible. In other words, how large the incident luminance frequency coefficient L ′ needs to have a non-zero value up to the frequency order f depends on the BRDF frequency coefficient B ′.

<Flowchart of Flash Necessity Determination Unit 112>
FIG. 8 is a block diagram showing each processing unit of the flash necessity determination unit 112. FIG. 9 is a flowchart showing the operation of the flash necessity determination unit 112. Hereinafter, with reference to FIG. 9, the flow of operation of the flash necessity determination unit 112, which is the main point of the present embodiment, will be described.

  In step S901, the light source frequency conversion unit 801 performs frequency conversion on the light source data. The frequency conversion method is as described above.

  In step S902, the flash necessity determination unit 802 determines whether the light source sufficiently includes a high-frequency component. The determination method will be described later. If so, it is determined in step S903 that “flash is not necessary”, and if not, it is determined in step S904 that “flash is necessary”.

<Details of flash necessity determination processing>
FIG. 10 is a diagram illustrating the relationship between the frequency order f and the frequency order threshold value t. Hereinafter, with reference to FIG. 10, a method for determining whether or not a light source sufficiently includes a high-frequency component will be described. First, a method for providing a fixed threshold value for the frequency order f will be described. FIG. 10A and FIG. 10B show the incident luminance frequency coefficient L ′ when the frequency of different light sources is converted. Specifically, in FIG. 10A, frequency conversion is performed on a light source that sufficiently includes high frequencies such as sunlight, and in FIG. 10B, a light source that does not sufficiently include high frequencies such as cloudy sky is converted. In order to determine whether or not the light source sufficiently includes a high frequency component, a threshold value t of a frequency order f as shown in FIGS. 10A and 10B is determined in advance, and a frequency order smaller than the threshold value t is determined. What is necessary is just to confirm whether the incident luminance frequency coefficient L ′ becomes a small value at f. That is, if the incident luminance frequency coefficient L ′ is a small value at a frequency order f smaller than the threshold value t, it is determined that the high frequency component is not sufficiently included, and otherwise, it is determined that the high frequency component is sufficiently included. Note that a threshold value is set in advance to determine how small the incident luminance frequency coefficient L ′ does not contain sufficient high-frequency components when the incident luminance frequency coefficient L ′ is small, and the incident luminance frequency coefficient L ′ is smaller than the threshold value. Whether or not it should be judged.

  Next, a method for providing a variable threshold value for the frequency order f will be described with reference to FIG. 10C in which two threshold values t1 and t2 are provided for the frequency order f. Which of the threshold values t1 and t2 is used to determine whether flash is necessary or not depends on the BRDF frequency coefficient B ′ as described above. Therefore, if it is possible to determine which threshold value is used according to the BRDF frequency coefficient B ′, it is possible to determine whether flash is necessary or not with an appropriate threshold value. However, since the texture data is unknown, the BRDF frequency coefficient B ′ is also unknown, and the BRDF frequency coefficient B ′ cannot be used directly for determining the threshold. Therefore, it is considered that the BRDF frequency coefficient B ′ is roughly predicted, and the threshold is determined using the prediction. For example, an object recognition technique may be used to recognize the type of subject, and the threshold value may be determined using the highest frequency BRDF frequency coefficient B ′ that can be possessed by that type of subject. Alternatively, the type of subject may be set by the photographer, and a threshold value suitable for the type of subject may be set.

  The above is the description of the method for determining whether or not the flash is necessary from the incident luminance frequency coefficient L ′. Note that the present embodiment is not limited to directly determining whether flash is necessary or not from the incident luminance frequency coefficient L ′. In the following, a method for determining whether or not the flash is necessary from other than the incident luminance frequency coefficient L ′ will be described.

  First, a method of predicting the light source frequency from the imaging mode and determining the necessity of flash using the prediction result will be described. Here, the imaging mode refers to scene information used for automatically determining imaging settings of the camera, such as “cloudy mode” and “sunny mode”. In the present embodiment, the imaging mode is also used for determining whether or not the flash is necessary, and thus can be referred to as light source information. In this method, the relationship between the imaging mode and the light source frequency as shown in FIG. 11 is held as a table, and the necessity of flash is determined from the light source frequency with reference to this table. The table can be created by actually calculating the light source frequency in an imaging environment suitable for each imaging mode. Further, the relationship between the imaging mode and the necessity of flash may be held directly as a table.

  Next, as another method for determining whether or not flash is necessary, a method of performing pre-imaging before main imaging and determining whether or not flash is necessary depending on whether or not the pre-captured image data sufficiently includes a high-frequency component explain. Since the pre-captured image data is also used for determining whether or not the flash is necessary, it can be referred to as light source information. Specifically, if the pre-captured image is subjected to frequency conversion and the frequency coefficient I ′ (hereinafter referred to as “captured image frequency coefficient”) sufficiently includes a high-frequency component, “flash is unnecessary”. If it is not sufficient, it is determined that “flash is necessary”. The reason for making such a determination is that when the pixel value I does not include a high frequency component from Equation 4, either the incident luminance frequency coefficient L ′ or the BRDF frequency coefficient B ′ is a value close to zero for a large f. It is because it can be judged that it is. In other words, since there is a possibility that the incident luminance frequency coefficient L ′ is a value close to zero for a large f, it is determined that “flash is necessary”. Note that when the dynamic range of the pre-captured image is narrow, it may be determined that the pre-captured image does not sufficiently contain the high-frequency component.

  In the first embodiment, the imaging mode and pre-captured image data are light source information used for determining whether or not a flash is necessary. As the light source data used for texture estimation, the light source data acquired from the outside can be used as described above.

  As described above, according to the first embodiment, whether or not a flash is necessary at the time of imaging is determined from the light source information, and flash imaging is performed as necessary. Data can be obtained.

  In the first embodiment, the method for estimating the texture data using the light source data acquired from the outside has been described. In the second embodiment, a method of acquiring light source data using an imaging unit of the image processing apparatus and estimating light source data as well as texture data will be described.

<Flowchart of Example 2>
FIG. 12 is a flowchart showing the operation of the image processing apparatus shown in FIG. Hereinafter, the operation of the image processing apparatus according to the second embodiment will be described with reference to FIG.

  In step S1201, the flash necessity determination unit 112 determines whether it is necessary to throw a flash during imaging. Whether or not it is necessary to fire the flash may be determined from light source information such as the light source frequency predicted from the imaging mode and the captured image frequency coefficient I ′ of the pre-captured image, as described in the first embodiment. Note that light source data related to illumination information such as the position, direction, and luminance of the light source, which is an example of the light source information described in the first embodiment, is unknown in the second embodiment. It is not used for the determination.

  In step S <b> 1202, the imaging units 101 and 102 perform imaging according to the determination result of the flash necessity determination unit 112. Specifically, when the flash necessity determination unit 112 determines that “flash is not necessary”, imaging is performed without a flash, and when it is determined that “flash is necessary”, imaging is performed with no flash and with flash. That is, first image data obtained by imaging a subject using a flash and second image data obtained by imaging the subject without using a flash are acquired. When pre-imaging is performed in step S1201, the pre-captured image may be used as a non-flash captured image.

  In step S1203, the scene element data acquisition unit 106 acquires subject shape data. Since the shape data acquisition method is the same as that of the first embodiment, the description thereof is omitted.

  In step S1204, the scene element data acquisition unit 106 estimates subject texture data and light source data. Basically, the BRDF of formula 2 and the incident luminance L may be obtained by the same optimization calculation as in the first embodiment. However, in general, there are many combinations of BRDF and incident luminance L that satisfy Equation 2. Therefore, if the initial value of the optimization calculation is far from the actual texture data or light source data, it falls into a local solution and the estimation fails. Therefore, it is important that the initial value of the optimization calculation is close to actual texture data and light source data.

<Flowchart of Scene Element Data Acquisition Unit 106 in Embodiment 2>
In the second embodiment, the light source data is not acquired from the outside as in the first embodiment, but an appropriate estimation method and an initial value of the optimization calculation are obtained from the comparison result between the captured image with flash and the captured image without flash. Estimate both light source data and texture data. FIG. 13 is a diagram illustrating an example of each processing unit of the scene element data acquisition unit according to the second embodiment. Unlike the first embodiment, the input data is captured image data and shape data, and texture data and light source data are output based on these data.

  FIG. 14 is a flowchart illustrating the operation of the texture / light source data acquisition unit 1303 according to the second embodiment. Hereinafter, the operation of the texture / light source data acquisition unit 1303 according to the second embodiment will be described with reference to FIG.

  In step S1401, it is determined in step S1202 whether imaging has been performed with a flash.

  If it is determined that the image has been taken with the flash, in step S1402, the captured image data with the flash and the captured image data without the flash are frequency-converted.

In step S1403, the frequencies of the captured image data with flash and the captured image data without flash are compared. Specifically, comparing the minimum frequency order f1_min and f2_min ₂ 'frequency coefficients I ₁ and the flash There captured image data' frequency coefficients I flash no captured image data has a value close to zero, respectively. As to how close the frequency order is when the value becomes close to zero, a threshold value is set in advance, and the frequency order when the frequency coefficient I ′ becomes smaller than the threshold value may be compared.

  If the difference between f1_min and f2_min is larger than a preset threshold value, estimation is performed using a difference image between the captured image with flash and the captured image without flash in step S1404. Details of the estimation process using the difference image will be described later.

  If the difference between f1_min and f2_min is equal to or smaller than the threshold value, or if it is determined in step S1401 that imaging is not performed with a flash, an initial value for optimization calculation is calculated in step S1405. Specifically, when the non-flash captured image sufficiently includes high-frequency components, BRDF uses texture data that sufficiently includes high-frequency components as an initial value. On the other hand, if the captured image without flash does not contain sufficient high frequency components, BRDF sets the texture data that does not contain sufficient high frequency components as the initial value. The reason for this initial value is that, from Equation 2, when the pixel value I includes a high frequency component, the BRDF also includes a high frequency component, and when the pixel value I does not include a high frequency component, the BRDF also does not include a high frequency component. Because I understand. Note that what kind of texture data is specifically set as the initial value is arbitrary. For example, a high-frequency BRDF such as metal and a low-frequency BRDF such as plastic may be actually measured, and these BRDFs may be set as the initial value of the texture data according to the frequency of the captured image. As described above, when many high-frequency components are included, an optimization calculation process that gives an initial value corresponding to a high frequency is performed, and when many high-frequency components are not included, an optimization calculation process that gives an initial value corresponding to a low frequency is performed. Do.

  Subsequently, in step S1406, texture estimation and light source estimation processing are performed using only the non-flash captured image. The texture estimation process is the same as in the first embodiment. Further, since the pixel value, shape data, and texture data are known, the light source data can be estimated by obtaining the incident luminance L satisfying Equation 2 by optimization calculation.

  Here, the reason why the texture is estimated using only the non-flash captured image data when the difference between f1_min and f2_min is equal to or smaller than the threshold will be described. When f1_min and f2_min are the same, there is no difference in the quality estimation success / failure described in the first embodiment between the captured image data without flash and the captured image data with flash. This means that the light source in the scene contained sufficient high-frequency components even without a flash, and it can be understood that estimation can be made only from a captured image without flash. The above is the reason why estimation is performed using only the non-flash captured image when the difference between f1_min and f2_min is small. Note that the above is not intended to exclude performing texture estimation using a captured image with flash when the difference between f1_min and f2_min is small. That is, when the difference between f1_min and f2_min is small, texture estimation may be performed using difference image data as described later.

<Details of texture / light source data estimation processing using difference images>
FIG. 15 is a flowchart showing the flow of texture / light source data estimation processing using difference image data by the texture / light source data acquisition unit 1303. Hereinafter, the texture / light source data estimation process using the difference image data will be described with reference to FIG.

  In step S1501, difference image data is generated. The difference image data may be generated by taking the difference between the pixel values of the captured image data with flash and the captured image data without flash. Here, the difference image data should be the same as the image data obtained by capturing the scene using only the flash as the light source. This is because the captured image data with flash is captured image data using both the light source and flash that were originally in the scene, and the captured image data without flash is captured image data using only the light source that was originally in the scene as the light source. That's why. If there is a difference in camera position between the captured image data with flash and the captured image data without flash, the position between the images may be aligned by an image alignment technique before the difference image data is generated. However, it is generally difficult to perform alignment between images having different light source environments. Therefore, it is desirable to devise a method that does not cause misalignment, for example, by taking a fixed image on a tripod or by taking image data with flash and image data without flash almost simultaneously.

  In step S1502, the texture data is estimated from the difference image data. Here, if the flash information is known, the texture data can be estimated from the pixel value of the difference image data, the shape data, and the flash (light source) information as in the first embodiment.

  In step S1503, light source data at the time of shooting without flash is estimated from captured image data without flash. Here, since the texture data is estimated in step S1502, the pixel value, the shape data, and the texture data are already known. Therefore, the incident luminance L satisfying Equation 2 can be obtained by optimization calculation.

  As described above, the pixel value, shape data, and light source data are known when estimating the texture data, and the pixel value, shape data, and texture data are known when estimating the light source data. There is relatively little worry that the initial value of the optimization calculation falls into a local solution. Therefore, an arbitrary value can be used as the initial value, but it is more desirable to set an initial value close to actual light source data or texture data.

  As described above, according to the second embodiment, when an unknown light source does not sufficiently contain a high-frequency component, by performing imaging with and without flash, and estimating from the captured image data, It is possible to estimate both the texture data and the light source data.

<Other examples>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (14)

Determining means for determining whether to estimate the texture data of the subject from the image data captured using the flash based on the light source information;
An image processing apparatus comprising: texture data estimation means for estimating texture data of the subject according to determination by the determination means.   The image processing apparatus according to claim 1, wherein when the light source information includes a high frequency component, the determination unit determines to estimate the texture data of the subject from image data captured using the flash.   The image processing apparatus according to claim 1, further comprising a light source information acquisition unit that acquires light source data including at least one of a position, a direction, and luminance of the light source as the light source information.   The image processing apparatus according to claim 3, wherein the light source information acquisition unit acquires the light source data obtained by imaging the periphery of the subject.   The image processing apparatus according to claim 1, further comprising a light source information acquisition unit that acquires data indicating an imaging mode when the subject is imaged as the light source information.   The image processing apparatus according to claim 1, further comprising a light source information acquisition unit that acquires pre-captured image data obtained by capturing the subject without using a flash as the light source information. Image data acquisition means for acquiring image data obtained by imaging the subject;
Shape data acquisition means for acquiring shape data of the subject;
Light source data acquisition means for acquiring light source data including at least one of the position, direction, and luminance of the light source,
The texture data estimation means estimates the texture data using the image data acquired by the image data acquisition means, the shape data acquired by the shape data acquisition means, and the light source data acquired by the light source data acquisition means. An image processing apparatus according to claim 1, wherein   The image processing apparatus according to claim 1, further comprising a light source data estimation unit that estimates light source data of the subject according to the determination by the determination unit. Image data acquisition means for acquiring first image data obtained by imaging the subject using a flash and second image data obtained by imaging the subject without using a flash;
The apparatus further comprises control means for causing the texture data estimation means and the light source data estimation means to perform estimation processing according to a frequency order difference between the first image data and the second image data. The image processing apparatus according to claim 8.   When the frequency order difference is greater than a threshold value, the control unit uses the difference image data, which is a difference between the first image data and the second image data, to the texture data estimation unit. The image processing apparatus according to claim 9, wherein the light source data estimation unit causes the light source data estimation unit to estimate the light source data using the estimated texture data and the second image data.   The control means causes the texture data estimation means to estimate the texture data using the second image data when the frequency order difference is equal to or less than the threshold value, and causes the light source data estimation means to perform the second image. The image processing apparatus according to claim 9, wherein the light source data is estimated using data.   The texture data estimation means performs an optimization calculation process that gives an initial value corresponding to a high frequency if it contains a lot of high frequency components, and an optimization that gives an initial value that corresponds to a low frequency if it does not contain a lot of high frequency components The image processing apparatus according to claim 11, wherein calculation processing is performed. A determination step of determining based on the light source information whether to estimate the texture data of the subject from the image data captured using the flash;
An image processing method comprising: a texture data estimation step of estimating texture data of the subject according to the determination in the determination step.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 12.

Images