JP2015015676A - Image processing device, and method and program of controlling the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To select a region where a mirror surface reflection component is extracted in an image, and to perform light source color estimation with accuracy.SOLUTION: When performing light source color estimation processing for estimating a light source color of a light source in an image obtained by photographing a subject in such a photographing environment that the light source is present, an image processing unit 105 detects a mirror surface reflection candidate region where mirror surface reflection by the light source may occur and a periphery region located at the periphery of the mirror surface reflection candidate region, in a first image obtained by photographing the subject while making auxiliary light be in a non-light-emission state, and extracts a reflection light component by the auxiliary light on the basis of the first image, and a second image obtained by photographing the subject while emitting the auxiliary light. The image processing unit compares the reflection light components in the mirror surface reflection candidate region and in the peripheral region with each other, and determines whether or not the mirror surface reflection candidate region is the mirror surface reflection region where mirror surface reflection by the light source occurs, and performs the light source color estimation processing by using the determined mirror surface reflection region.

Description

  The present invention relates to an image processing apparatus, a control method therefor, and a control program, and more particularly to an image processing apparatus that performs light source color estimation processing for estimating the color of a light source that illuminates a subject (subject image) present in an image.

  In general, an imaging apparatus such as a digital camera is known that includes an image processing apparatus that estimates a light source color based on a dichroic reflection model. The dichroic reflection model is a model in which reflected light from an object is composed of a specular reflection component and a diffuse reflection component.

  Since the specular reflection component has characteristics depending on the light source color, the light source color can be estimated by extracting the specular reflection component from the reflected light.

  For example, there is an image processing apparatus that obtains a difference in image values between adjacent pixels of an input image, extracts a specular reflection component from the difference, and performs light source color estimation (see Patent Document 1).

JP 2007-13415 A

  By the way, in the image processing apparatus described in Patent Document 1, light source color estimation is performed assuming that specular reflection occurs when there is a luminance difference between pixels.

  However, the luminance difference is not necessarily caused by the specular reflection component. For example, the luminance difference may exist due to the difference in the object color near the boundary between the red subject and the black subject.

  Thus, if the luminance difference does not depend on the specular reflection component, the specular reflection component cannot be extracted from the luminance difference, and as a result, the light source color cannot be accurately estimated.

  Therefore, an object of the present invention is to provide an image processing apparatus, a control method thereof, and a control program that can select a region from which a specular reflection component is extracted in an image and accurately estimate a light source color.

  In order to achieve the above object, an image processing apparatus according to the present invention performs a light source color estimation process for estimating a light source color of a light source in an image obtained by photographing a subject in a photographing environment where the light source exists. In the first image obtained by photographing the subject without using predetermined auxiliary light, the specular reflection candidate region in which specular reflection by the light source may occur and the specular reflection candidate Based on a specular reflection region detecting means for detecting a peripheral region located around the region, the first image and a second image obtained by photographing the subject by emitting the auxiliary light. The reflected light extraction means for extracting the reflected light component by the auxiliary light is compared with the reflected light component in the specular reflection candidate area and the peripheral area, and the specular reflection candidate area is determined according to the comparison result. Specifying means for specifying whether the specular reflection area that specular reflection is caused by the source, and having a light source color estimation means for performing said light source color estimation processing using the specular reflection region.

  A control method according to the present invention is a control method for an image processing apparatus that performs light source color estimation processing for estimating a light source color of a light source in an image obtained by photographing a subject in a photographing environment where a light source exists, and is determined in advance. In the first image obtained by photographing the subject with the auxiliary light not emitted, the specular reflection candidate area where the specular reflection by the light source may occur is located around the specular reflection candidate area A reflected light component by the auxiliary light based on a specular reflection area detecting step for detecting a peripheral area, and a second image obtained by photographing the subject by emitting the first light and the auxiliary light. The reflected light extraction step of extracting the reflected light component in the specular reflection candidate area and the peripheral area is compared, and the specular reflection candidate area is determined by the light source according to the comparison result. A specifying step of specifying whether the specular reflection region where reflection occurs, and having a light source color estimation step of the light source color estimation processing using the specular reflection region.

  A control program according to the present invention is a control program used in an image processing apparatus that performs a light source color estimation process for estimating a light source color of the light source in an image obtained by photographing a subject in a photographing environment where a light source exists. A specular reflection candidate area in which a specular reflection by the light source may occur in the first image obtained by photographing the subject without emitting predetermined auxiliary light to a computer provided in the image processing apparatus And a specular reflection region detecting step for detecting a peripheral region located around the specular reflection candidate region, and a second image obtained by photographing the subject by emitting the first image and the auxiliary light The reflected light extraction step for extracting the reflected light component by the auxiliary light based on the above and the reflected light component in the specular reflection candidate area and the peripheral area are compared. A step of specifying whether or not the specular reflection candidate region is a specular reflection region in which specular reflection by the light source occurs according to the comparison result, and a light source that performs the light source color estimation process using the specular reflection region And a color estimation step.

  According to the present invention, the specular reflection candidate region and its peripheral region are detected in the first image, and the reflected light component by the auxiliary light is extracted based on the first image and the second image. Then, the reflected light components in the specular reflection candidate area and the peripheral area are compared to determine whether the specular reflection candidate area is a specular reflection area where specular reflection is caused by the light source, and light source color estimation is performed using the specular reflection area Since the process is performed, the light source color can be estimated with high accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram for explaining a configuration of an imaging apparatus that is an example of an image processing apparatus according to a first embodiment of the present invention, where (a) is a block diagram showing the configuration of the imaging apparatus, and (b) is (a). It is a block diagram which shows the structure of the image process part shown in FIG. It is a figure for demonstrating an example of imaging | photography of the to-be-photographed object by the camera shown in FIG. It is a flowchart for demonstrating the light source color estimation process performed with the camera shown in FIG. It is a flowchart for demonstrating a specular reflection candidate area | region and surrounding area detection process shown in FIG. It is a flowchart for demonstrating the reflected light extraction process shown in FIG. It is a figure which shows an example of the reflected light data storage area for storing the 1st difference and 2nd difference with which the camera shown to Fig.1 (a) was equipped. It is a flowchart for demonstrating the specular reflection area | region specific process shown in FIG. 4 is a flowchart for explaining white balance gain calculation processing shown in FIG. 3. It is a flowchart for demonstrating the light source color estimation process performed with the camera by the 2nd Embodiment of this invention. It is a figure which shows the division | segmentation of the image performed by the image process part with which the camera by the 2nd Embodiment of this invention was equipped. 10 is a flowchart for explaining a specular reflection candidate area and peripheral area detection process shown in FIG. 9. 12 is a flowchart for explaining the group division processing shown in FIG. 11. It is a figure for demonstrating an example of the classification table (data structure) which shows the result of the group classification of the pixel performed in the image process part by the 2nd Embodiment of this invention. 10 is a flowchart for explaining reflected light extraction processing of auxiliary light shown in FIG. 9. It is a figure which shows an example of the reflected light data storage area with which the camera by the 2nd Embodiment of this invention was equipped. It is a flowchart for demonstrating the specular reflection area | region specific process shown in FIG. It is a figure which shows an example of the flag storage area which stores a flag with respect to the block identified as a specular reflection area. 10 is a flowchart for explaining white balance gain calculation processing shown in FIG. 9.

  Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. Here, an imaging apparatus such as a digital camera which is one of the image processing apparatuses will be described as an example.

[First Embodiment]
FIG. 1 is a block diagram for explaining a configuration of an imaging apparatus which is an example of an image processing apparatus according to the first embodiment of the present invention. FIG. 1A is a block diagram illustrating the configuration of the imaging apparatus, and FIG. 1B is a block diagram illustrating the configuration of the image processing unit illustrated in FIG.

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera) 100 and includes a photographing lens unit (hereinafter simply referred to as a photographing lens) 101. A shutter 102 having an aperture function is disposed at the rear stage of the photographing lens 101. A subject image (optical image) incident through the photographing lens 101 is formed on the imaging unit 103.

  The imaging unit 103 has an imaging element such as a CCD or CMOS element, and outputs an electrical signal (analog image signal) corresponding to the optical image. The analog image signal is converted into a digital image signal by the A / D converter 104, and the digital image signal is given to the image processing unit 105.

  The image processing unit 105 performs TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing according to the digital image signal. Further, the image processing unit 105 performs AWB (auto white balance) processing on the digital image signal and outputs the image data.

  The memory control unit 107 controls the memory 106 under the control of the system control unit 110 and writes the image data obtained by the image processing unit 105 into the memory 106 in a predetermined format. Further, the memory control unit 107 receives the digital image signal which is the output of the A / D converter 104 and displays an image corresponding to the digital image signal on the display unit 109 via the D / A converter 108.

  The memory control unit 107 also reads the image data recorded in the memory 106 and displays an image corresponding to the image data on the display unit 109 via the D / A converter 108.

  The nonvolatile memory 124 is an electrically erasable and recordable memory. For example, an EEPROM is used. The nonvolatile memory 124 stores constants and programs for operating the system control unit 110. Here, the program is, for example, a program for executing a flowchart described later.

  The system control unit 110 controls the entire camera 100. The system control unit 110 executes a program recorded in the nonvolatile memory 124 and performs each process described later. For example, a RAM is used as the system memory 125. In the system memory 125, constants and variables for operation of the system control unit 110, a program read from the nonvolatile memory 124, and the like are expanded.

  The system control unit performs display control by controlling the memory 107, the D / A converter 108, the display unit 109, and the like.

  The camera 100 is provided with an interface (I / F) 111, and a recording medium 112 for recording image data is connected to the I / F 111. Furthermore, an operation unit 120, a power switch 121, and a power supply control unit 122 are connected to the system control unit 110, and a power supply unit (for example, a battery) 123 is connected to the power supply control unit 122. The auxiliary light emitting device 201 is detachably attached to the camera 100, and the system control unit 110 controls the light emission of the auxiliary light emitting device 201.

  The operation unit 120 is operated by a user, and the user instructs the operation of the camera 100 using the operation unit 120. The power supply control unit 122 supplies power from the power supply unit 121 to the system control unit 110, and detects whether the power supply unit 121 is attached, the type of the power supply unit 121, and the remaining battery level.

  As shown in FIG. 1B, the image processing unit 105 includes a specular reflection candidate area detection unit 131, a reflected light detection unit 132, a specular reflection area specification unit 133, and a white balance (WB) gain calculation unit 134. ing. The light source color estimation process in the image processing unit 105 will be described later.

  FIG. 2 is a diagram for explaining an example of photographing a subject by the camera 100 shown in FIG.

  Here, an attempt is made to photograph the subject 202 to be photographed by the camera 100. In the illustrated shooting environment, there is a light source 200 whose color temperature is unknown. The auxiliary light emitting device 201 emits light in synchronization with photographing by the camera 100. The color temperature of light emitted by the auxiliary light emitting device 201 is known.

  Here, an outline of the operation when photographing the subject 202 in the camera 100 shown in FIG. 1 will be described.

  As described above, the imaging unit 103 outputs an analog image signal corresponding to an optical image incident through the photographing lens 101 and the shutter 102. The A / D converter 104 converts the analog image signal into a digital image signal and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs gamma processing, noise reduction processing, and the like on the digital image signal output from the A / D converter 104 or the image data sent from the memory control unit 107. Further, the image processing unit 105 performs a light source color estimation process, which will be described later, and performs an AWB (auto white balance) process.

  Image data that is an output of the image processing unit 105 is recorded on the recording medium 112 via the I / F 111 under the control of the system control unit 110. As described above, the auxiliary light emitting device 201 is controlled by the system control unit 110 and emits auxiliary light in synchronization with photographing.

  Here, the light source color estimation process will be described with reference to FIG. In FIG. 1B, only blocks necessary for the light source color estimation process are shown.

  The specular reflection candidate area detecting unit 131 detects a specular reflection candidate area in which specular reflection by the light source may occur in the image data sent from the memory control unit 107 and its peripheral area, and the specular reflection candidate area The detection result is stored in the memory 106.

  The reflected light extraction unit 132 extracts reflected light of the light emitted from the auxiliary light emitting device 201 and reflected by the subject 202 or the like based on the specular reflection candidate region detection result and the image data by the specular reflection candidate region detection unit 131. And stored in the memory 106 as the reflected light extraction result.

  The specular reflection area specifying unit 133 specifies the specular reflection area from the specular reflection candidate area based on the reflected light extraction result extracted by the reflected light extraction unit 132 and stores the specular reflection area in the memory 106 as the specular reflection area specifying result. The white balance gain calculation unit 134 calculates a white balance gain based on the specular reflection region specification result specified by the specular reflection region specification unit 133.

  As described above, the image processing unit 105 detects the specular reflection candidate region, specifies the specular reflection region from the specular reflection candidate region using the reflected light of the auxiliary light, and performs light source color estimation.

  FIG. 3 is a flowchart for explaining a light source color estimation process performed by the camera 100 shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the system control unit 110.

  Now, when the shutter button provided in the operation unit 120 is pressed by the user, the system control unit 110 stops the light emission by the auxiliary light emitting device 201 and outputs a digital image signal that is an output from the A / D converter 104. Is sent to the image processing unit 105.

  Here, a digital image signal obtained in a state where light emission of the auxiliary light emitting device 201 is stopped (non-light emission) is referred to as a first image.

  Subsequently, after acquiring the first image, the system control unit 110 causes the auxiliary light emitting device 201 to emit light within a predetermined time, and outputs the digital image signal that is the output of the A / D converter 104 to the image processing unit. Sent to 105.

  Here, a digital image signal obtained in a state where the auxiliary light emitting device 201 emits light is referred to as a second image. However, when shooting the first and second images, the shooting conditions other than the light emission and non-light emission of the auxiliary light emitting device 201 are not changed, and the saturated area when shooting the second image is the light source color. It shall not be used for estimation processing.

  As described above, the system control unit 110 captures the first and second images (step S301), and sends the first and second images to the image processing unit 105.

  Subsequently, in the image processing unit 105, the specular reflection candidate area detection unit 131 detects a specular reflection candidate area and a peripheral area generated by the light source 200 that illuminates the subject 202 from the first image (step S302). The process in step S302 will be described later.

  Next, the reflected light extraction unit 132 extracts the reflected light of the auxiliary light from the first image and the second image (step S303). Then, the specular reflection region specifying unit 133 performs the first and the second based on the reflected light of the auxiliary light extracted in step S303 (that is, the comparison result between the reflected light component in the specular reflection candidate region and the reflected light component in the peripheral region). It is specified whether or not each pixel in the second image is a specular reflection region (step S304).

  Subsequently, the white balance gain calculation unit 134 calculates a white balance gain according to the pixel identified as the specular reflection region in step S304 (step S305), and ends the light source color estimation process.

  FIG. 4 is a flowchart for explaining the specular reflection candidate region and peripheral region detection processing shown in FIG.

  When the area detection process is started, the specular reflection candidate area detection unit 131 sequentially loops for all pixels in the first image (step S401), and performs the processes of steps S402 to S403 for all pixels. The specular reflection candidate area detection unit 131 determines whether or not the pixel value of the target pixel in the first image is larger than the pixel value of the neighboring pixel of the target pixel (step S402).

  Here, as a neighboring pixel, a pixel separated by a predetermined d pixels in any one of left, right, top, and bottom with respect to the target pixel is selected as the neighboring pixel. For example, 10 or 20 is used as d, but the value of d may be changed as appropriate.

  Now, if the pixel value of the target pixel is represented by P (i, j), the pixel value of the pixel that is d pixels away from the target pixel to the left can be represented as P (i, j-d). When comparing the magnitudes of the two pixel values, the following equation (1) is satisfied using a predetermined threshold T1 (first threshold) in consideration of the influence of noise, that is, When the difference is larger than the threshold value T1, the specular reflection candidate area detection unit 131 determines that the pixel value of the target pixel is larger than the pixel value of the neighboring pixel.

P (i, j) -P (i, j-d)> T1 (1)
When RGB is used as the pixel value, the following formula (2) to formula (4) are used instead of formula (1), and when the formula (2) to formula (4) are satisfied, the specular reflection candidate The region detection unit 131 determines that the pixel value of the target pixel is larger than the pixel value of the neighboring pixels.

  When RGB is used as the pixel value, the R component of the threshold T1 is set to T1. R and G and B are similarly set to threshold values T1. G and T1. B.

P (i, j). RP (i, j-d). R> T1. R (2)
P (i, j). GP (i, j-d). G> T1. G (3)
P (i, j). BP (i, j-d). B> T1. B (4)
In the above-described example, the case where the pixel that is d pixels away from the target pixel is selected as the neighboring pixel has been described. However, when the pixel that is d pixels away from the target pixel right, up, or down is selected as the neighboring pixel. Are compared in the same way.

  Instead of performing comparison using RGB as the pixel value, luminance may be calculated from RGB, and comparison may be performed based on the luminance.

  If the pixel value of the target pixel is equal to or smaller than the pixel value of the neighboring pixel (NO in step S402), that is, if the difference is equal to or smaller than the threshold value, the specular reflection candidate area detection unit 131 sets the next pixel as the target pixel in step S402. Perform the process.

  On the other hand, if the pixel value of the target pixel is larger than the pixel value of the neighboring pixel (YES in step S402), that is, if the difference between the neighboring pixel value and the target pixel value is larger than the threshold value T1, the specular reflection candidate region detection unit 131 detects the target pixel as a specular reflection candidate area and its neighboring pixels as a peripheral area (step S403).

  As described above, when the processes of steps S402 and S403 are performed for all pixels, the specular reflection candidate area detection unit 131 reaches the end of all the pixel loops (step S404), and the specular reflection candidate area and peripheral area detection processing is performed. Exit.

  FIG. 5 is a flowchart for explaining the reflected light extraction processing shown in FIG.

  When the reflected light detection process is started, the reflected light extraction unit 132 sequentially loops for all pixels in each of the first and second images (step S501), and performs the processing of steps S502 to S504 for all pixels. First, the reflected light extraction unit 132 determines whether or not the target pixel is a specular reflection candidate area in step S302 illustrated in FIG. 3 (step S502).

  If the target pixel is not a specular reflection candidate region (NO in step S502), the reflected light extraction unit 132 performs the process of step S502 with the next pixel as the target pixel.

  On the other hand, if the target pixel is a specular reflection candidate area (YES in step S502), the reflected light extraction unit 132 obtains the difference between the pixel values of the corresponding target pixels in the first and second images (step S503). Here, the reflected light extraction unit 132 subtracts the pixel value of the first image from the pixel value of the second image for the target pixel to obtain the difference D1 (i, j), and the difference (first difference). Let D1 (i, j) be the reflected light data (also referred to as reflected light component) of the auxiliary light in the specular reflection candidate region.

  Since the pixel value is expressed in RGB, when calculating the difference, the reflected light extraction unit 132 calculates the difference using the following equations (5) to (7).

D1 (i, j). R = Q (i, j). RP (i, j). R (5)
D1 (i, j). G = Q (i, j). GP (i, j). G (6)
D1 (i, j). B = Q (i, j). BP (i, j). B (7)
Here, D1 (i, j) is a difference, P (i, j) is a pixel of the first image, and Q (i, j) is a pixel of the second image.

  Note that the difference D1 (i, j) obtained by the processing in step S503 is stored in a specular reflection candidate area of the target pixel described later.

  Subsequently, the reflected light extraction unit 132 subtracts the pixel value of the first image from the pixel value of the second image for the neighboring pixels related to the target pixel to obtain a difference D2 (i, j). Then, the reflected light extraction unit 132 sets the difference (second difference) D2 (i, j) as reflected light data of auxiliary light in the peripheral region.

  6 stores the first difference D1 (i, j) and the second difference D2 (i, j) provided in the camera (here, for example, the memory 106) shown in FIG. 1 (a). It is a figure which shows an example of the reflected light data storage area | region of.

  The reflected light data storage area shown in the figure has a specular reflection candidate area column and a peripheral area column, and the first difference D1 (i, j) is included in the specular reflection candidate area column for each target pixel (pixel No). To be recorded. Further, the second difference D2 (i, j) is recorded for each target pixel in the peripheral area column.

  In FIG. 6, i = 0 to L (L is an integer of 2 or more), and j = 0 to N (N is an integer of 2 or more).

  Since the calculation method of the second difference D2 (i, j) is the same as the calculation method of the first difference D1 (i, j), description thereof is omitted here. When a plurality of neighboring pixels are selected, the average value of the second differences D2 (i, j) calculated for each neighboring pixel is stored in the peripheral area of the target pixel.

  As described above, when the processes of steps S502 to S504 are performed for all the pixels, the reflected light extraction unit 132 has reached the end of all the pixel loops (step S505), and the reflected light extraction process ends.

  FIG. 7 is a flowchart for explaining the specular reflection area specifying process shown in FIG.

  When the specular reflection area specifying process is started, the specular reflection area specifying unit 133 sequentially loops for all pixels in each of the first and second images (step S601), and performs the processing of steps S602 to S604 for all pixels.

  First, the specular reflection area specifying unit 133 determines whether or not the target pixel is a specular reflection candidate area in step S302 illustrated in FIG. 3 (step S602). If the target pixel is not a specular reflection candidate area (NO in step S602), the specular reflection area specifying unit 133 performs the process of step S602 with the next pixel as the target pixel.

  On the other hand, if the target pixel is a specular reflection candidate area (YES in step S602), the specular reflection area specifying unit 133 determines whether or not the level of the reflected light of the auxiliary light approximates the target pixel and its neighboring pixels. (Step S603).

  Here, the specular reflection region specifying unit 133 uses the predetermined threshold value T2 (second threshold value) and the above-described reflected light data (that is, the first and second differences) to obtain the following equation (8). When satisfied, it is determined that the level of the reflected light of the auxiliary light is approximate between the target pixel and the neighboring pixels.

| D1 (i, j) -D2 (i, j) | <T2 (8)
When the pixel value is expressed in RGB, the specular reflection area specifying unit 133 uses the predetermined threshold values T2 _R , T2 _G , and T2 _B instead of the formula (8), and the specular reflection area specifying unit 133 uses the following formula (9 If all of () to (11) are satisfied, it is determined that the level of the reflected light of the auxiliary light approximates the target pixel and its neighboring pixels.

| D1 (i, j). R-D2 (i, j). R | <T2 _R (9)
| D1 (i, j). G-D2 (i, j). G | <T2 _G (10)
| D1 (i, j). B-D2 (i, j). B | <T2 _B (11)
If the level of the reflected light of the auxiliary light is not approximated in the target pixel and its neighboring pixels (NO in step S603), the specular reflection area specifying unit 133 returns to the process in step S602, and the next pixel is set as the target pixel. The process of S602 is performed.

  On the other hand, when the level of the reflected light of the auxiliary light approximates the target pixel and its neighboring pixels (YES in step S603), that is, when the above formula (8) is satisfied, the specular reflection region specifying unit 133 performs specular reflection. It is estimated that the candidate area and the peripheral area have substantially the same diffuse reflectance of auxiliary light and are the same subject.

  Then, since the specular reflection area exists, the specular reflection area specifying unit 133 assumes that there is a difference between the reflected light of the light source 200 in the specular reflection candidate area and the reflected light of the light source 200 in the peripheral area. The reflection area is identified (step S604).

  When Expression (8) is not satisfied, the specular reflection area specifying unit 133 assumes that the specular reflection candidate area and the peripheral area are different subjects, and the specular reflection candidate area is not a specular reflection area.

  As described above, when the processes in steps S602 to S604 are performed for all the pixels, the specular reflection area specifying unit 133 reaches the end of all the pixel loops (step S605), and the specular reflection area specifying process ends.

  FIG. 8 is a flowchart for explaining the white balance gain calculation processing shown in FIG.

  When the white balance gain calculation process is started, the WB gain calculation unit 134 sequentially loops for all pixels in each of the first and second images (step S701), and performs the processes of steps S702 and S703 for all pixels.

  First, the WB gain calculation unit 134 determines whether or not the target pixel is specified as a specular reflection region in step S304 illustrated in FIG. 3 (step S702). If the target pixel is not specified as the specular reflection area (NO in step S702), the WB gain calculation unit 134 performs the process of step S702 with the next pixel as the target pixel.

  On the other hand, when the target pixel is specified as the specular reflection area (YES in step S702), the WB gain calculation unit 134 uses the pixel value RGB of the target pixel in the first image to express the following formulas (12) and ( 13), the white balance gain relating to the target pixel is calculated (step S703).

GainR = G / R (12)
GainB = G / B (13)
Then, the white balance gain is added to the target pixel specified as the specular reflection region using the following equations (14) and (15).

SumGainR = ΣGainR (14)
SumGainB = ΣGainB (15)
Here, each time the addition process is performed, the WB gain calculation unit 134 increments the addition pixel number counter (Count) by one.

  As described above, when the processing of steps S702 and S703 is performed for all the pixels, the WB gain calculation unit 134 has reached the end of all pixel loops (step S704), and the following equations (16) and (17) Accordingly, the white balance gain in the entire image is calculated (step S705). Then, the WB gain calculation unit 134 ends the white balance gain calculation process. That is, the WB gain calculation unit 134 ends the light source color estimation process.

TotalGainR = SumGainR / Count (16)
TotalGainB = SumGainB / Count (17)
Thus, in the first embodiment of the present invention, the specular reflection region by the light source 200 that illuminates the subject 202 is specified using the reflected light when the auxiliary light emitting device 201 emits light. As a result, the white balance gain can be calculated using only the specular reflection region. As a result, the accuracy of white balance correction in the image is improved.

  In the first embodiment, the first and second images are taken at the same angle of view. Further, the position of the auxiliary light emitting device 201 is appropriately changed so that the irradiation direction of the auxiliary light does not overlap the irradiation direction of the light source. And the image image | photographed in the state from which the irradiation direction of auxiliary light and the irradiation direction of a light source differ is made into a 2nd image.

  When the difference between the reflected light of the auxiliary light in the specular reflection candidate region and the reflected light of the auxiliary light in the peripheral region is calculated, and the hue of this difference is similar to the auxiliary light, the specular reflected light of the auxiliary light is It is strongly included, and the irradiation direction of the auxiliary light overlaps with the irradiation direction of the light source. At this time, the system control unit 110 adjusts the light emitting direction of the auxiliary light emitting device 201.

[Second Embodiment]
Next, an imaging apparatus which is an example of an image processing apparatus according to the second embodiment of the present invention will be described. The configuration of the imaging apparatus according to the second embodiment is the same as that of the imaging apparatus shown in FIG.

  In the first embodiment described above, an example has been described in which it is determined whether each pixel is a specular reflection region and light source color estimation is performed according to the determination result. In the second embodiment, an image is divided into blocks (regions) of a predetermined size, and it is determined whether or not a specular reflection region is included for each block, and light source color estimation is performed according to the determination result. An example will be described.

  FIG. 9 is a flowchart for explaining a light source color estimation process performed by the camera according to the second embodiment of the present invention.

  Note that the processing according to the illustrated flowchart is performed under the control of the system control unit 110. Further, in the flowchart shown in FIG. 9, the same steps as those shown in FIG.

  As described in step S <b> 301 shown in FIG. 3, the image processing unit 105 obtains the first and second images under the control of the system control unit 110. Thereafter, in the image processing unit 105, the specular reflection candidate area detection unit 131 divides each of the first and second images into blocks (areas) of a predetermined size (step S802).

  FIG. 10 is a diagram showing image division performed by the image processing unit provided in the camera according to the second embodiment of the present invention.

  In the example shown in FIG. 10, the first image is divided into 20 blocks in total, which are divided into 4 parts in the vertical direction and 5 parts in the horizontal direction. Similarly, the second image is also divided into 20 blocks.

  In the example shown in the drawing, each of the first and second images is divided into a total of 20 blocks of 4 × 5. However, the size (size) of the block is arbitrary, and an optimal size is appropriately selected. What is necessary is just to divide into. However, the first and second images are divided into blocks of the same size.

  Referring to FIG. 9 again, after each of the first and second images is divided into a plurality of blocks as described above, image processing unit 105 executes the processes of steps S803 to S806 described later. Then, the light source color estimation process is terminated.

  In FIG. 9, as described later, it is determined whether or not a specular reflection area is included for each block.

  FIG. 11 is a flowchart for explaining the specular reflection candidate area and peripheral area detection process shown in FIG. 9.

  When the area detection process is started, the specular reflection candidate area detection unit 131 first performs a group division process for dividing the pixels into a plurality of groups for each block based on the pixel values included in the block for each first image (step S1). S901).

  FIG. 12 is a flowchart for explaining the group division processing shown in FIG.

  When the group division process is started, the specular reflection candidate area detection unit 131 sequentially loops all blocks in the first image (step S1001), and then sequentially loops all pixels for each group (step S1002). The processing of S1003 to S1005 is performed for all pixels.

  The specular reflection candidate area detection unit 131 acquires a pixel value S for each pixel included in the block for each block (step S1003). Here, RGB is used as the pixel value. Subsequently, the specular reflection candidate area detection unit 131 calculates the luminance value Y of the pixel according to the pixel value (RGB), and determines which group the pixel is classified according to the luminance value Y ( Step S1004).

  Now, when the luminance value Y is defined in the range of “0” to “255” (assuming that the luminance value Y is an integer from “0” to “255”), for example, a specular reflection candidate region detection unit If the luminance value Y is “0” or more and “15” or less, the pixel is classified into the first group. Similarly, when the luminance value Y is “16” or more and “31” or less, the specular reflection candidate region detection unit 131 classifies the pixel into the second group. As described above, the specular reflection candidate area detection unit 131 performs pixel group classification using a fixed value regardless of a block.

  FIG. 13 is a diagram for explaining an example of a classification table (data structure) indicating a result of pixel group classification performed by the image processing unit according to the second embodiment of the present invention. The illustrated classification table is stored in, for example, the memory 106 shown in FIG.

  In the illustrated example, a classification table (hereinafter referred to as a first classification table) related to non-assist light emission (that is, the first image) and a classification table (hereinafter referred to as the second image) related to auxiliary light emission (that is, the second image). Hereinafter referred to as a second classification table).

  Both the first and second classification tables have a group No column and a block No column, and are associated with the group No column and the block No column, respectively, and the total pixel value (Sum), the total number of pixels (Count), and the luminance average The value (Avg) is recorded. In the example shown in the figure, the groups are divided into 16 groups, but the optimum number of groups may be appropriately set.

  Referring again to FIG. 12, the specular reflection candidate area detection unit 131 performs group grouping of pixels, and then updates the total pixel value Sum of the group when auxiliary light is not emitted (first image), and the pixel value. Each time is added, the total number of pixels Count is incremented (step S1006).

  The total pixel value is calculated using the following equation (18).

Sum = ΣS (18)
However, S is a pixel value.

  When RGB is used as the pixel value S, the total pixel value is calculated for each of RGB using the following equations (19) to (21) instead of the equation (18).

Sum. R = ΣS. R (19)
Sum. G = ΣS. G (20)
Sum. B = ΣS. B (21)
As described above, when the processing of steps S1003 to S1005 is performed for all the pixels, the specular reflection candidate area detecting unit 131 reaches the end of all pixel loops (step S1006), and the following equation (22) is obtained for each classification group. ) To calculate the average pixel value Avg (step S1007).

Avg = Sum / Count (22)
In addition, when using RGB as a pixel value, an average pixel value is calculated about each of RGB using Formula (23)-Formula (25) instead of Formula (22).

Avg. R = Sum. R / Count (23)
Avg. G = Sum. G / Count (24)
Avg. B = Sum. B / Count (25)
As described above, the average pixel value for each group is stored in the first or second classification table described with reference to FIG.

  Subsequently, the specular reflection candidate area detection unit 131 refers to the first and second classification tables shown in FIG. 13 and searches for the group with the highest luminance value and the group with the second highest luminance value for each block. The A group (first group) and the B group (second group), respectively.

  When searching for the A group, the specular reflection candidate area detection unit 131 searches for groups including pixels in order from the group with the highest luminance value Y. For example, when the group is classified into 16 groups (in this case, the luminance value is assumed to decrease in order from the 16th), the specular reflection candidate area detection unit 131 searches from the 16th group, It is determined whether or not a pixel is included in the 16th group. When the pixel is included in the 16th group, the specular reflection candidate area detection unit 131 sets the 16th group as the A group.

  If no pixel is included in the 16th group, the specular reflection candidate area detection unit 131 determines whether the 15th group includes a pixel. In this way, when a search is performed in order from the group with the highest luminance value, and there is a group including pixels, that group is set as the A group.

  After searching for the A group, the specular reflection candidate area detection unit 131 next searches for the B group. Here, the specular reflection candidate area detection unit 131 searches for a group including pixels in order from a group having a luminance value one lower than that of the A group.

  For example, when the A group is the twelfth group, the specular reflection candidate area detection unit 131 determines whether or not the pixel is included in the eleventh group. When the eleventh group includes a pixel, the specular reflection candidate area detection unit 131 sets the eleventh group as the B group.

  On the other hand, when the pixel is not included in the eleventh group, the specular reflection candidate area detection unit 131 determines whether the pixel is included in the tenth group. In this way, the search for the B group is sequentially performed in the group having the luminance value lower than that of the A group.

  Note that the specular reflection candidate area detecting unit 131 does not have at least one of the specular reflection candidate area and the peripheral area for blocks that have almost no difference in the luminance value Y of each pixel in the block and cannot be divided into two or more groups. Is determined.

  After searching for the A group and the B group, the specular reflection candidate area detecting unit 131 sets the average pixel value of the A group as the A value and the average pixel value of the B group as the B value (step S1008). As described above, when the processes of steps S1007 and S1008 are performed for all the pixels, the specular reflection candidate area detection unit 131 has reached the end of all the block loops (step S1009), and the group division process ends.

  Referring to FIG. 11 again, when the group division process is completed as described above, the specular reflection candidate area detecting unit 131 sequentially loops for all blocks (step S902), and the processes of steps S903 and S904 are performed for all blocks. Do about.

  First, based on the result of the group division process, the specular reflection candidate area detecting unit 131 has a group A and a B group for each block (that is, whether they are A and B blocks). Is determined (step S903).

  If the A group and the B group do not exist (NO in step S903), the specular reflection candidate area detection unit 131 performs the process of step S902 with the next block as the target block.

  On the other hand, when the A group and the B group exist (YES in step S903), the specular reflection candidate area detection unit 131 detects the A group as a specular reflection candidate area including a strong specular reflection component, and the B group is specular. It is detected as a peripheral region that does not contain much reflected light component (step S904).

  As described above, since the A value and the B value are calculated for the first image, the specular reflection candidate area detection unit 131 sets the A value as the 1A value and the B value as the 1B value. Then, the specular reflection candidate area detecting unit 131 uses the 1A value as the reflected light of the light source 200 in the specular reflection candidate area, and the 1B value as the reflected light of the light source 200 in the peripheral area, and the corresponding block of the light source reflected light table described later. Store in the column.

  Note that for a block in which the A group and the B group do not exist, the specular reflection candidate area detection unit 131 determines that the specular reflection area does not exist in the block.

  When the processes of steps S903 and S904 are completed for all blocks, the specular reflection candidate area detection unit 131 determines that the end of all pixel loops has been reached (step S905), and ends the specular reflection candidate area and peripheral area detection processing.

  FIG. 14 is a flowchart for explaining the reflected light extraction processing of auxiliary light shown in FIG.

  When the auxiliary light reflected light extraction processing is started, the reflected light extraction unit 132 performs pixel value group division processing on the second image (step S1101). Here, using the second image as an input image (target image), the reflected light extraction unit 132 divides each pixel into a plurality of groups according to pixel values for each block. Since the group division process in step S1101 is the same as the group division process described in step S901, the description thereof is omitted here.

  Subsequently, the reflected light extraction unit 132 sequentially loops for all blocks (step S1102), and performs the processing of steps S1103 and S1104 for all blocks. Similar to the method described in FIG. 11, the reflected light extraction unit 132 sets the average pixel value of the A group to the A value and the average pixel value of the B group in the second image based on the result of the group division process. Is a B value. Then, the reflected light extraction unit 132 sets the A value of the second image as the 2A value and the B value as the 2B value.

  The reflected light extraction unit 132 extracts the difference obtained by subtracting the 1A value from the 2A value for each block of the first and second images as reflected light data of auxiliary light in the specular reflection candidate area (step SD1103).

  Subsequently, the reflected light extraction unit 132 extracts the difference obtained by subtracting the 1B value from the 2B value as reflected light data of the auxiliary light in the peripheral region (step S1104). Here, it is assumed that the same subject exists in the corresponding blocks in the first and second images, and the increase in pixel value due to the reflected light of the auxiliary light in each pixel is substantially equal. The data is stored in the corresponding block of the data structure shown in FIG.

  FIG. 15 is a diagram illustrating an example of the reflected light data storage area (data structure) provided in the camera according to the second embodiment of the present invention. The reflected light data storage area shown in the figure is defined in the memory 106 shown in FIG. 1, for example.

  The illustrated example has a light source reflected light table for storing light source reflected light data and an auxiliary light reflected light table for storing auxiliary light reflected light data. Both the light source reflected light table and the auxiliary light reflected light table have block numbers. Column, specular reflection candidate region column, and peripheral region column.

  In the light source reflected light table, 1A value and 1B value (that is, light source reflected light data) are recorded in association with these block NO column, specular reflection candidate region column, and peripheral region column, respectively.

  Similarly, in the reflected light reflected light table, (2A value-1A value) and (2B value-1B value) are associated with the block NO column, the specular reflection candidate region column, and the peripheral region column, respectively, and the auxiliary light reflected light data. As recorded.

  When the processing of steps S1103 and S1104 is completed for all blocks, the reflected light extraction unit 132 assumes that the end of all the block loops has been reached (step S1105), and ends the reflected light extraction processing.

  FIG. 16 is a flowchart for explaining the specular reflection area specifying process shown in FIG.

  When the specular reflection area specifying process is started, the specular reflection area specifying unit 133 sequentially loops for all blocks (step S1201), and performs the processing of steps S1202 to S1204 for all blocks.

  First, the specular reflection region specifying unit 133 determines whether or not the specular reflection candidate region and the peripheral region are detected in the target block of the first image based on the result of the specular reflection candidate region and the peripheral region detection process. That is, the specular reflection area specifying unit 133 determines whether the A and B groups exist in the target block of the first image (step S1202).

  If the A and B groups do not exist in the target block of the first image (NO in step S1202), the specular reflection area specifying unit 133 performs the process of step S1202 with the next block as the target block.

  On the other hand, when the A and B groups exist in the target block (YES in step S1202), the specular reflection area specifying unit 133 approximates the pixel values related to the reflected light of the auxiliary light in the specular reflection candidate area and the peripheral area. It is determined whether or not (step S1203).

  Here, when the following equation (26) is satisfied using the predetermined threshold value T3 and the auxiliary light reflected light data shown in FIG. 15, the specular reflection area specifying unit 133 determines the pixel related to the reflected light of the auxiliary light. It is determined that the values are approximate.

| (2A value-1A value)-(2B value-1B value) | <T3 (26)
When the pixel value is expressed in RGB, all of the following formulas (27) to (29) are satisfied using predetermined threshold values T3 _R , T3 _G , and T3 _B instead of formula (13). In this case, the specular reflection region specifying unit 133 determines that the pixel value related to the reflected light of the auxiliary light is approximate.

| (2A value-1A value). R- (2B value-1B value). R | <T3 _R (27)
| (2A value-1A value). G- (2B value-1B value). G | <T3 _G (28)
| (2A value-1A value). B- (2B value-1B value). B | <T3 _B (29)
If the pixel values related to the reflected light of the auxiliary light are not approximated in the specular reflection candidate area and the peripheral area (NO in step S1203), the specular reflection area specifying unit 133 returns to the process in step S1202 and targets the next block. The process of step S1202 is performed as a block.

  On the other hand, when the pixel values related to the reflected light of the auxiliary light approximate in the specular reflection candidate area and the peripheral area (YES in step S1203), that is, when the above formula (26) is satisfied, the specular reflection area specifying unit 133 is satisfied. The specular reflection candidate area and the peripheral area are estimated to be the same subject because the diffuse reflectance of the auxiliary light is substantially equal.

  The specular reflection area specifying unit 133 determines that the specular reflection candidate area is a specular surface because there is a difference between the reflected light of the light source 200 in the specular reflection candidate area and the reflected light of the light source 200 in the peripheral area because the specular reflection area exists. The reflection area is identified (step S1204).

  When Expression (26) is not satisfied, the specular reflection area specifying unit 133 assumes that the specular reflection candidate area and the peripheral area are different subjects, and the specular reflection candidate area is not a specular reflection area.

  Then, the specular reflection area specifying unit 133 stores a flag in a flag storage area, which will be described later, for the block specified as the specular reflection area.

  FIG. 17 is a diagram illustrating an example of a flag storage area (data structure) for storing a flag for a block identified as a specular reflection area.

  The illustrated flag storage area is defined, for example, in the memory 106 shown in FIG. 1A, and a flag table is recorded in this flag storage area. The flag table has a block number column, and a flag is recorded in association with the block identified as the specular reflection area.

  As described above, when the processes of steps S1202 to S1204 are performed for all the blocks, the specular reflection area specifying unit 133 reaches the end of all the block loops (step S1205), and the specular reflection area specifying process ends.

  FIG. 18 is a flowchart for explaining the white balance gain calculation processing shown in FIG.

  When the white balance gain calculation process is started, the WB gain calculation unit 134 sequentially loops for all blocks (step S1301), and performs the processes of steps S1302 and S1303 for all blocks. First, the WB gain calculation unit 134 refers to the flag table shown in FIG. 17 and determines whether or not it is specified that a specular reflection area exists in the target block (step S1302).

  If the specular reflection area does not exist in the target block (NO in step S1202), the WB gain calculation unit 134 performs the processing in step S1202 with the next block as the target block. On the other hand, if the specular reflection area exists in the target block (YES in step S1202), the WB gain calculation unit 134 calculates the difference D by subtracting the 1B value from the 1A value based on the light source reflected light data shown in FIG. To do. Then, the WB gain calculation unit 134 calculates a block WB gain value according to the difference D (step S1303).

  When the pixel value is expressed in RGB, the difference D is expressed in RGB, and the block WB gain value is calculated using the following equations (30) and (31).

Gain. R = D. G / D. R (30)
Gain. B = D. G / D. B (31)
As described above, when the processing of steps S1202 and S1203 is performed for all blocks, the WB gain calculation unit 134 assumes that the end of all block loops has been reached (step S1204), and calculates the average of the block WB balance gains of all blocks as white. The WB gain is calculated (step S1205). Then, the WB gain calculation unit 134 ends the white balance gain calculation process. That is, the WB gain calculation unit 134 ends the light source color estimation process.

  As described above, in the second embodiment of the present invention, the first and second images are divided into a plurality of blocks, and the subject 202 is illuminated with the reflected light emitted from the auxiliary light emitting device 201. The specular reflection area by the light source 200 is specified. As a result, the white balance gain can be calculated using only the specular reflection region. As a result, the accuracy of white balance correction in the image is improved.

  As is clear from the above description, in the example shown in FIG. 1, the system control unit 110 and the specular reflection region candidate detection unit 131 function as specular reflection region detection means, and the system control unit 110 and the reflected light extraction unit 132 reflect. It functions as a light extraction means.

  Further, the system control unit 110 and the specular reflection area specifying unit 133 function as specifying means, and the system control unit 110 and the white balance gain calculating unit 134 function as light source color estimating means. Further, the system control unit 110 functions as an adjustment unit.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least a specular reflection region detection step, a reflected light extraction step, a specifying step, and a light source color estimation step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 103 Image pick-up part 105 Image processing part 106 Memory 107 Memory control part 110 System control part 131 Specular reflection candidate area | region detection part 132 Reflection light extraction part 133 Specular reflection area specification part 134 White balance gain calculation part 201 Auxiliary light emission device

Claims (10)

An image processing apparatus that performs a light source color estimation process for estimating a light source color of the light source in an image obtained by photographing a subject in a photographing environment where a light source exists,
In a first image obtained by photographing the subject with no predetermined auxiliary light emitted, a specular reflection candidate area in which specular reflection by the light source may occur and around the specular reflection candidate area Specular reflection area detecting means for detecting a peripheral area located;
Reflected light extraction means for extracting a reflected light component by the auxiliary light based on the first image and the second image obtained by photographing the subject by emitting the auxiliary light;
A specification for comparing the reflected light components in the specular reflection candidate area and the peripheral area and specifying whether the specular reflection candidate area is a specular reflection area where specular reflection by the light source occurs according to the comparison result Means,
Light source color estimation means for performing the light source color estimation processing using the specular reflection region;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the specular reflection area detection unit sets an area located within a predetermined range from the specular reflection candidate area as the peripheral area.   The specular reflection area detecting means sets the pixels of the first image in order as the target pixels, and the difference between the pixel value of the target pixel and the pixel value of another pixel in a predetermined range is a predetermined first threshold value The image processing apparatus according to claim 1, wherein the target pixel is the specular reflection candidate area.   The specular reflection region detection means sets the target pixel as the peripheral region when a difference between a pixel value of the target pixel and a pixel value of the other pixel is a predetermined threshold value or less. The image processing apparatus according to any one of 1 to 3.   The specifying unit specifies the specular reflection candidate area as the specular reflection area when a difference between a reflected light component in the specular reflection candidate area and a reflected light component in the peripheral area is smaller than a predetermined second threshold. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The light source color estimation means calculates a difference between the reflected light of the light source in the specular reflection area and the reflected light of the light source in the peripheral area as the light source color estimation process, and estimates the light source color according to the difference. The image processing apparatus according to claim 1, wherein the image processing apparatus is performed.   7. The adjusting device according to claim 1, further comprising an adjusting unit that adjusts a position where the auxiliary light is emitted so that a specular reflection region by the light source and a specular reflection region by the auxiliary light do not overlap each other. Image processing apparatus.   The adjustment means is configured such that the difference between the reflected light component in the specular reflection candidate area and the reflected light component in the peripheral area is determined by the direction of irradiation of the auxiliary light according to the comparison result between the hue of the auxiliary light and the auxiliary light. The image processing apparatus according to claim 7, wherein it is determined whether or not the irradiation direction overlaps. A control method for an image processing apparatus that performs a light source color estimation process for estimating a light source color of the light source in an image obtained by shooting a subject in a shooting environment where a light source exists,
In a first image obtained by photographing the subject with no predetermined auxiliary light emitted, a specular reflection candidate area in which specular reflection by the light source may occur and around the specular reflection candidate area A specular reflection region detecting step for detecting a peripheral region located;
A reflected light extraction step of extracting a reflected light component of the auxiliary light based on the first image and a second image obtained by photographing the subject by emitting the auxiliary light; and
A specification for comparing the reflected light components in the specular reflection candidate area and the peripheral area and specifying whether the specular reflection candidate area is a specular reflection area where specular reflection by the light source occurs according to the comparison result Steps,
A light source color estimation step for performing the light source color estimation process using the specular reflection region;
A control method characterized by comprising: A control program used in an image processing apparatus for performing a light source color estimation process for estimating a light source color of the light source in an image obtained by photographing a subject in a photographing environment where a light source exists,
In the computer provided in the image processing apparatus,
In a first image obtained by photographing the subject with no predetermined auxiliary light emitted, a specular reflection candidate area in which specular reflection by the light source may occur and around the specular reflection candidate area A specular reflection region detecting step for detecting a peripheral region located;
A reflected light extraction step of extracting a reflected light component of the auxiliary light based on the first image and a second image obtained by photographing the subject by emitting the auxiliary light; and
A specification for comparing the reflected light components in the specular reflection candidate area and the peripheral area and specifying whether the specular reflection candidate area is a specular reflection area where specular reflection by the light source occurs according to the comparison result Steps,
A light source color estimation step for performing the light source color estimation process using the specular reflection region;
A control program characterized by causing

Images