KR20140083853A - Imaging device imaging method and imaging program thereof - Google Patents

Abstract

An imaging device according to the present invention comprises a flash light mixing ratio calculator (145a) which calculates a flash light mixing ratio based on a flash emission image and a non-flash emission image, a main subject light mixing ratio selector (145b) which selects a flash light mixing ratio of an area of a main subject, a preliminary white balance coefficient determination unit which determines a preliminary white balance coefficient used in adjusting a white balance, a mixing ratio effective range calculator (145c) which calculates an upper limit value and a lower limit value of an effective range of the flash light mixing ratio based on the light mixing ratio of the main subject and the preliminary white balance coefficient, a flash light mixing ratio resetting unit which resets the flash light mixing ratio so that the flash light mixing ratio can be in the calculated effective range, and a final white balance coefficient calculator (145d) which calculates a final white balance coefficient based on the reset flash light mixing ratio.

Description

[0001] IMAGING DEVICE IMAGING METHOD AND IMAGING PROGRAM [0002]

The present invention relates to an imaging apparatus imaging method and an imaging program.

BACKGROUND ART Conventionally, it has been known to adjust white balance in order to express white color naturally in an image obtained by imaging under various light sources including ambient light (external light) or flash light such as a camera. By adjusting the white balance, the difference of the color temperature of the subject affected by various light sources is adjusted, and the appropriate color tone is naturally expressed according to the light source for each image. This function is called the Auto White Balance (AWB) function.

As a technique related to this, a final WB coefficient value (hereinafter referred to as " WB coefficient value ") is calculated in consideration of the flash light mixing ratio indicating the degree of contribution of the flash light to the pixel value based on the white balance coefficient value Hereinafter referred to as a final WB coefficient value) (see, for example, Patent Document 1). In this technique, first, the preliminary WB coefficient value for external light relating to only the external light when the flash light is not used by the AWB function for a captured image (hereinafter referred to as flash non-flash image) without subjecting the flash light to the subject is acquired do. Further, a value based on the specifications of the device, an actual value or the like is used for the preliminary WB coefficient value for flash light with respect to only the flash light not including the external light.

Next, the flash light mixing ratio is calculated. The flash light mixing ratio can be calculated by comparing the flash non-emission image and the corresponding pixels of a captured image (hereinafter referred to as a flash light emission image) by irradiating the subject with flash light. When a camera-shake or a change in external light occurs, inconsistency occurs in the corresponding pixel between the flash emission image and the flash non-emission image, and an erroneous flash mixing ratio may be calculated.

Here, the technique of Patent Document 1 applies a low pass filter (LPF) to the flash light mixing ratio for each pixel. Since the LPF is applied to each pixel, even when there is a pixel whose flash light mixing ratio value becomes abnormal due to a camera shake or a change in external light, the change of the flash light mixing ratio in the contrast with the flash light mixing ratio of the surrounding pixels is gradual . That is, a steep change in flash light mixing ratio over the entire image is suppressed.

Finally, the final WB coefficient value is calculated using the flash light mixing ratio adjusted to moderate the change. By doing so, an appropriate final WB coefficient value corresponding to the contribution of the flash light can be obtained, and the change in the final WB coefficient value between the pixels can be smoothed. By using white balance adjustment as the final WB coefficient value, it is possible to reduce the out of color registration according to the use of the flash light mixing ratio which rapidly varies between the pixels.

However, as in the technique described in Patent Document 1, it may not be sufficient to suppress the steep change in the flash light mixing ratio to be gentle by applying the LPF to the flash light mixing ratio. This is because the upper and lower limits of the value of the flash light mixing ratio are not set even if the change of the flash light mixing ratio is gentle. Therefore, the position of the corresponding pixel is shifted due to camera shake or the like, and a false flash light mixing ratio is calculated, so that an abnormally high or low flash light mixing ratio may be calculated. The final WB coefficient value calculated on the basis of the flash light mixing ratio of the abnormal value up to now is also not appropriate and even if the white balance adjustment is performed using this final WB coefficient value, the pixel positional deviation eventually occurs.

It is an object of the present invention to provide an imaging device imaging method capable of calculating an optimum final WB coefficient value by limiting the numerical effective range of the flash light mixing ratio in consideration of the content of an image And an image pickup program.

According to an aspect of the present invention, there is provided an imaging apparatus including a flash emission image picked up using flash light, and an acquisition unit acquiring a flash non-emission image picked up by external light without using flash light, Calculating a flash light mixing ratio that calculates a flash light mixing ratio indicating a degree to which the flash light contributes to the pixel value of the flash light image based on the flash light image and the flash non- A local flash light mixing ratio selector for selecting the flash light mixing ratio for the corresponding part of the flash light emission image and the corresponding part of the non-flash light emission image as a local flash light mixing ratio; A preliminary white balance coefficient that determines the preliminary white balance coefficient for light and ambient light A mixing ratio effective range calculating unit for calculating an upper limit value and a lower limit value of the effective range of the flash light mixing ratio on the basis of the selected local flash light mixing ratio and the determined preliminary white balance coefficient, A flash light mixing ratio resetting unit for resetting the flash light mixing ratio calculated by the flash light mixing ratio calculating unit and a final white balance coefficient calculating unit for calculating a final white balance coefficient to be applied to the flash light based on the reset flash light mixing ratio And a final white balance coefficient calculating unit.

Further, the blending ratio effective range calculating section may set the local flash light mixing ratio as the upper limit value.

The blending ratio effective range calculating section may calculate a lower limit value of the flash light mixing ratio based on a provisional value obtained by dividing the flash white light preliminary white balance coefficient by the preliminary white balance coefficient for external light .

According to an embodiment of the present invention, there is provided an imaging method including the steps of: (a) acquiring a flash light emission image captured by using a flash light and a flash non-light emission image captured by external light without using flash light; Calculating a flash light mixing ratio indicating a degree to which the flash light contributes to a pixel value of the flash light image based on the obtained flash light image and the non-flash non-light image; (c) Selecting a flash light blending ratio for a corresponding part of the area of the flash non-luminescent image as a local flash light blending ratio; (d) determining a preliminary white balance coefficient for the flash light and ambient light for adjustment of white balance (E) selecting, based on the selected local flash light mixture ratio and the determined preliminary white balance coefficient, (F) resetting the calculated flash light mixing ratio to be within the calculated effective range, and (g) resetting the calculated flash light mixing ratio based on the reset flash light mixing ratio And calculating a final white balance coefficient to be applied to the flash light emission image.

In addition, the local flash light mixing ratio selected in the step (c) may be set as an upper limit value of the flash light mixing ratio calculated in the step (e).

(H) obtaining a provisional value by dividing the flash white light preliminary white balance coefficient by the preliminary white balance coefficient for the external light, and (i) calculating a lower limit value of the flash light blend ratio based on the provisional value The method comprising the steps of:

Further, in a non-volatile readable medium storing computer executable program code according to an embodiment of the present invention, the method executed by the program code comprises the steps of: (a) generating a flash light emission image And obtaining a non-flash non-emission image captured by the external light without using flash light, (b) calculating a flash non-emission image based on the obtained flash light emission image and the non-flash non- (C) selecting the flash light mixing ratio for a region of a corresponding part of the flash light emission image and the non-flash light emission image as a local flash light mixing ratio , (d) determining a preliminary white balance coefficient for the flash light and the ambient light for adjusting the white balance (E) calculating an upper limit value and a lower limit value of the effective range of the flash light mixing ratio based on the selected local flash light mixture ratio and the determined preliminary white balance coefficient, (f) And (g) calculating a final white balance coefficient to be applied to the flash emission image based on the reset flash light mixing ratio.

In addition, the local flash light mixing ratio selected in the step (c) may be set as an upper limit value of the flash light mixing ratio calculated in the step (e).

(H) obtaining a preliminary value by dividing the preliminary white balance coefficient for flash light by the preliminary white balance coefficient for the external light to obtain a preliminary white balance coefficient; (i) And calculating a lower limit value of the flash light mixing ratio.

According to the present invention, the optimum final WB coefficient value can be calculated by limiting the numerical effective range of the flash light mixing ratio in consideration of the content of the image.

1 is a block diagram showing the configuration of an image pickup apparatus according to an embodiment of the present invention.
2 is a block diagram showing a configuration of a WB correction processing section according to the present embodiment.
3 is a flowchart showing a method of processing by the WB correction processing section according to the present embodiment.
4 is a flowchart showing a processing method for obtaining the upper limit value and the lower limit value of the flash light mixing ratio by the WB correction processing unit according to the present embodiment.
5 is a flowchart showing a processing method for calculating the final WB coefficient value by the WB correction processing unit according to the present embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a block diagram showing the configuration of an image pickup apparatus according to an embodiment of the present invention. 1, an image pickup apparatus 1 including a digital still camera or the like mainly includes a lens 2, a diaphragm 4, a shutter 6, a CCD / CMOS (Charge Coupled Device / Compression Metal Oxide Semiconductor) 8, a signal processing circuit 10, a preprocessing circuit 12, a post-processing circuit 14, a conversion circuit 16, a data compression circuit 18, a memory card interface 20, a memory card 22, An AE (Automatic Exposure) processing unit 26, an exposure control circuit 28, and a flash light emitting unit 30. [ The configuration of these will be described in detail below.

The lens 2 receives light from the object. The diaphragm 4 limits the light incident through the lens 2. The shutter 6 receives the light incident through the diaphragm 4 in accordance with the imaging exposure time.

The CCD / CMOS 8 is an imaging plate for converting an optical signal into an electrical signal. That is, the CCD / CMOS 8 as an acquiring unit receives light from a subject incident through the aperture stop 4 and the shutter 6, and photoelectrically converts the received light. The CCD / CMOS 8 transmits an electrical signal as sensed image data obtained by photoelectric conversion to the signal processing circuit 10.

In the present embodiment, the object refers to all objects including the main object and the background in the captured image generated by photoelectric conversion by the CCD / CMOS 8, and the main object refers to a person, Or something.

In this embodiment, it is assumed that image data of two captured images are acquired in one imaging operation. One of the captured images is a flash emission image picked up by using the flash 30 and another picked up image is a flash non-emission image picked up without using the flash 30. These images are captured at almost the same time in an extremely short time.

The signal processing circuit 10 has a color classification circuit 101 and an A / D (Analog / Digital) conversion section 102 and converts electrical signals for each pixel into red (R), blue (B) Or A / D conversion is performed.

The pre-processing circuit 12 includes a black level correction unit 121, a defective pixel correction unit 122, a shading correction unit 123, an AE evaluation calculation unit 124, and an AWB evaluation calculation unit 125.

The black level correcting unit 121 corrects the darkest part (black level) in the pixel to be constant so as to express the accurate color or brightness.

The defective pixel correction unit 122 corrects the pixel value based on the pixel value of the surrounding pixels with respect to the pixel whose pixel value is not normal due to a defect of the CCD / CMOS 8 or the like.

The shading correction unit 123 corrects the captured image so as to have a uniform brightness throughout, thereby eliminating the luminance unevenness.

The AE evaluation calculator 124 calculates the AE evaluation value indicating the brightness by integrating the luminance value in the predetermined area in the captured image. The calculated AE evaluation value is output to the AE processing unit 26. [

The AE processing section 26 determines parameters such as the shutter speed iris and the ISO sensitivity based on the AE evaluation value calculated by the AE evaluation calculation section 124 and transmits these parameters to the exposure control circuit 28. [ The exposure control circuit 28 controls the diaphragm 4, the shutter 6, the CCD / CMOS 8, and the flash 30 based on the received parameters, respectively.

The AWB evaluation calculation section 125 integrates each signal relating to R G and B to calculate an AWB evaluation value and transmits the calculated AWB evaluation value to the AWB processing section 23. In particular, the AWB evaluation calculator 125 calculates the AWB evaluation value for the flash non-emission image.

The AWB processing section 23 determines a preliminary WB coefficient value for performing AWB processing on the captured image as the preliminary white balance coefficient determination section. Specifically, the AWB processing unit 23 calculates the preliminary WB coefficient value based on the AWB evaluation value calculated by the AWB evaluation calculation unit 125 described above with respect to the flash non-emission image, calculates the preliminary WB coefficient value by the WB coefficient for external light Value.

On the other hand, the AWB processing section 23 determines a specified value based on the specification of the image capture device 1 or an actually measured value for the flash light as the flash WB coefficient. These specified values or measured values are stored and held in advance by the AWB processing section 23. The value determined as the preliminary WB coefficient value for external light and the preliminary WB coefficient value for flash light is transmitted to the WB correction processing section 145, which will be described later, to calculate the final WB coefficient value.

The captured image data processed by the preprocessing circuit 12 is appropriately stored in the image memory 13. [ The image memory 13 stores a group of local image patterns that are statistically analyzed for noise reduction processing of the captured image, for example.

The post-processing circuit 14 receives the sensed image data transmitted from the preprocessing circuit 12 or reads sensed image data stored in the image memory 13, and executes post-processing on the image data of the preprocessed sensed image. 1, the post-processing circuit 14 includes a demosaicing processing section 141, an edge emphasis processing section 142, a color correction processing section 143, a gamma correction processing section 144, a WB correction processing section 145, And a noise reduction processing unit 146.

The demosaicing processing section 141 estimates the color of R, G or B, which is insufficient compared with the surrounding pixels, from the image data of each pixel, and compensates for the deficiency from the color in the surrounding pixels.

The edge emphasis processing section 142 forms a sharp image by detecting the outline in the captured image data and sharpening the slope of the color change in the outline.

The color correction processing section 143 corrects the color of each pixel in the captured image.

The gamma correction processing unit 144 corrects the pixel value of the sensed image data so that the sensed image data is displayed with a higher luminance in a small display (not shown) illuminating an image for user confirmation or an externally connected display.

The WB correction processing section 145 corrects the preliminary WB coefficient value transmitted from the above-described AWB processing section 23 to a more appropriate value. A detailed description will be given later with reference to Figs. 2 to 5.

The noise reduction processing unit 146 reduces the image data noise in the captured image due to the above-described heat generation of the CCD / CMOS 8 or the like.

The conversion circuit 16 receives the picked-up image data processed by the post-processing circuit 14 and converts the image data from the RGB color space to the YCbCr color space.

The image data converted by the conversion circuit 16 is compressed by the data compression circuit 18 and stored in the memory card 22 through the memory card interface 20. [

Each of the above-described units of the image pickup apparatus 1 performs each function by a CPU (Central Processing Unit) executing a program stored in a storage unit such as a flash memory.

Next, the functional configuration of the WB correction processing unit 145 will be described in detail.

2 is a block diagram showing a configuration of a WB correction processing section according to the present embodiment.

2, the WB correction processing section 145 includes a flash light mixing ratio calculating section 145a, a main subject mixture ratio selecting section 145b, a mixing ratio effective range calculating section 145c, a WB coefficient calculating section 145d And a WB adjusting unit 145e.

The flash light mixing ratio calculating unit 145a acquires the sensed image data that has been preprocessed by the preprocessing circuit 12. [ As described above, the flash light emission image and the non-flash light emission image are captured at once, and the image data for both images are acquired as the image data A and the image data B by the flash light mixture ratio calculating section 145a.

The flash light mixing ratio calculating unit 145a compares the image data A and the image data B to calculate the flash light mixing ratio indicating the degree to which the flash light contributes to the pixel value of the flash light image. Specifically, the image data A and the image data B are divided into corresponding regions, the average of the pixel values is calculated for each region, and the flash light mixing ratio is calculated based on the ratio of the average value of the image data A and the average value of the image data B . The calculation of the flash light mixing ratio for each region will be described later in detail with reference to Equation (1).

The main subject mixture ratio selection unit 145b sets the flash light mixture ratio in the main subject in the captured image. The main object blending ratio selector 145b obtains information about a region representing the main subject as a local flash light blending ratio selector and selects the flash light blending ratio in the region of the main object as the main body blending ratio. The area of the main subject is the area focused by the area auto focus function of the subject set by the user so as to track the area designated by the user in an input unit (not shown) such as a touch panel.

In the present embodiment, since the area of the main subject is specified by user selection, the flash light mixing ratio of the main subject area can be calculated. Specifically, the flash light mixing ratio of the main subject is calculated from the average of the pixel values in the corresponding main subject areas of the video data A and the video data B, and the calculated value is set as the main subject mixture ratio. The calculation of the flash light mixing ratio for each region will be described later with reference to Equation (1).

The blending ratio effective range calculating section 145c calculates the numerical effective range (upper limit value and lower limit value) of the flash light blending ratio. More specifically, the blending ratio effective range calculating section 145c calculates the blending ratio effective range calculating section 145c based on the preliminary WB coefficient values for external light and the preliminary WB coefficient values for flash light obtained from the AWB processing section 23, The upper limit value and the lower limit value of the flash light mixing ratio are calculated based on the ratio. Details of the calculation method of the upper limit value and the lower limit value will be described later. In the present embodiment, the external light is light other than the flash light under the environment of the image pickup such as the light of the lighting device or the sunlight.

The WB coefficient calculating unit 145d obtains the preliminary WB coefficient value for flash light and the WB coefficient value of the flash light from the AWB processing unit 23 as the preliminary WB coefficient value for the external light for the flash non-light emission image. Further, the WB coefficient calculating section 145d obtains the flash light mixing ratio from the above-described flash light mixing ratio calculating section 145a as the flash light mixing ratio resetting section and the last white balance coefficient calculating section, and the mixing ratio effective range calculating section 145c The upper limit value and the lower limit value of the flash light mixing ratio are obtained. The WB coefficient calculator 145d calculates the final WB coefficient value (final WB coefficient value) to optimally adjust the white balance in the flash emission image based on the obtained values. Details will be described later with reference to Figs.

The WB adjusting unit 145e adjusts the white balance of the flash light image by multiplying the pixel value of each pixel of the flash light emitting image by the final WB coefficient value obtained from the WB coefficient calculating unit 145d.

Next, the operation of the WB correction processor 145 according to the present embodiment will be described in detail with reference to FIG. 3 to FIG. FIG. 3 is a flowchart showing a processing method of the WB correction processing section according to the present embodiment. FIG. 4 is a flowchart showing a method of processing for obtaining the upper limit value and the lower limit value of the flash light mixing ratio by the WB correction processing section according to the present embodiment, 5 is a flowchart showing a method of processing for calculating the final WB coefficient value by the WB correction processing unit according to the present embodiment. The processing shown in Figs. 3 to 5 can also be executed every imaging using the flash 30. The final WB coefficient value is hereinafter simply referred to as the final WB coefficient value.

As shown in FIG. 3, first, the flash light mixture ratio calculating unit 145a acquires image data (step S1). In this step, the flash light mixing ratio calculating unit 145a calculates the flash light blending ratio calculating unit 145a based on the image data (image data A) of the flash light emission image captured by using the flash 30 and the image of the flash non-light emission image (Video data B).

Subsequently, the WB coefficient calculating section 145d obtains the preliminary WB coefficient value for external light (step S2). The WB coefficient value obtained at this stage is a value calculated by the AWB function for the flash non-emission image and determined as the preliminary WB coefficient value for the external light. The calculation of the preliminary WB coefficient values for the flash non-luminescent image will not be described based on well-known methods.

Subsequently, the WB coefficient calculating section 145d obtains the preliminary WB coefficient value for flash light (step S3). The WB coefficient value obtained at this stage is a value determined as the preliminary WB coefficient value for flash light by the AWB processing section 23 as described above.

Then, the flash light mixing ratio calculating section 145a calculates the flash light mixing ratio (step S4). In this step, the flash light mixing ratio calculating unit 145a calculates the flash light mixing ratio based on the flash light emission image and the flash non-light emission image obtained in step S1.

Specifically, the flash light mixing ratio calculating unit 145a divides the flash light image and the non-flash light image equally into equal areas, and calculates the flash light mixing ratio for each area. For example, the flash light mixing ratio calculating section 145a divides the flash light emission image and the flash non-light emission image into four 2x2 regions.

The flash light mixing ratio calculating unit 145a calculates the average value of the pixel values of the green pixels for the corresponding regions of the flash light emission image and the flash non-light emission image. From the average pixel value G _A of the green calculated from the image data A of the flash light emitting image and the average pixel value G _B of the green calculated from the image data B of the flash non-light emitting image, the flash light mixing ratio (Ratio ).

The flash light mixing ratio (Ratio) calculated for each area is output to the main object mixture ratio selection unit 145b. Further, the flash light mixing ratio (Ratio) is reset to the final flash light mixing ratio for each region in a later step S7.

Subsequently, the main subject mixture ratio selection unit 145b sets the flash light mixture ratio in the main subject in the captured image as the main subject mixture ratio (step S5). Specifically, the main subject mixture ratio is calculated by using the pixel value (or an average value) of the pixels included in the main subject area selected as described above and the equation (1). Here, the calculated main subject mixture ratio is output to the blending ratio effective range calculation unit 145c for calculating the effective range of the flash light blending ratio for the flash light image in the following step S6.

Subsequently, the blending ratio effective range calculating section 145c calculates the flash light blending ratio effective range (step S6). In this step, the blending ratio effective range calculating unit 145c calculates the numerical effective range (upper limit value and lower limit value) of the flash light mixing ratio with respect to the flash light emission image. The upper limit value and the lower limit value of the flash light mixing ratio are calculated based on the preliminary WB coefficient values for the external light and the preliminary WB coefficient for flash light obtained in steps S2 and S3 and the main object mixture ratio set in step S5. Details of the setting of the effective range of the flash light mixing ratio will be described later with reference to Fig.

Subsequently, the blending ratio effective range calculating section 145c clips the flash light blending ratio (step S7). In this step, the blending ratio effective range calculating unit 145c calculates the blending ratio of the flash light for each 2x2 area calculated by the flash light blending ratio calculating unit 145a to the upper limit value and the lower limit value To the value within the effective range between " 0 " That is, the flash light mixing ratio is reset to be between the upper limit value and the lower limit value with respect to the flash light mixing ratio of each area. For example, when the flash mixing ratio is larger than the upper limit value, the upper limit value is set; when the flash mixing ratio is lower than the lower limit value, the lower limit value is set. If the flash light blending ratio is within the effective range, the value is not changed as it is. The value of the final flash light mixing ratio within the effective range thus obtained is output to the WB coefficient calculation section 145d for calculation of the final WB coefficient value.

Subsequently, the WB coefficient calculating section 145d calculates the final WB coefficient value (step S8). In this step, based on the preliminary WB coefficient values acquired in steps S2 and S3 and the flash light mixture ratio reset in step S7, the WB coefficient calculation section 145d calculates the final WB coefficient value for each area. Details of calculation of the final WB coefficient value will be described later with reference to Fig.

Then, the WB adjusting unit 145e adjusts the white balance for the flash light image (step S9). In this step, the WB adjusting unit 145e adjusts the white balance of the flash light image by multiplying the pixel value of each pixel in the corresponding region of the flash light emission image by the final WB coefficient value for each region calculated in the previous step S8 . The white balance adjusted flash emission image is output from the WB adjustment unit 145e. Thereafter, the processing by the WB correction processing section 145 is terminated.

Hereinafter, a method of obtaining the upper limit value and the lower limit value of the flash light mixing ratio (step S6) and a method of calculating the final WB coefficient value (step S8) according to the present embodiment will be described in detail with reference to FIGS.

First, a method of obtaining the upper limit value and the lower limit value of the flash light mixing ratio will be described with reference to FIG.

As shown in Fig. 4, the blending ratio effective range calculating section 145c first sets the upper limit value of the flash light blending ratio (step S11). In this step, the blending ratio effective range calculating unit 145c sets the main object blending ratio set in step S5 as the upper limit value (UU) of the flash light blending ratio.

Then, the blending ratio effective range calculating section 145c calculates the lower limit value LL of the flash light blending ratio. Therefore, the blending ratio effective range calculating section 145c first calculates the provisional values rg and b (step S12). Rgb provisional value is the ratio of the flash-light pre-WB coefficient value obtained in step S3 to the external light pre-WB coefficient value _(E WB) for obtained in step S2 for each color of RGB (WB _F). The provisional value is calculated by the following equation (2).

Where WBR _F WBG _F and WBB _F are the preliminary WB coefficient values for flash light of each color of RGB, and WBR _E WBG _E and WBB _E are the preliminary WB coefficient values of the external light for each color.

Subsequently, the blending ratio effective range calculating section 145c calculates the WB coefficient value for the main subject (step S13). In this step, the WB coefficient value corresponding to the flash light mixing ratio (i.e., the upper limit value (UU) of the flash light mixing ratio) of the main subject is calculated to obtain an appropriate WB coefficient value for the main subject. More specifically, the WB coefficient values WBR _UU and WBB _UU of red and blue corresponding to the upper limit value UU of the flash light mixing ratio are calculated by the following equation (3).

Also, when the white balance is adjusted by red and blue based on green as in the present embodiment, the WB coefficient value of the green for the flash emission image and the flash non-emission image may be corrected to WBG _F = WBG _E.

Subsequently, the mixing ratio effective range calculating section 145c calculates the maximum value and the minimum value of the WB coefficient values (step S14). In this step, the preliminary WB coefficient value for flash light and the WB coefficient corresponding to the upper limit value (UU) of flash light mixing ratio are calculated based on the magnitude relationship between the provisional values rg and b calculated using Equation (2) The maximum and minimum values are determined for each value. That is calculated from the maximum value MAX _E and the minimum value MIN _E flash-light pre-WB coefficient value WBR _F WBB _F WBG _F of the maximum value MAX _F and the minimum value MIN _F Equation (2) of the external light pre-WB coefficient value WBR _E WBB _E WBG _E for to the WBR WBB _{UU UU UU} WBG of the maximum value mAX and the minimum value MIN _{UU UU} is determined from equation (4).

Subsequently, the mixing ratio effective range calculating section 145c sets a threshold value of the chroma difference (step S15). In this step, the mixing ratio effective range calculating section 145c sets the threshold value S _Lim of the chrominance difference S to set the lower limit value LL for the upper limit value UU of the flash light mixing ratio. As will be described later, the saturation difference S is a value indicating the irregularity of the WB coefficient value corresponding to the upper and lower limit values of the main subject flash light mixing ratio. The irregularity of the WB coefficient value can be limited by setting the threshold value S _Lim to the saturation difference S as described later.

The threshold value S _Lim of the chroma difference can be set in advance to a value such as "0.35 " and stored in the storage unit and read out by the mixing ratio effective range calculation unit 145c. The threshold value may be a value that is set by the user every time.

Here, the saturation difference (S) is the WB coefficients (WB _UU) corresponding to the main subject in the upper limit value (UU) of the flash light mixing ratio to the WB coefficients (WB _LL) corresponding to the lower limit value (LL) of the flash light mixing ratio Is a value based on a ratio. In other words, the saturation difference (S) represents the irregularity of the WB coefficient value corresponding to the upper and lower limit values of the main subject flash light mixing ratio. The detailed definitions are as shown in the following expressions (5) and (6).

The maximum value MIN (r, g, b) among r, g, and b is a minimum value among r, g, and b .

At this stage, the WB coefficient value WB _LL corresponding to the lower limit value LL of the flash light mixing ratio is unknown and therefore the chrominance difference S is unknown. However, as described below, the threshold value of the chrominance difference S S _Lim ) to calculate the lower limit value LL.

Subsequently, the blending ratio effective range calculating section 145c calculates a lower limit value of the flash light blending ratio (step S16). In this step, the WB coefficient value for the flash light, the WB coefficient value for the external light, and the maximum value (MIN) and the threshold value (S _Lim ) of the chrominance difference among the WB coefficient values of the main subject, The lower limit value LL of the flash mixing ratio is calculated by the following equation (7).

Thus, in the present embodiment, the WB coefficient value corresponding to the upper limit value UU of the flash light mixing ratio (i.e., the main subject flash light mixing ratio) is calculated first.

Further, the lower limit value LL of the flash light mixing ratio is calculated by using a saturation difference threshold value which acts to limit the irregularity of the flash light mixing ratio and the preliminary WB coefficient value including the WB coefficient value. That is, the effective range of the flash light mixing ratio can be obtained. Since the effective range of the flash light mixing ratio is determined based on the flash light mixing ratio of the main subject, the flash light mixing ratio based on the main subject can be reset.

Thereafter, the process proceeds to step S7 described above.

Next, a processing method for calculating the final WB coefficient value in step S8 will be described.

As shown in Fig. 5, first, the WB coefficient calculating section 145d calculates a color balance value (step S21). In this step, the WB coefficient calculating unit 145d sets the values of the preliminary WB coefficients determined in steps S2 and 3, respectively, to the external light and the flash light, respectively. By doing so, the preliminary color balance value CB _F of the light source color of the flash light and the preliminary color balance value CB _E of the external light are calculated for each color of RGB.

Then, the WB coefficient calculating section 145d calculates the mixed color balance value of each color (step S22). In this step, the WB coefficient calculating unit 145d calculates the WB coefficient based on the final flash light mixing ratio (Ratio ') obtained in step S7 and the preliminary color balance values (CB _F ) and (CB _E ) And the balance value (CB _Mix ) is calculated. Specifically, the WB coefficient calculating unit 145d calculates a mixed color balance value (CB _Mix ) for each color of RGB by taking the color balance of the external light and the flash light into consideration by the following expression (8).

Here, i and j are variables of a natural number for specifying a region in the captured image.

Subsequently, the WB coefficient calculating section 145d calculates the final WB coefficient (step S23). In this step, the WB coefficient calculation unit 145d calculates the final WB coefficient value in each area by inverting the mixed color balance value (CB _Mix ) of each color calculated in step S22. More specifically, the final WB coefficient value WB _Mix is calculated by the following equation (9).

Then, the WB coefficient calculation section 145d normalizes the final WB coefficient value (step S24). In this step, the WB coefficient values are normalized so that the minimum value of the WB coefficient values for all areas and colors calculated in step S23 is 1.0. The normalized WB coefficient value is output to the WB coefficient correction unit 145e.

Thereafter, the process proceeds to the above-described step S9.

As described above, according to the present embodiment, the numerical effective range of the flash light mixing ratio is set in consideration of the local flash light mixing ratio calculated by the region of the main subject (a part of the entire image). Therefore, even if the blurred flash light mixing ratio is actually obtained due to a change in hand movement or external light, clipping can be performed at least within the numerical effective range, and the flash light mixing ratio becomes an appropriate value.

By using such a flash light mixing ratio, the final WB coefficient value becomes appropriate and the out-of-color registration of the pixel can be effectively alleviated. In particular, since the upper and lower limit values of the flash light mixing ratio in consideration of the main subject area are set, the final WB coefficient value based on the main subject can be calculated without damaging the flash light mixing ratio with respect to the main subject. Can be ensured.

As a result, an image that can be satisfied by the user can be obtained. According to the embodiment described above, the white balance adjustment for the main subject can be optimized and the optimum white balance adjustment can be performed over the entire image.

Although the embodiment of the present invention has been described above, it goes without saying that various modifications are possible within the scope of the technical idea described in the claims of the present application.

For example, although it has been described that the flash light mixing ratio is calculated for every 2 占 divided areas, it is not limited thereto. The divided area may be an arbitrary area, or may be a color filter array (CFA) unit instead of the area.

The WB coefficient WBUU corresponding to the upper limit value UU of the flash light mixing ratio is calculated as _WBUU corresponding to the lower limit value LL, The coefficient value WB _LL may be calculated. More specifically, the WB coefficient value (WB _LL ) corresponding to the lower limit value can be additionally calculated by substituting LL into the calculated lower limit value LL in place of UU in the right side of the equation (2). For this reason, the saturation difference S may be calculated for verification or the like by using Equations (5) and (6).

Also, the preliminary WB coefficient value or the final WB coefficient value of each color of RGB can be set in any manner. For example, the WB coefficient value WBG of green may be fixed at 1.0 for the flash light image and the flash non-light image, and R and B may be changed to adjust the white balance.

The means and method for performing various processes in the image pickup apparatus according to various embodiments of the present invention described above can be realized by at least one of a dedicated hardwired circuit or a programmed computer. The program may be provided by, for example, a computer-readable recording medium such as a memory stick and a CD-ROM, or may be provided online via a network such as the Internet. In this case, a program recorded on a computer-readable recording medium is usually transferred to a storage unit such as a hard disk and stored. The program may be provided as a single application software, or may be included in the software of the apparatus as one function of the imaging system.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

1: Image pickup device 2: Lens
4: Aperture 6: Shutter
8: CCD / CMOS (acquiring unit) 10: signal processing circuit
101: color discrimination circuit 102: A / D conversion section
12: preprocessing circuit 121: black level correction section
122: defective pixel correction unit 123: shading correction unit
124: AE evaluation calculation unit 125: AWB evaluation calculation unit
13: image memory 14: post-processing circuit
141: demosaicing processing unit 142: edge emphasis processing unit
143: color correction processing unit 144: gamma correction processing unit
145: WB correction processing section 145a: flash light mixing ratio calculating section
145b: Main object blending ratio selection unit (local flash light blending ratio selection unit)
145c: Mixing ratio effective range calculating section
145d: WB coefficient calculating unit (final white balance coefficient calculating unit for flash light mixing ratio resetting unit)
145e: white balance adjustment unit 146: noise reduction processing unit
16: conversion circuit 18: data compression circuit
20: memory card interface 22: memory card
23: AWB processing section (preliminary white balance coefficient determination section)
26: AE processing unit 28: exposure control circuit
30: Flash

Claims (9)

An acquiring unit for acquiring a flash light emission image captured using flash light and a flash non-light emission image captured by external light without using flash light;
A flash light mixing ratio calculating unit for calculating a flash light mixing ratio indicating a degree to which the flash light contributes to a pixel value of the flash light image based on the flash light image and the flash non-light image obtained by the obtaining unit;
A local flash light mixing ratio selector for selecting the flash light mixing ratio for a corresponding part of the flash light emission image and the non-flash light emission image as a local flash light mixing ratio;
A preliminary white balance coefficient determiner for determining a preliminary white balance coefficient for the flash light and the external light for adjustment of white balance;
A mixing ratio effective range calculating unit for calculating an upper limit value and a lower limit value of the effective range of the flash light mixing ratio based on the selected local flash light mixing ratio and the determined preliminary white balance coefficient;
A flash light mixing ratio resetting unit for resetting the flash light mixing ratio calculated by the flash light mixing ratio calculating unit so as to be within the calculated effective range; And
And a final white balance coefficient calculating unit for calculating a final white balance coefficient to be applied to the flash light emission image based on the reset flash light mixing ratio. The method according to claim 1,
The mixing ratio effective range calculating unit may calculate,
And sets the local flash light mixing ratio as the upper limit value. 3. The method according to claim 1 or 2,
The mixing ratio effective range calculating unit may calculate,
Wherein the lower limit value of the flash light mixing ratio is calculated based on a provisional value obtained by dividing the flash white light preliminary white balance coefficient by the preliminary white balance coefficient for the external light. (a) acquiring a flash light image captured by using flash light and a flash non-light image captured by external light without using flash light;
(b) calculating a flash light mixing ratio indicating a degree to which the flash light contributes to the pixel value of the flash light emission image, based on the obtained flash light emission image and the flash non-light emission image;
(c) selecting the flash light mixing ratio as a local flash light mixing ratio for a region of a corresponding part of the flash light emission image and the flash non-light emission image;
(d) determining a preliminary white balance coefficient for the flash light and ambient light for adjustment of white balance;
(e) calculating an upper limit value and a lower limit value of the effective range of the flash light mixing ratio based on the selected local flash light mixture ratio and the determined preliminary white balance coefficient, respectively;
(f) resetting the calculated flash light mixing ratio to be within the calculated effective range; And
(g) calculating a final white balance coefficient to be applied to the flash light emission image based on the reset flash light mixing ratio. 5. The method of claim 4,
The local flash light mixing ratio selected in the step (c)
Is set as an upper limit value of the flash light mixing ratio calculated in the step (e). The method according to claim 4 or 5,
(h) obtaining a provisional value by dividing the flash white light preliminary white balance coefficient by the preliminary white balance coefficient for external light; And
(i) calculating a lower limit value of the flash light mixing ratio based on the provisional value. A non-transiently readable medium storing computer executable program code, the method being executed by the program code,
(a) acquiring a flash light image captured by using flash light and a flash non-light image captured by external light without using flash light;
(b) calculating a flash light mixing ratio indicating a degree to which the flash light contributes to the pixel value of the flash light emission image, based on the obtained flash light emission image and the flash non-light emission image;
(c) selecting the flash light mixing ratio as a local flash light mixing ratio for a region of a corresponding part of the flash light emission image and the flash non-light emission image;
(d) determining a preliminary white balance coefficient for the flash light and ambient light for adjustment of white balance;
(e) calculating an upper limit value and a lower limit value of the effective range of the flash light mixing ratio based on the selected local flash light mixture ratio and the determined preliminary white balance coefficient, respectively;
(f) resetting the calculated flash light mixing ratio to be within the calculated effective range;
(g) calculating a final white balance coefficient to be applied to the flash emission image based on the reset flash light mixing ratio. The method of claim 7, wherein
The local flash light mixing ratio selected in the step (c)
Is set as an upper limit value of the flash light mixing ratio calculated in the step (e). The method according to claim 7 or 8, wherein
The method being executed by the program code,
(h) obtaining a provisional value by dividing the flash white light preliminary white balance coefficient by the preliminary white balance coefficient for external light; And
(i) calculating a lower limit value of the flash light mixing ratio based on the provisional value.

Images