JP5566679B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method.

  In an electronic imaging apparatus such as a digital still camera, the conventional white balance correction process is a process that uniformly applies one white balance gain to the entire image.

  For example, in a scene where the illumination light (environment light) is dark, shooting is performed by emitting a flash in order to eliminate insufficient exposure. When shooting with the flash fired, the white balance gain is calculated according to the flash emission amount and the like. At that time, when shooting in a scene in which ambient light and flash light are mixed, a subject close to the digital still camera can have an appropriate white balance. However, a subject at a far distance is more dependent on ambient light than flash light. Therefore, when the environmental light is a fluorescent lamp or the like, the color temperature difference between the environmental light and the flash light becomes large. As a result, when the white balance correction process is applied to the entire image, there arises a problem that an inappropriate white balance occurs such as a white subject at a long distance becomes yellow.

  For this reason, in order to achieve an appropriate white balance for the entire image, the flash reflection amount is calculated by comparing the flash emission image and the flash non-emission image, and the optimal white balance gain is calculated for each pixel from the flash reflection amount. The technology to be applied is disclosed.

  Patent Document 1 discloses a technique for calculating an optimal white balance gain by interpolation when the flash emission image and the flash non-emission image have different image sizes. The entire flash emission image and flash non-emission image are divided into a plurality of blocks, a white balance gain is obtained by comparing the corresponding blocks, and is set as the white balance gain for the central pixel of the block. The white balance gain of the non-center pixel is obtained by interpolation based on the distance from the center pixel of the adjacent block.

  Further, Patent Document 2 discloses a method of performing interpolation using a method different from that of Patent Document 1. A flash non-emission image and a flash emission image with the number of pixels reduced by thinning are acquired, and a color temperature condition is calculated for each pixel. In the case where the flash non-light-emitting image is thinned, the white balance gain is obtained by interpolating the thinned pixels. As the color temperature condition of the interpolation target pixel, three pixels close to the interpolation target pixel are selected, and the color temperature condition of the pixel close to the luminance value of the interpolation target pixel is set as the color temperature condition of the interpolation target pixel. The white balance gain of the interpolation target pixel is obtained from this color temperature condition.

JP 2005-347811 A JP 2002-369111 A

  By the way, in Patent Document 1, when an object having a boundary such as an edge has a large change in color temperature at the boundary, interpolation at a distance from the center pixel results in an inappropriate white balance gain near the boundary. There is.

  Further, in Patent Document 2, interpolation is appropriately performed for a subject having a boundary such as an edge, but in a subject whose white balance gain changes smoothly, a step is generated and the interpolation is not smooth. There's a problem.

  When a flash non-light emitting image having a smaller number of pixels than a flash light emitting image, such as a live view image, is used, a white balance gain must be obtained by interpolation for the thinned pixels. In such a method, there is a problem that an appropriate white balance gain cannot be obtained if interpolation is performed based on the distance between pixels or the values of neighboring pixels remain unchanged.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide an image at the time of flash non-emission with a small number of pixels and an actual shooting image with a large number of pixels at the time of flash emission. In the method of calculating the white balance gain for each pixel by comparing with the image of the image, it is possible to adjust the entire image to an appropriate white balance by appropriately interpolating the thinned pixels. Another object of the present invention is to provide an improved imaging apparatus and imaging method.

  In order to solve the above-described problem, according to one aspect of the present invention, a first image when the flash is not emitted with a small number of pixels is acquired by thinning-out processing, and a flash emission with more pixels than the first image is obtained. An image capturing unit that acquires the second image, an image thinning unit that performs a thinning process on the second image so that the second image has the same size as the first image, and acquires a third image; A subject brightness calculation unit that calculates a first subject brightness for all pixels of the acquired first image and calculates a second subject brightness for all pixels of the acquired third image; and a first for each pixel A subject brightness calculation unit that calculates a subject brightness difference, which is a difference between the subject brightness of each pixel and a second subject brightness for each pixel, for all pixels, and a flash that calculates a first flash reflection amount for each pixel from the subject brightness difference A reflection amount calculation unit; Using the luminance value of each pixel of the third image, the first flash reflection amount for each pixel is subjected to interpolation processing to reduce the influence of the positional deviation between the first image and the third image. Using the first flash reflection amount interpolation unit for calculating the second flash reflection amount and the luminance value in pixel units of the second image, the third image with respect to the second flash reflection amount for each pixel A second flash reflection amount interpolation unit that calculates a third flash reflection amount by performing an interpolation process for enlarging the image to the size of the second image, and calculates a white balance gain in pixel units from the third flash reflection amount. An imaging apparatus is provided that includes a white balance gain calculation unit.

  The first subject luminance of the first image when the flash does not emit light with a small number of pixels is calculated, and the second image when the flash is emitted is thinned out to be the same as the size of the first image The second subject brightness of the image is calculated. Then, with respect to the first flash reflection amount based on the subject brightness difference between the first and second subject brightness for each pixel, the first image and the third image are obtained from the brightness value in pixel units of the third image. The second flash reflection amount is calculated by performing interpolation for reducing the influence of the image position shift. In addition, the third flash reflection amount is calculated by interpolating the third image to the size of the second image with respect to the second flash reflection amount from the luminance value of the pixel unit of the second image. Is done. Then, a white balance gain is calculated for each pixel from the third flash reflection amount. As a result, since interpolation is appropriately performed on the thinned pixels, the entire image can be adjusted to an appropriate white balance.

  The image processing apparatus further includes a white balance control unit that applies the white balance gain to all pixels of the second image in units of pixels and controls white balance in units of pixels. With this configuration, the white balance gain is applied to all pixels of the second image in units of pixels, and the white balance is controlled in units of pixels.

  Based on the light received from the subject, the first exposure control value at which the brightness of the first image obtained by shooting without firing the flash is appropriate, and the first exposure control value obtained by shooting with the flash An exposure control value calculation unit that calculates a second exposure control value at which the brightness of the second image is appropriate, and an exposure control value difference that is a difference between the first exposure control value and the second exposure control value And an exposure control value difference calculating unit that calculates the image, and the imaging unit captures a first image by capturing based on the first exposure control value, and captures a second image by capturing based on the second exposure control value. The subject brightness calculation unit calculates a subject brightness difference that is a difference between the exposure control value difference, the first subject brightness for each pixel, and the second subject brightness for each pixel for all pixels. .

  The first exposure control value and the second exposure control value are individually calculated by the exposure control value calculation unit, and the first exposure control value and the second exposure control value do not need to match.

  The imaging unit captures a second exposure control value obtained by matching the aperture value among the first exposure control values calculated by the exposure control value calculation unit, and acquires a second image.

  A color balance calculation unit that linearly interpolates the color balance of the ambient light and the color balance of the flash light based on the first flash reflection amount calculated by the flash reflection amount calculation unit, and calculates the color balance of the target pixel; The white balance gain calculating unit calculates a white balance gain based on the color balance.

  The first flash reflection amount interpolating unit calculates the second flash reflection amount based on the luminance values of the upper, lower, left and right interpolation source pixels of the pixel to be interpolated at the center and the first flash reflection amount. .

  In the first flash reflection amount interpolating unit, the distance between the pixel to be interpolated at the center and the interpolation source pixel in the vertical direction and the horizontal direction is constant.

  In order to solve the above-described problem, according to another aspect of the present invention, a first image when the flash does not emit light with a small number of pixels is acquired by thinning processing, and the flash has a larger number of pixels than the first image. Obtaining a second image at the time of light emission, obtaining a third image by thinning the second image so that the second image is the same as the size of the first image, Calculating a first subject brightness for all pixels of the acquired first image and calculating a second subject brightness for all pixels of the acquired third image; and a first subject brightness for each pixel Calculating a subject brightness difference, which is a difference between the second subject brightness for each pixel, for all pixels, calculating a first flash reflection amount in pixel units from the subject brightness difference, and a third image Brightness in pixel units Calculating the second flash reflection amount by performing an interpolation process to reduce the influence of the positional deviation between the first image and the third image on the first flash reflection amount for each pixel using And using the luminance value of each pixel of the second image, the second flash reflection amount for each pixel is subjected to an interpolation process for enlarging the third image to the size of the second image. An imaging method is provided, comprising: calculating a third flash reflection amount; and calculating a white balance gain in pixel units from the third flash reflection amount.

  As described above, according to the present invention, the image when the flash is not emitted with a small number of pixels is compared with the image when the flash is emitted, which is a main image with a large number of pixels, and the white balance gain is calculated for each pixel. In the method, the entire image can be adjusted to an appropriate white balance by appropriately interpolating the thinned pixels.

It is a block diagram showing an imaging device concerning one embodiment of the present invention. 4 is a flowchart showing a white balance gain calculation operation of the imaging apparatus 100 according to the embodiment. 4 is a flowchart showing a white balance gain calculation operation of the imaging apparatus 100 according to the embodiment. It is explanatory drawing which shows a Bayer unit. It is explanatory drawing which shows a thinning process. It is explanatory drawing which shows the image acquired by flash light emission imaging | photography. It is explanatory drawing which shows the image acquired by flash non-light-emission photography. It is explanatory drawing which shows the image which applied the exposure control value difference with respect to the flash non-light-emission image. It is explanatory drawing which shows an interpolation object pixel and an interpolation source pixel. It is explanatory drawing which shows the concept of an interpolation process. It is explanatory drawing which shows the concept of an interpolation process. It is explanatory drawing which shows an interpolation object pixel and an interpolation source pixel.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  FIG. 1 is a block diagram of an image processing pipeline in the imaging apparatus 100 according to the present embodiment. FIG. 2 is a flowchart illustrating a white balance gain calculation operation of the imaging apparatus 100 according to the present embodiment.

  The light that has passed through the lens 102 forms an image on an image sensor such as a CCD image sensor 108 or a CMOS image sensor. As a result, an RGB image signal is obtained from the CCD image sensor 108 via the signal processing unit 110. The signal processing unit 110 performs color separation processing, A / D conversion, and the like on the signal from the CCD image sensor 108.

  The RGB image signal obtained by the signal processing unit 110 is subjected to processing such as defective pixel correction, black level correction, and shading correction by the preprocessing unit 112. The preprocessing unit 112 includes an AE evaluation value calculation unit 124, and calculates an AE evaluation value based on the RGB image signal. The AE evaluation value is sent to the appropriate AE calculation unit 150. When shooting is actually performed, the exposure control unit 152 adjusts the aperture 104 and the shutter 106 based on the AE value calculated by the appropriate AE calculation unit 150.

  The image signal processed by the preprocessing unit 112 is subjected to various processes such as white balance correction processing, Bayer color interpolation (demosaic) processing, color correction processing, edge enhancement processing, gamma correction processing, and noise reduction processing by the postprocessing unit 114. Image processing is performed. Thereafter, the RGB → YCC conversion unit 116 converts the RGB signal into a YCC signal, and the data compression unit 118 performs compression processing in a format such as JPEG. Then, image data is recorded on the memory card 122 via the memory card connection unit 120 serving as an interface.

  The present embodiment is characterized in that an appropriate white balance gain is calculated for each pixel with respect to a flash emission image acquired by shooting with the flash 160 being emitted.

  Hereinafter, the shooting operation of the imaging apparatus 100 according to the present embodiment will be described. Shooting is started by the user pressing the shutter button. Hereinafter, a case where flash emission is necessary depending on exposure conditions will be described. In the case of an exposure condition that requires flash emission, in one shooting, flash non-flash shooting without flash 160 and flash flash shooting with flash 160 are performed.

  First, photographing is performed without the flash 160 being emitted, and an image when the flash is not emitted is acquired (step S101). At this time, a first image made up of RGB image signals of only ambient light before flash emission as shown in FIG. The first image is held in storage means such as the image memory 130 by the first imaging storage unit 132 (step S102). This first image becomes an image having a smaller image size than the size of the actual captured image by thinning-out processing or the like like a live view image.

  Subsequently, a second image made up of RGB image signals after flash emission as shown in FIG. 5 is acquired by the second imaging, which is shooting that emits light from the flash 160 (step S103). The second image is held in storage means such as the image memory 130 by the second imaging storage unit 134 (step S104).

  The color balance calculation unit 144 of the multi-AWB processing unit 140 calculates a color balance from the input image signal. The color balance calculation unit 144 integrates the RGB image signals of the flash non-emission image for each color (RGB) for the entire screen, and the color balance CBer (= R / G in the case of only the environmental light not emitted by the flash 160). ), CBeb (= B / G) is calculated. At the same time, the color balances CBfr and CBfb in the case where an image is picked up only with flash light without ambient light are previously stored in the ROM as known values and are read from the ROM.

  FIG. 1 is a block diagram of an image processing pipeline in an imaging apparatus 100 according to an embodiment of the present invention. The input values of the multi-AWB processing unit 140 are an RGB image signal when the flash is not emitted and an RGB image signal when the flash is emitted, and the white balance gain of the output value is the white balance gain corresponding to the total number of pixels. The white balance control unit 126 applies white balance gain in units of pixels in all pixels.

  First, the subject brightness in units of pixels is calculated based on the RGB image signal when the flash is not emitted and the RGB image signal when the flash is emitted, which is held in a storage unit such as the image memory 130 as an input value. As shown in FIG. 3, the calculation of the subject luminance in units of pixels is performed by regarding the Bayer unit 14 including the four pixels 12 of R, G, G, and B as one set, or by Bayer color interpolation (demosaic) processing. This is performed by generating one pixel of each color (R, G, B) from four pixels of R, G, G, B in the Bayer image.

  Next, the second image that is the flash emission image is changed to the same size as the first image that is the flash non-emission image (step S105). The change to the first image size is performed by the image thinning process, and the third image is calculated. FIG. 4 is a conceptual diagram showing the thinning process. The left figure of FIG. 4 shows the image before thinning. The thinning process is to extract only the upper left pixel 22 out of the four pixels in the thick line frame when thinning out the pre-thinning image in half in the vertical and horizontal directions, and to obtain an image as shown in the right figure. The thinning process is the same procedure even when thinning out the pre-thinned image by 1/3 of the vertical and horizontal directions.

  When the thinning process is performed on the second image, it is necessary to change the thinning method depending on how the first image, which is a non-flash image, is thinned. In FIG. 4, thinning is performed so as to leave the upper left pixel 22 in the thick line frame. However, when the thinning method of the first image is an average value of all the pixels in the thick line frame in FIG. Similarly, the third image needs to be thinned so as to calculate the average value.

  For each pixel of the first image and the third image, the multi-AWB processing unit 140 calculates a subject luminance value in units of pixels. The calculation formula is as follows. In Equations 1 and 2, Y is the luminance value Y of each pixel. Further, since the G signal value is substantially the same as the luminance value Y, the G signal value can be used instead of the luminance value Y. AE is a target luminance signal value targeted by the appropriate AE calculation unit 150 that calculates the exposure control value. In this calculation, the first subject luminance value BVlp of the first image and the second subject luminance value BVsp of the third image are obtained in units of pixels. Here, BVlp and BVsp are logarithmic signal values, and the unit system is aligned with the APEX value.

  At this time, when the exposure control value when shooting the first image as shown in FIG. 6 and the exposure control value when shooting the second image as shown in FIG. Consider differences in exposure control values. The exposure control value has the relationship of Equation 3. Here, TV is the shutter speed, AV is the aperture value, SV is the sensitivity, and BV is the subject brightness, which are expressed as APEX values.

  The subject brightness value BV calculated from the TV, AV, and SV when the flash is not emitted is the first exposure control value BV1, and the subject brightness value BV calculated from the TV, AV, and SV when the flash is emitted is the second exposure control. A value BVs is obtained by Equation 3. In the case of the first image, TVl, AVl, and SVl are set as shown in Formula 4, and in the case of the second image, TVs, AVs, and SVs are set as shown in Formula 5, respectively.

  The difference BVdiff between BVl and BVs obtained from Equation 4 and Equation 5 is obtained by Equation 6. At this time, if the exposure control values of the first image and the second image match, the difference BVdiff is zero.

  In consideration of the first subject luminance value BVlp of the first image in the pixel unit thus obtained and the difference BVdiff between the exposure control values, the flash non-luminous subject luminance difference BVlpdiff is obtained by Expression 7. When the calculation of Expression 7 is applied to the flash non-light-emitting image photographed with the appropriate exposure of FIG. 6, the image has an appropriate brightness as shown in FIG.

  The multi-AWB processing unit 140 calculates the flash emission image subject luminance difference BVdiffp by the difference between the second subject luminance value BVsp of the third image in pixel unit and the flash non-emission subject luminance difference BVlpdiff in pixel unit by Expression 8. Ask.

  The flash reflection amount calculation unit 141 calculates the first flash reflection amount of each pixel of the third image, which is the flash emission image, from the BVdiffp calculated by Expression 8 (step S106). The light quantity Lf of the third image is as shown in Equation 9.

  From the light amount Lf of the third image, the first flash reflection amount (Ratio) when the environmental light amount in the first image which is a flash non-light-emitting image is 1 is obtained as shown in Expression 10. Then, the first flash reflection amount is calculated for all the pixels of the third image (step S107).

  Based on the luminance value of each pixel of the third image and the first flash reflection amount obtained by Expression 10, the first flash reflection amount interpolation unit 142 performs interpolation, and for each pixel of the third image, A second flash reflection amount is obtained (steps S108 and S109). Since the G signal value of each pixel is almost the same as the luminance value, it is also possible to use the G signal value instead of the luminance value. The purpose of the first flash reflection amount interpolation unit 142 to calculate the second flash reflection amount by performing the interpolation is that the first image and the third image are not affected by blurring because the timing of imaging is not simultaneous. This is because there is a possibility that the first flash reflection amount is erroneously calculated due to the deviation. By interpolating from the values of the surrounding pixels, the effect of deviation can be reduced.

  Here, if the interpolation target pixel in the third image in FIG. 8 is C55, the upper, lower, left, and right pixels C15, C95, C51, and C59 are set as interpolation source pixels. At this time, a pixel that is one pixel away from the interpolation target pixel is used as an interpolation source pixel, but two or more pixels may be adjacent to each other.

  The luminance values Y1 and Y2 of the interpolation source pixel and the first flash reflection amounts Ratio1 and Ratio2 are known from the equation 10 and the like, and the luminance value of the interpolation target pixel Y is also known. A conceptual diagram of this interpolation is as shown in FIG. From these values, the second flash reflection amount of the interpolation target pixel of the third image is obtained in Expression 11.

  Since Equation 11 is an interpolation between two points, the flash reflection amount Ratio_v in the vertical direction and the flash reflection amount Ratio_h in the left-right direction are respectively obtained by Equation 11, and the second flash reflection amount Ratio_result is obtained as an average value as shown in Equation 12. calculate.

  Further, instead of calculating the average value as shown in Equation 12, the vertical direction and the horizontal direction interpolation may be separated by performing the horizontal direction interpolation after performing the vertical direction interpolation for all pixels. Good.

  If the interpolation calculation as described above is performed, even when there are a plurality of interpolation target pixels A to E between the interpolation source pixels 1 and 2 as shown in FIG. 10 and the luminance value of the interpolation target pixel changes smoothly, Even when a boundary having a large color temperature difference such as an edge occurs, it is possible to appropriately interpolate and appropriately calculate the flash reflection amount.

  Next, enlargement is performed from the image size of the third image to the image size of the second image. As shown in FIG. 11, the second flash reflection amount interpolating unit is referred to by using Expressions 11 and 12 from the luminance values of pixels that are thinned out based on the second image with reference to the pixels in the vertical and horizontal directions. 143 performs interpolation (step S110). In this way, interpolation is performed for all pixels of the image size of the second image, and the third flash reflection amount is calculated (step S111).

  Based on the third flash reflection amount, the color balance calculation unit 144 linearly interpolates the color balance (CBf) of the flash light and the color balance (CBe) of the ambient light to obtain the color balance (CB) of the target pixel ( Step S112). The color balance CB is obtained separately for each of R and B.

  The white balance gain calculation unit 145 calculates the white balance gain (WBGainR, WBGainB) of the target pixel according to Expression 14 from the R and B color balance obtained in Expression 13 (Step S113).

  By performing Expression 13 and Expression 14 for all the pixels (step S114), an appropriate white balance gain is obtained for a scene in which flash light and ambient light are mixed even when the image size of the flash non-light-emitting image and the flash light-emitting image are different. Can be obtained in pixel units.

  After calculating the white balance gain of all the pixels, the white balance control unit 126 applies the white balance gain to the corresponding pixel of the flash emission image, so that an appropriate white balance process can be performed.

  Conventionally, there has been a problem that an appropriate white balance gain cannot be calculated when the image sizes of the non-flash image and the flash image are different. On the other hand, according to the present embodiment, even in a non-flash light image having a different image size, an appropriate white balance can be obtained for each pixel even in a scene in which ambient light and flash light are mixed by performing interpolation based on the luminance value of the pixel. The gain can be calculated.

  In other words, an image without flash, such as a live view image, with a small number of pixels is compared with an image with flash, which is a full-size main image with a large number of pixels, and an appropriate white balance gain is set for each pixel. In the calculation method, the entire image can be adjusted to an appropriate white balance by appropriately interpolating the thinned pixels.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 100 Imaging device 102 Lens 104 Aperture 106 Shutter 108 CCD image sensor 110 Signal processing unit 112 Preprocessing unit 114 Postprocessing unit 116 RGB → YCC conversion unit 118 Data compression unit 120 Memory card connection unit 122 Memory card 124 AE evaluation value calculation unit 126 White balance control unit 130 Image memory 132 First imaging storage unit 134 Second imaging storage unit 140 Multi-AWB processing unit 141 Flash reflection amount calculation unit 142 First flash reflection amount interpolation unit 143 Second flash reflection amount interpolation unit 144 Color balance calculation unit 145 White balance gain calculation unit 150 Proper AE calculation unit 152 Exposure control unit 160 Flash

Claims (9)

An imaging unit that acquires a first image at the time of flash non-emission with a small number of pixels by thinning processing, and acquires a second image at the time of flash emission that has more pixels than the first image;
An image thinning unit that performs a thinning process on the second image so as to obtain a third image so that the second image has the same size as the first image;
A subject brightness calculation unit that calculates a first subject brightness for all pixels of the acquired first image, and calculates a second subject brightness for all pixels of the acquired third image;
A subject brightness calculator that calculates a subject brightness difference for all pixels, which is a difference between the first subject brightness for each pixel and the second subject brightness for each pixel;
A flash reflection amount calculation unit for calculating a first flash reflection amount in pixel units from the subject luminance difference;
Interpolation that reduces the influence of misalignment between the first image and the third image with respect to the first flash reflection amount for each pixel using the luminance value of the third image in units of pixels. A first flash reflection amount interpolating unit for processing and calculating a second flash reflection amount;
Using the luminance value in pixel units of the second image, interpolation processing is performed to enlarge the third image to the size of the second image with respect to the second flash reflection amount for each pixel. A second flash reflection amount interpolation unit for calculating the third flash reflection amount,
An image pickup apparatus comprising: a white balance gain calculation unit that calculates a white balance gain for each pixel from the third flash reflection amount.   The imaging apparatus according to claim 1, further comprising a white balance control unit configured to apply the white balance gain to all pixels of the second image in units of pixels and control white balance in units of pixels. Based on the light received from the subject, the first exposure control value at which the brightness of the first image obtained by photographing without emitting the flash is appropriate, and obtained by photographing by emitting the flash An exposure control value calculation unit that calculates a second exposure control value at which the brightness of the second image to be set is appropriate,
An exposure control value difference calculating unit that calculates an exposure control value difference that is a difference between the first exposure control value and the second exposure control value;
The imaging unit captures the first image by capturing based on the first exposure control value, acquires the second image by capturing based on the second exposure control value,
The subject luminance calculation unit calculates, for all pixels, a subject luminance difference that is a difference between the exposure control value difference, the first subject luminance for each pixel, and the second subject luminance for each pixel. Item 3. The imaging device according to Item 1 or 2. The first exposure control value and the second exposure control value are individually calculated by the exposure control value calculation unit, and the first exposure control value and the second exposure control value do not need to match. The imaging device according to claim 3 . The imaging unit captures the second exposure control value obtained by matching the aperture value among the first exposure control values calculated by the exposure control value calculation unit, and acquires the second image; The imaging device according to claim 3 or 4 . Based on the first flash reflection amount calculated by the flash reflection amount calculation unit, a color balance calculation unit that linearly interpolates the color balance of ambient light and the color balance of flash light and calculates the color balance of the target pixel In addition,
The imaging device according to claim 1, wherein the white balance gain calculation unit calculates the white balance gain based on the color balance.   The first flash reflection amount interpolating unit calculates the second flash reflection amount based on the luminance values of the upper and lower interpolation pixels of the pixel to be interpolated at the center and the first flash reflection amount. The imaging device according to claim 1, wherein the imaging device is calculated. The imaging apparatus according to claim 7, wherein a distance between a pixel to be interpolated located at the center and each of the upper , lower, left , and right interpolation source pixels is constant. Acquiring a first image at the time of flash non-emission with a small number of pixels by thinning processing, and acquiring a second image at the time of flash emission having a larger number of pixels than the first image;
Thinning the second image so that the second image has the same size as the first image to obtain a third image;
Calculating a first subject brightness for all pixels of the acquired first image, and calculating a second subject brightness for all pixels of the acquired third image;
Calculating a subject brightness difference, which is a difference between the first subject brightness for each pixel and the second subject brightness for each pixel, for all pixels;
Calculating a first flash reflection amount in pixel units from the subject luminance difference;
Interpolation that reduces the influence of misalignment between the first image and the third image with respect to the first flash reflection amount for each pixel using the luminance value of the third image in units of pixels. Processing to calculate a second flash reflection amount;
Using the luminance value in pixel units of the second image, interpolation processing is performed to enlarge the third image to the size of the second image with respect to the second flash reflection amount for each pixel. Calculating a third flash reflection amount,
And a step of calculating a white balance gain in pixel units from the third flash reflection amount.

Images