JP5146015B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus, an imaging method, and a computer-readable recording medium, and more particularly to an imaging apparatus and an imaging method having a white balance control function capable of performing good white balance correction.

  In conventional imaging devices, if the light source is uniform over the entire screen, a method of estimating the light source and applying the same white balance to the entire screen is widely used. However, when there are irradiation areas of different types of light sources in the screen, it is not possible to achieve an appropriate white balance for each of the irradiation areas of different types of light sources if the same white balance is applied to the entire screen. .

Therefore, Patent Document 1 discloses a white balance control function that does not cause color misregistration with respect to the entire screen by applying white balance to each area divided into small areas.
Further, Patent Document 2 discloses a white balance control function for multiplying a pixel by a white balance coefficient in order to eliminate a color shift at an area boundary where different white balance is applied in adjacent areas.
Further, Patent Document 3 discloses a white balance control function that divides a color image into regions corresponding to a plurality of types of light sources according to luminance values of illumination components, and calculates a correction coefficient for each region.

JP 2002-271638 A JP 2005-347811 A JP 2007-129622 A

However, in the inventions described in Patent Documents 1 to 3, when there are a plurality of light sources for a subject located at the same distance, for example, assuming that half of the face is sunny and half is shade, On the other hand, a plurality of white balances are applied, resulting in an unnatural image.
In addition, since yellow has a relatively high luminance, while blue has a low luminance, there is a reduction in that the blue in the sun is lower than the shaded yellow, so depending on the shooting scene, area division based only on the luminance component However, the problem is that it is not always possible to perform white balance correction corresponding to the light source.

In addition, even if the area is divided for each light source and white extraction is performed in each area, a white extraction range is provided for the purpose of widening the color temperature tracking range. There is a problem that white balance cannot be applied. That is, the distribution of color information is similar between the sun and the shade, and even if white extraction is performed for each area, as a result of being affected by the chromatic color, the white balance between the sun area and the shade area becomes almost the same. For this reason, as in the case where the same white balance is applied to the entire screen, in a scene where the sun (high brightness area) and the shade (low brightness area) exist, the shade area becomes blue when the optimal white balance is applied to the sun, The shaded area becomes pale. Conversely, when the optimal white balance is applied to the shaded area, the sunlit area becomes yellow, and the white part of the shaded area and the sky evaluation value are distributed at almost the same position, which causes the problem of whitening the sky. .
Conventionally, in an imaging apparatus using a sensor with a narrow dynamic range, if there is a shaded area and sky (the sunlit area), the optimal white balance for the shaded area can be applied to the sky with an exposure that matches the shaded area. In the first place, the sky wasn't a problem because it was white. In addition, even when the white balance that does not make the sky white is applied to the shaded area in the exposure that matches the sky, there was no problem because the shaded area was darkened and the saturation was extremely low in the first place. However, in the future, it is expected that the imaging device using a sensor with a wide dynamic range or the expansion of the dynamic range by image processing is expected, so even if the optimum white balance is applied to the shaded area, the sky will not be whitened, the sunny area, Even when the white balance optimal for the sky is applied, it is required not to make the skin in the shaded area pale.

  The present invention has been made in view of the above-described problems in the prior art, and performs imaging that performs stable region division regardless of the luminance component, and also performs good white balance correction for subjects located at the same distance. An object is to provide an apparatus, an imaging method, and a computer-readable recording medium.

That is, in order to solve the above problems, an imaging apparatus and an imaging method, and a computer-readable recording medium according to the present invention specifically have the technical features described in (1) to (15) below.
(1): an image sensor that converts light incident from the optical system into an electrical signal and outputs it as an image signal, distance information acquisition means for acquiring distance information from a distance measurement target, and a white balance evaluation value from the image signal A white balance evaluation value acquisition unit that extracts a white balance evaluation value, and an area dividing unit that divides the imaging signal into a plurality of areas based on the distance information and color difference information; and a white balance for each of the divided areas An imaging apparatus comprising: correction coefficient calculation means for calculating a white balance correction coefficient for each area based on an evaluation value; and white balance calculation means for setting the white balance correction coefficient for each divided area. It is.

(2): The area dividing unit divides the high-luminance area and the low-luminance area from the distance information, the color information, and the luminance information of the imaging signal, respectively. In contrast, a white extraction unit that performs white extraction by changing a white extraction range for each region, and the correction coefficient calculation unit includes a white extraction result obtained by the white extraction unit, and a result for each of the divided regions. The image pickup apparatus according to (1), wherein a white balance correction coefficient for each region is calculated based on the white balance evaluation value.

(3): The white extraction means performs white extraction based on a white balance evaluation value of the high luminance region and a white extraction range for the high luminance region in the high luminance region, and in the low luminance region, The imaging apparatus according to (2), wherein white extraction is performed based on a white balance evaluation value of a low luminance area and a white extraction range for the low luminance area.

(4) : a block dividing unit that divides the imaging signal into a plurality of blocks, the divided block having a target pixel and a non-target pixel, and the white balance calculating unit First correction coefficient acquisition means for acquiring the white balance correction coefficient for a target pixel for each block; a white balance correction coefficient for a peripheral target pixel for each of the divided blocks; and the non-target pixel Second correction coefficient acquisition means for acquiring the white balance correction coefficient by interpolation based on the distance from the surrounding target pixel to the surrounding target pixel, and acquired by the first correction coefficient acquisition means and the second correction coefficient acquisition means The imaging apparatus according to any one of (1) to (3) , wherein the white balun correction coefficient is set for each pixel.

(5) The imaging device according to (4) , wherein the target pixel is a pixel located at a center of the divided block.

(6) : a photoelectric conversion step of converting light incident from the optical system into an electrical signal and outputting it as an imaging signal, a distance information acquisition step of acquiring distance information with respect to a distance measuring object, and white balance evaluation from the imaging signal A white balance evaluation value acquiring step for extracting a value, an area dividing step for dividing the imaging signal into a plurality of regions based on the distance information and color difference information, and a white for each of the divided regions An imaging device comprising: a correction coefficient calculation step for calculating a white balance correction coefficient for each region based on a balance evaluation value; and a white balance calculation step for setting the white balance correction coefficient for each of the divided regions. Is the method.

(7) : The region dividing step divides the high luminance region and the low luminance region from the distance information, the color information, and the luminance information of the imaging signal into a high luminance region and a low luminance region, respectively. In contrast, a white extraction step of performing white extraction by changing a white extraction range for each region, the correction coefficient calculation step includes a white extraction result obtained by the white extraction step, and a result of each of the divided regions. The white balance correction value for each region is calculated based on the white balance evaluation value, and the imaging method according to (6) above.

(8) : In the high luminance region, the white extraction step performs white extraction based on a white balance evaluation value of the high luminance region and a white extraction range for the high luminance region, and in the low luminance region, The imaging method according to (7) , wherein white extraction is performed based on a white balance evaluation value of a low luminance region and a white extraction range for the low luminance region.

(9) : a block dividing step of dividing the imaging signal into a plurality of blocks, the divided block having a target pixel and a non-target pixel, and the white balance calculating step A first correction coefficient acquisition step of acquiring the white balance correction coefficient for the target pixel for each block; a white balance correction coefficient for a peripheral target pixel for the non-target pixel for each of the divided blocks; and the non-target pixel A second correction coefficient acquisition step of acquiring the white balance correction coefficient by interpolation based on a distance from the surrounding target pixel to the surrounding target pixel, acquired by the first correction coefficient acquisition step and the second correction coefficient acquisition step The imaging method according to any one of (6) to (8) , wherein the white balun correction coefficient is set for each pixel.

(10) The imaging method according to (9) , wherein the target pixel is a pixel located at a center of the divided block.

(11) : photoelectric conversion processing for converting light incident from the optical system into an electrical signal and outputting it as an imaging signal; distance information acquisition processing for acquiring distance information with respect to a distance measurement target; and white balance evaluation from the imaging signal A computer-readable recording medium recording a program for causing a computer to execute a white balance evaluation value acquisition process for extracting a value, wherein the imaging signal is divided into a plurality of regions based on the distance information and color difference information. Area division processing for dividing, correction coefficient calculation processing for calculating a white balance correction coefficient for each area based on the white balance evaluation value for each of the divided areas, and the white balance correction coefficient for each of the divided areas. A computer-readable recording medium comprising: white balance calculation processing to be set.

(12) : The area dividing process divides the high luminance area and the low luminance area from the distance information, the color information, and the luminance information of the imaging signal, respectively. In contrast, a white extraction process for performing white extraction by changing a white extraction range for each area is provided, and the correction coefficient calculation process includes a white extraction result obtained by the white extraction process, and a result for each of the divided areas. The computer-readable recording medium according to (11) , wherein a white balance correction coefficient for each region is calculated based on the white balance evaluation value.

(13) : The white extraction process performs white extraction in the high luminance region based on a white balance evaluation value of the high luminance region and a white extraction range for the high luminance region. The computer-readable recording medium according to (12) , wherein white extraction is performed based on a white balance evaluation value of a low luminance area and a white extraction range for the low luminance area.

(14) : a block division process for dividing the imaging signal into a plurality of blocks, wherein the divided block includes a target pixel and a non-target pixel, and the white balance calculation process is performed by the division A first correction coefficient acquisition process for acquiring the white balance correction coefficient for a target pixel for each block, a white balance correction coefficient for a peripheral target pixel for each of the divided blocks, and the non-target pixel And a second correction coefficient acquisition process for acquiring the white balance correction coefficient by interpolation based on a distance from the surrounding target pixel to the surrounding target pixel, and acquired by the first correction coefficient acquisition process and the second correction coefficient acquisition process. The computer-readable data according to any one of (11) to (13) , wherein the white balun correction coefficient is set for each pixel. It is a capable recording medium.

(15) The computer-readable recording medium according to (14) , wherein the target pixel is a pixel located at a center of the divided block.

  According to the present invention, an image pickup apparatus and image pickup method that perform stable region division regardless of luminance components, and further perform good white balance correction for subjects located at the same distance, and a computer-executable recording medium Can be provided.

  An image pickup apparatus according to the present invention includes an image pickup element that converts light incident from an optical system into an electric signal and outputs the signal as an image pickup signal, distance information acquisition means for acquiring distance information from a distance measurement target, and white from the image pickup signal. In an imaging apparatus including a white balance evaluation value acquisition unit that extracts a balance evaluation value, an area dividing unit that divides the imaging signal into a plurality of areas based on the distance information and color difference information, and for each of the divided areas Correction coefficient calculation means for calculating a white balance correction coefficient for each area based on the white balance evaluation value, and white balance calculation means for setting the white balance correction coefficient for each of the divided areas. To do.

Next, the basic configuration of the imaging apparatus according to the present invention will be described in more detail with reference to the drawings.
The embodiments described below are preferred embodiments of the present invention, and thus various technically preferable limitations are given. However, the scope of the present invention is particularly limited in the following description. Unless otherwise stated, the present invention is not limited to these embodiments.

[First Embodiment]
(Appearance structure of digital camera)
FIG. 1 is a schematic diagram illustrating an appearance of a digital camera that is a first embodiment of an imaging apparatus according to the present embodiment.
As shown in FIG. 1, a release button (shutter button), a power button, and a shooting / playback switching dial are provided on the upper surface side of the digital camera according to the present embodiment, and the front (front) side of the digital camera. Includes a lens barrel unit having a photographing lens system, a strobe light emitting unit, and an optical viewfinder.

  On the back side of the digital camera is a liquid crystal monitor (LCD), the eyepiece of the optical viewfinder, wide-angle zoom (W) switch, telephoto zoom (T) switch, menu (MENU) button, and enter button (OK button) Etc. are provided. In addition, a memory card storage unit for storing a memory card for storing captured image data is provided inside the side surface of the digital camera.

(Digital camera system configuration)
FIG. 2 is a block diagram showing a configuration of a digital camera which is the first embodiment of the imaging apparatus according to the present invention.
As shown in FIG. 2, in this digital camera, a CCD, a CCD as a solid-state imaging device, on which a subject image incident through a photographing lens system (optical system) installed in a lens barrel unit is formed on a light receiving surface. An analog front end unit (AFE unit) that processes electrical signals (analog RGB image signals) that are output into digital signals (imaging signals), a signal processing unit that processes digital signals output from the AFE unit, and data temporarily An SDRAM to be stored, a ROM in which a control program is stored, a motor driver, and the like are provided.

  The lens barrel unit includes a photographic lens system having a zoom lens, a focus lens, and the like, an aperture unit, and a mechanical shutter unit. Each drive unit (not shown) of the photographic lens system, the aperture unit, and the mechanical shutter unit is a motor driver. Driven by. The motor driver is driven and controlled by a drive signal from a control unit (CPU) of the signal processing unit.

  In the CCD, RGB primary color filters as color separation filters are arranged on a plurality of pixels constituting the CCD, and electrical signals (analog RGB image signals) corresponding to the three primary colors of RGB are output.

  The AFE unit includes a TG (timing signal generation unit) that drives the CCD, a CDS (correlated double sampling unit) that samples an electrical signal (analog RGB image signal) output from the CCD, and an image signal sampled by the CDS. An AGC (analog gain control unit) that adjusts the gain and an A / D that converts the image signal gain-adjusted by the AGC into a digital signal (RAW-RGB data) are provided.

  The signal processing unit outputs a screen horizontal synchronizing signal (HD) and a screen vertical synchronizing signal (VD) to the TG of the AFE unit, and the RAW− output from the A / D of the AFE unit according to these synchronizing signals. A CCD interface (CCD I / F) that captures RGB data, a memory controller that controls the SDRAM, a YUV converter that converts the captured RAW-RGB data into YUV format image data that can be displayed and recorded, and display and recording A resizing processing unit that changes the image size according to the size of the image data to be displayed, a display output control unit that controls display output of the image data, a data compression unit for recording image data by JPEG formation, and the like Media interface that writes data to a memory card or reads image data written to a memory card ( And Deer I / F), based on the operation input information from the operation unit, and a control section for entire digital camera system control or the like based on the control program stored in a ROM (CPU).

  The operation unit includes a release button, a power button, a shooting / playback switching dial, a wide-angle zoom switch, a telephoto zoom switch, a menu button, a confirmation button, and the like provided on the external surface of the digital camera (see FIG. 1). A predetermined operation instruction signal is input to the control unit by a photographer's operation.

  In the SDRAM, the RAW-RGB data captured by the CCD I / F is stored, and YUV data (YUV format image data) converted by the YUV conversion unit is stored, and further compressed by the data compression unit. The image data such as JPEG formation is stored. The OSD (On Screen Display) is data to be displayed on the LCD display, and is obtained by superimposing operation system setting information on YUV data or JPEG images.

  Note that YUV of the YUV data is luminance data (Y), color difference (difference (U) between luminance data and blue (B) component data, and difference (V) between luminance data and red (R) component data). It is a format that expresses color with information. When performing this image conversion, WB gains Rg and Bg are set from the control unit to the ISP.

  An ISP (image signal processor) is a main part of a signal processing unit that performs white balance, gamma correction, color correction, and the like.

(Digital camera monitoring and still image shooting)
Next, the monitoring operation and still image shooting operation of the digital camera will be described. This digital camera performs a still image shooting operation while performing a monitoring operation as described below in the still image shooting mode.

  First, when the photographer turns on the power button and sets the photographing / playback switching dial to the photographing mode (still image photographing mode), the digital camera is activated in the recording mode. When the control unit detects that the power button is turned on and the shooting / playback switching dial is set to the shooting mode, the control unit outputs a control signal to the motor driver, and moves the lens barrel unit to the shooting enabled position. In addition, the CCD, the AFE unit, the signal processing unit, the SDRAM, the ROM, the liquid crystal monitor, and the like are activated.

  Then, by directing the photographic lens system of the lens barrel unit toward the subject, a subject image incident through the photographic lens system is formed on the light receiving surface of each pixel of the CCD. An electrical signal (analog RGB image signal) corresponding to the subject image output from the CCD is input to the A / D via the CDS and AGC, and converted into 12-bit (RAW) RGB data by the A / D. Convert.

  This RAW-RGB data is taken into the CCD I / F of the signal processing unit and stored in the SDRAM via the memory controller. The RAW-RGB data read from the SDRAM is input to the YUV converter and converted into YUV data (YUV signal) that can be displayed, and then the YUV data is stored in the SDRAM via the memory controller. Is done.

  Then, the YUV data read from the SDRAM via the memory controller is sent to the liquid crystal monitor (LCD display) via the display output control unit, and a photographed image (moving image) is displayed. At the time of monitoring in which a photographed image is displayed on the above-described liquid crystal monitor (LCD display), one frame is read out in a time of 1/30 second by thinning out the number of pixels by the CCD I / F.

  During this monitoring operation, the photographed image (movie) is only displayed on the liquid crystal monitor (LCD display) functioning as an electronic viewfinder, and the release button has not been pressed (including half-pressed). is there.

  By displaying the photographed image on a liquid crystal monitor (LCD), the composition for photographing a still image can be confirmed. In addition, it can output as a TV video signal from a display output control part, and a picked-up image (moving image) can also be displayed on external TV (television) via a video cable.

Then, the CCD I / F of the signal processing unit calculates an AF (automatic focus) evaluation value, an AE (automatic exposure) evaluation value, and an AWB (auto white balance) evaluation value from the captured RAW-RGB data.
In the present invention, the luminance information is an AE evaluation value, the white balance evaluation value is an AWB evaluation value, and the white balance evaluation value acquisition unit and the distance information acquisition unit are respectively CCDI / Part of F.

  The AF evaluation value is calculated by, for example, the output integrated value of the high frequency component extraction filter or the integrated value of the luminance difference between adjacent pixels. When in the in-focus state, the edge portion of the subject is clear, so the high frequency component is the highest. By using this, at the time of AF operation (focus detection operation), AF evaluation values at each focus lens position in the photographing lens system are acquired, and AF operation is performed with the maximum point as the focus detection position. Is executed.

  The AE evaluation value and the AWB evaluation value are calculated from the integrated values of the RGB values in the RAW-RGB data. For example, the screen corresponding to the light receiving surface of all the pixels of the CCD is equally divided into 256 areas (blocks) by a detection circuit (block dividing means) that is a part of the CCD I / F of the signal processing unit (horizontal 16 divisions, vertical 16 divisions). ), The RGB integrated value of each area (block) is calculated by the CCD I / F.

  Then, the control unit reads the calculated RGB integrated values and calculates WB gains Rgain and Bgain so as to achieve an appropriate white balance. In the AE process, the brightness of each area (block) on the screen is calculated, and an appropriate exposure amount is determined from the brightness distribution. Based on the determined exposure amount, exposure conditions (the number of electronic shutters of the CCD, the aperture value of the aperture unit, the insertion and removal of the ND filter, etc.) are set. Further, in the AWB process, an AWB control value that matches the color of the light source of the subject is determined from the RGB distribution. By this AWB process, white balance is adjusted when the YUV conversion unit performs conversion processing to YUV data. The AE process and AWB process described above are continuously performed during the monitoring.

  When a still image shooting operation in which the release button is pressed (half-pressed to fully pressed) is started during the monitoring operation described above, an AF operation that is a focus position detection operation and a still image recording process are performed.

  That is, when the release button is pressed (half-pressed to fully pressed), the focus lens of the photographic lens system is moved by a drive command from the control unit to the motor driver. For example, a contrast evaluation method called so-called hill-climbing AF AF operation is executed.

  So-called hill-climbing AF means that when the AF (focusing) target range is the entire range from infinity to close, the focus lens of the taking lens system is at each focus position from close to infinity or from infinity to close. The controller reads the AF evaluation value at each focus position calculated by the CCD I / F (distance information acquisition means). Then, the focus lens is moved to the in-focus position with the point where the AF evaluation value at each focus position is maximized as the in-focus position, and in-focus.

  Then, the AE process described above is performed, and when the exposure is completed, the mechanical shutter unit is closed by a drive command from the control unit to the motor driver, and an analog RGB image signal for a still image is output from the CCD. Then, as in the monitoring, the A / D conversion unit of the AFE unit converts the data into RAW-RGB data.

  The RAW-RGB data (imaging signal) is taken into the CCD I / F of the signal processing unit, converted into YUV data by the YUV conversion unit, and stored in the SDRAM via the memory controller. The YUV data is read from the SDRAM, converted into a size corresponding to the number of recorded pixels by the resizing processing unit, and compressed to image data in JPEG format or the like by the data compression unit. The compressed image data such as JPEG format is written back to the SDRAM, read out from the SDRAM via the memory controller, and stored in the memory card via the media I / F.

Next, the imaging operation that characterizes the present invention will be described.
FIG. 3 shows a flow of a series of white balance (hereinafter also referred to as WB) correction processing in the present embodiment.
First, after a series of signal processing such as WB, gamma correction, and color correction in a still image shooting operation is finished, YUV data output from the ISP and Raw-RGB data output from the CCD-I / F before signal processing. Are stored in the SDRAM. The YUV data stored here is data necessary for performing WB correction for each area, and is not data that is finally stored in the memory card.

  Then, the YUV data stored in the SDRAM is read again, and is divided into a number of areas divided in units of pixels based on the color and edge information (S1; area dividing means). Since there are many fine regions in the division result at this time, inconveniences such as an increase in processing time are required to perform individual WB processing for each region. Cannot use the result of division in

  Next, distance information is further used for the image divided into regions based on the color and edge information, and the regions divided in S1 are grouped as subject regions and background regions according to the distance information (S2; region division) means). The distance information is output in block units of horizontal 16 × vertical 16 divisions, and the division in S1 is similarly performed in block units. There is a method to divide the area only with distance information, but distance information cannot always be acquired for all blocks, so first divide based on color and edge information, and assist with finely divided areas Use distance information for the purpose.

The distance information mentioned here is a high-pass filter output value obtained at each focus position by changing the focus position in the autofocus operation during imaging. The output value of the high pass filter is extracted from the Raw-RGB data by the CCD-I / F at the time of imaging in block units of horizontal 16 × vertical 16 divisions. Then, focusing is performed by peak detection from the high-pass filter output value for each focus position, and the focused block is set as the subject area and the non-focused block is set as the background area.
FIG. 4 shows an area divided in units of blocks, where 1 represents a subject area and 2 represents a background area.

  Next, in order to obtain the RGB integrated value for each area, the Raw-RGB data stored in the SDRAM is divided into blocks of horizontal 16 × vertical 16 as in the area division and distance information acquisition, and S2 The RGB of a plurality of blocks corresponding to the respective areas divided by the area division are integrated (S3; block dividing means).

  Next, white extraction is performed for each region (S4; white extraction means). In the white extraction, G / R and G / B for each block are obtained from the RGB integrated values acquired in units of blocks for each region, and blocks below the threshold value are stored as white extraction blocks in comparison with the threshold value.

  Among each region, a region having a certain number of white extraction blocks equal to or greater than a certain reference is calculated by weighting the G / R and G / B of the white extraction block and the average luminance value of the block. For a region where the number of white extraction blocks is less than a certain standard, a white balance correction coefficient is calculated by weighting the average luminance value of each block for G / R and G / B of all blocks in the region, A white balance correction coefficient calculated in an area where the number of white extraction blocks is equal to or greater than a certain reference is set as a white balance correction coefficient for an area where the number of white extraction blocks is less than a certain reference (S5; correction coefficient calculation means).

  When there is no white extraction block satisfying a certain standard in any of the areas, the white balance correction coefficient is calculated by weighting the G / R and G / B of the entire screen and the average luminance value of the entire screen.

  If a specific example is shown in the captured image as shown in FIG. 4, the white extraction block is a block that exists in the area of white clothes in the image of the person who is the subject, and a white cloud area in the background image. Examples include (existing) blocks.

  In the present invention, the white balance correction coefficient is preferably a correction coefficient that emphasizes a region having a high average luminance value, and an operation that emphasizes a region having a high average luminance value with respect to G / R and G / B. It can be calculated by performing

  As shown in FIG. 5, the center of each block is set as the target pixel, and the white balance correction coefficients Rgain and Bgain calculated for each region are set to the target pixel of the block corresponding to each region (S6; white balance calculating means). 3 in FIG. 5 is a target pixel for setting white balance correction coefficients Rgain and Bgain calculated in the subject area, and 4 in FIG. 5 is a target pixel for setting white balance correction coefficients Rgain and Bgain calculated in the background area.

  The white balance correction coefficients Rgain and Bgain of the non-target pixel are obtained by interpolation based on the white balance correction coefficient set for the target pixel of the block and the peripheral target pixel and the distance from the pixel (S7; white balance calculation means).

  A method for calculating the white balance correction coefficient of the non-target pixel will be described with reference to FIG. The Rgain (Bgain) of the non-target pixel for which the white balance correction coefficient is to be calculated is 11, and the white balance correction coefficients Rgain (Bgain) of the surrounding target pixels are R0 (B0), R1 (B1), R2 (B2), R3. (B3). The position of R3 (B3) is normalized by 1, the position of the non-target pixel for which the correction coefficient is to be calculated is represented by x and y, and the correction coefficient is calculated by the following calculation formula.

Rgain = (1-x) (1-y) R0 + x (1-y) R1 + (1-x) yR2 + xyR3
Bgain = (1-x) (1-y) B0 + x (1-y) B1 + (1-x) yB2 + xyB3

Specifically, for example, Rgain of the non-target pixel when R0 = 1.5, R1 = 1.2, R2 = 1.2, R3 = 1.2, x = 0.7, and y = 0.6 Is as follows.
Rgain = {(1-0.7) × (1-0.6) × 1.5} + {0.7 × (1-0.6) × 1.2} + {(1-0.7) × 0.6 × 1.2} + {0.7 × 0.6 × 1.2} = 1.236

  When there is only one peripheral pixel of interest as in the pixel 12 in FIG. 7, the white balance correction coefficient for the block is set.

  When there are only two peripheral pixels of interest like the pixels 13, 14, 15, and 16 in FIG. 8, they are obtained by interpolation based on two white balance correction coefficients and a distance from the pixel. The calculation formula is shown below.

In the case of the region 13, Rgain = (1-x) R2 + xR3
Bgain = (1-x) B2 + xB3

For region 14 Rgain = (1-x) R0 + xR1
Bgain = (1-x) B0 + xB1

In the case of the region 15, Rgain = (1-y) R1 + yR3
Bgain = (1-y) B1 + yB3

For region 16 Rgain = (1-y) R0 + yR2
Bgain = (1-y) B0 + yB2

  The R and B data of the Raw-RGB data stored in the SDRAM is multiplied by the white balance correction coefficients Rgain and Bgain calculated for each pixel (S8).

  According to the present embodiment, even when the color temperature irradiated for each region is different, a good WB can be applied to each. Also, by dividing the area using distance information, it is possible to divide the area, for example, the subject and the background, the subject is in the shade and the background is in the sun, or vice versa. In a generally problematic scene where the other becomes an unnatural WB, a good WB can be applied to each area.

  In addition, even when a plurality of light sources are applied to a single subject, it is possible to divide the region such as the subject and the background by dividing the region using distance information, and the same correction coefficient is applied to the entire subject region. In addition, since the same correction coefficient is applied to the entire background area in the same manner, it is possible to display an image that is good and has no sense of incongruity without applying a plurality of WBs to the same subject.

  Furthermore, since a low-pass filter is applied to the correction coefficient between pixels, it is possible to avoid an unnatural WB from being applied at the boundary between areas in an image in which there are areas having different WB correction coefficients.

  Further, since the correction coefficient is calculated for each region, the amount of calculation required for calculating the correction coefficient can be reduced as compared with the method of calculating for each block or pixel.

Here, in this embodiment, when the distance information is acquired and when the area is divided, the entire screen is divided into 16 horizontal 16 × vertical blocks, but may be divided by an arbitrary number of divisions.
In this embodiment, when the distance information is acquired and the area is divided, the entire screen is divided into 16 horizontal 16 × vertical blocks, but may be divided by an arbitrary number of divisions. Further, the division is preferably in a lattice shape.

[Second Embodiment]
The second embodiment is different from the first embodiment in the area dividing method of S1 and S2. S3 to S8 are the same as those in the first embodiment.
In the first embodiment, distance information is acquired based on the output value of the high-pass filter from the image pickup signal at the time of area division. Region division is performed with the region focused on the basis of the distance information obtained by the image CMOS as the subject region and the other region as the background region. Even when the distance information can be acquired for each pixel, the region is divided into blocks of 16 horizontal 16 × vertical 16 blocks for the entire screen. Therefore, the determination of the area for each block is an area having a larger ratio in the block.

  The distance image CMOS includes a light source that irradiates an object, projects light from the light source, detects reflected light from the object with an image sensor, and calculates the distance to the object based on the flight time of light. is there. The distance image CMOS corresponds to the distance measuring means and the image sensor in the present invention.

  As a result, the time required to acquire the distance information can be significantly reduced compared to the acquisition of distance information by AF.

[Third Embodiment]
FIG. 9 shows a flow of a series of white balance correction processes according to the third embodiment.
In this embodiment, an example is described in which the high luminance area is a sunny area and the low luminance area is a shade area, but the present invention is not limited to this, and the area is divided at an arbitrary luminance. Can do.

  First, for the YUV data that has passed through a series of image processing such as AE processing and WB processing from RAW-RGB data in the still image capturing operation, Hyuga as a high luminance area for each block of 16 horizontal and 16 vertical divisions. The area is divided into a shaded area as a low luminance area (S1; area dividing means). The block set at this time is the same as the block set when acquiring the AWB evaluation value.

  As a method of dividing into the sunny area and the shaded area, any method may be used as long as it is divided from distance information, color information, and luminance information. For example, after dividing a fine area from the relationship between luminance and B / R, the distance is divided. There is a method of grouping areas using information supplementarily.

Specifically, a region having a luminance equal to or higher than a certain threshold may be a sunny region, a threshold of luminance may be expressed using B / R as a variable, and a region having a luminance higher than the threshold may be a sunny region. Next, the finely divided sunny area and shade area are grouped according to distance information.
There is a method to divide the area only by distance information, but distance information is not necessarily acquired by all blocks, so first divide into the sunny area and the shaded area, and assist the finely divided area Use distance information for the purpose.

The distance information mentioned here is a high-pass filter output value obtained at each focus position by changing the focus position in the autofocus operation during imaging. The output value of the high pass filter is extracted from the Raw-RGB data by the CCD-I / F at the time of imaging in block units of horizontal 16 × vertical 16 divisions. Then, focusing is performed by peak detection from the high-pass filter output value for each focus position, and the focused block is set as the subject area and the non-focused block is set as the background area.
FIG. 4 shows an area divided in units of blocks, where 1 represents a subject area and 2 represents a background area.

White extraction is performed from the AWB evaluation value for each area based on the division result of the sunny area and the shade area.
If the white extraction is within a white extraction range set in advance with respect to the AWB evaluation value acquired in units of blocks, it is stored as a white extraction block (S2; white extraction means). The AWB evaluation value of each block is expressed in the form of G / R and G / B. As shown in FIG. 10, the white extraction range is a two-dimensional color coordinate having G / R as the x-axis and G / B as the y-axis. Is set. The white extraction range in the present embodiment uses a plurality of elliptical frames, and each elliptical frame is a white extraction frame corresponding to each light source, and the white extraction range is different between the sunny area and the shaded area. When neither the sunny area nor the shade area exists, white extraction is performed within the white extraction range in which all white extraction frames are valid. Further, the white extraction range on the G / R-G / B axis shown in FIG. 10 is an example, and changes depending on the configuration of the CCD.

  Examples of the preset white extraction range include the examples shown in FIGS. 11 and 12. The shaded area in FIG. 11 is the white extraction range of the sunny area (white extraction range for the high brightness area) from about 2300 to about 5800 K (Kelvin), and the shaded area in FIG. (Extraction range) corresponds to about 5800 to about 8000K.

  FIG. 13 shows an area divided into a sunny area and a shaded area in units of blocks. 13 in FIG. 13 represents a shaded area, and 6 in FIG. 13 represents a sunny area.

  Since each step of S3-3 to S3-6 is the same as S5 to 8 in the first embodiment, the details are omitted.

  According to the present embodiment, it is possible to perform white extraction without being affected by the chromatic color in each of the sunny area (high brightness area) and the shade area (low brightness area).

  Moreover, even when the color temperatures irradiated in the sunlit area and the shaded area are different, a good white balance can be applied to each.

  Furthermore, since a low-pass filter is applied to the correction coefficient between pixels, it is possible to avoid an unnatural white balance being applied at the boundary between areas in an image in which there are areas having different WB correction coefficients.

  In addition, by changing the WB locally on the screen, it is possible to apply an optimal WB for each region without performing composition by shooting a plurality of images, and the time required for shooting can be shortened.

  Further, since the WB correction coefficient is calculated for each region, the amount of calculation required for calculating the WB correction coefficient can be reduced as compared with the method of calculating for each block or pixel.

  In this embodiment, the block is divided into 16 blocks in the horizontal direction and 16 blocks in the vertical direction. In that case, it is necessary to match the number of divisions in the region division, the number of divisions when acquiring the AWB evaluation values (G / R, G / B), and the number of divisions when setting the WB correction coefficient.

1 is a schematic diagram showing an appearance of a digital camera that is a first embodiment of an imaging apparatus according to the present invention. 1 is a block diagram illustrating a configuration in a first embodiment of an imaging apparatus according to the present invention. It is a flowchart which shows the flow of a series of processes of the white balance correction | amendment in 1st Embodiment. FIG. 2 is a schematic diagram in which an imaging signal in the first embodiment of the imaging apparatus according to the present invention is divided into grid-like blocks, and an explanatory diagram for explaining a subject area and a background area in the schematic diagram. FIG. 5 is an explanatory diagram illustrating a subject area block center pixel and a background area block center pixel in the schematic diagram of FIG. 4. It is explanatory drawing of the calculation method of the WB correction coefficient by interpolation. It is explanatory drawing explaining the area | region 12 in the schematic diagram of FIG. It is explanatory drawing explaining the area | regions 13, 14, 15, and 16 in the schematic diagram of FIG. It is a flowchart which shows the flow of a series of processes of the white balance correction | amendment in 2nd Embodiment. It is a graph which shows the white extraction range in a G / R-G / B coordinate. It is a graph which shows the white extraction range of the sunny area | region in FIG. It is a graph which shows the white extraction range of the shade area | region in FIG. It is the schematic diagram by which the imaging signal in 2nd Embodiment of the imaging device which concerns on this invention was divided | segmented into the grid | lattice-like block, and explanatory drawing explaining the sunny area and shade area in this schematic diagram.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Subject area 2 Background area 3 Subject area attention pixel 4 Background area attention pixel 5 Shade area 6 Hinata area 12, 13, 14, 15, 16 area

Claims (15)

An image sensor that converts light incident from the optical system into an electrical signal and outputs the signal as an image signal;
Distance information acquisition means for acquiring distance information with a distance measurement object;
In an imaging apparatus comprising a white balance evaluation value acquisition unit that extracts a white balance evaluation value from the imaging signal,
Area dividing means for dividing the imaging signal into a plurality of areas based on the distance information and color difference information;
Correction coefficient calculating means for calculating a white balance correction coefficient for each area based on the white balance evaluation value for each divided area;
An imaging apparatus comprising: a white balance calculation unit that sets the white balance correction coefficient for each of the divided areas. The region dividing unit divides the image signal from distance information, color information, and luminance information into a high luminance region and a low luminance region,
For each of the high luminance region and the low luminance region, comprising white extraction means for performing white extraction by changing the white extraction range for each region,
The correction coefficient calculation means calculates a white balance correction coefficient for each area based on a white extraction result obtained by the white extraction means and a white balance evaluation value for each of the divided areas. The imaging apparatus according to claim 1. The white extraction means includes
In the high luminance region, white extraction is performed based on the white balance evaluation value of the high luminance region and the white extraction range for the high luminance region,
3. The imaging apparatus according to claim 2, wherein white extraction is performed in the low luminance area based on a white balance evaluation value of the low luminance area and a white extraction range for the low luminance area. Block dividing means for dividing the imaging signal into a plurality of blocks;
The divided block has a target pixel and a non-target pixel,
The white balance calculation means includes:
First correction coefficient acquisition means for acquiring the white balance correction coefficient to the target pixel for each of the divided blocks;
Second correction for acquiring the white balance correction coefficient by interpolation based on the white balance correction coefficient of the peripheral target pixel and the distance from the non-target pixel to the peripheral target pixel for the non-target pixel for each of the divided blocks. Coefficient acquisition means,
The imaging apparatus according to any one of claims 1 to 3, characterized in that setting the white balance correction coefficients obtained with the first correction coefficient acquisition unit by the second correction coefficient acquisition unit for each pixel. The imaging apparatus according to claim 4 , wherein the target pixel is a pixel located at a center of the divided block. A photoelectric conversion step of converting light incident from the optical system into an electrical signal and outputting it as an imaging signal;
A distance information acquisition step of acquiring distance information with a distance measurement target;
In an imaging method comprising a white balance evaluation value acquisition step of extracting a white balance evaluation value from the imaging signal,
A region dividing step of dividing the imaging signal into a plurality of regions based on the distance information and color difference information;
A correction coefficient calculation step of calculating a white balance correction coefficient for each area based on the white balance evaluation value for each divided area;
An imaging method comprising: a white balance calculation step of setting the white balance correction coefficient for each of the divided areas. The region dividing step divides into a high luminance region and a low luminance region from distance information, color information, and luminance information of the imaging signal,
For each of the high luminance region and the low luminance region, comprising a white extraction step of performing white extraction by changing the white extraction range for each region,
The correction coefficient calculating step calculates a white balance correction coefficient for each region based on a white extraction result obtained by the white extraction step and a white balance evaluation value for each of the divided regions. The imaging method according to claim 6 . The white extraction step includes
In the high luminance region, white extraction is performed based on the white balance evaluation value of the high luminance region and the white extraction range for the high luminance region,
The imaging method according to claim 7 , wherein white extraction is performed in the low luminance area based on a white balance evaluation value of the low luminance area and a white extraction range for the low luminance area. A block dividing step of dividing the imaging signal into a plurality of blocks;
The divided block has a target pixel and a non-target pixel,
The white balance calculation step includes:
A first correction coefficient acquisition step of acquiring the white balance correction coefficient to the target pixel for each of the divided blocks;
Second correction for acquiring the white balance correction coefficient by interpolation based on the white balance correction coefficient of the peripheral target pixel and the distance from the non-target pixel to the peripheral target pixel for the non-target pixel for each of the divided blocks. A coefficient acquisition step,
The imaging method according to any one of claims 6 to 8 , wherein the white balun correction coefficient acquired in the first correction coefficient acquisition step and the second correction coefficient acquisition step is set for each pixel. The imaging method according to claim 9 , wherein the target pixel is a pixel located at a center of the divided block. Photoelectric conversion processing for converting light incident from the optical system into an electrical signal and outputting it as an imaging signal;
Distance information acquisition processing for acquiring distance information with a distance measurement target;
A computer-readable recording medium recording a program for causing a computer to execute a white balance evaluation value acquisition process for extracting a white balance evaluation value from the imaging signal,
Based on the distance information and color difference information, an area division process for dividing the imaging signal into a plurality of areas;
A correction coefficient calculation process for calculating a white balance correction coefficient for each area based on the white balance evaluation value for each divided area;
A computer-readable recording medium comprising: a white balance calculation process for setting the white balance correction coefficient for each of the divided areas. The region division processing is divided into a high luminance region and a low luminance region from the distance information, color information, and luminance information of the imaging signal,
For each of the high luminance region and the low luminance region, comprising a white extraction process for performing white extraction by changing the white extraction range for each region,
The correction coefficient calculation process calculates a white balance correction coefficient for each area based on a white extraction result obtained by the white extraction process and a white balance evaluation value for each of the divided areas. The computer-readable recording medium according to claim 11 . The white extraction process includes
In the high luminance region, white extraction is performed based on the white balance evaluation value of the high luminance region and the white extraction range for the high luminance region,
13. The computer-readable recording medium according to claim 12 , wherein white extraction is performed in the low luminance area based on a white balance evaluation value of the low luminance area and a white extraction range for the low luminance area. A block division process for dividing the imaging signal into a plurality of blocks;
The divided block has a target pixel and a non-target pixel,
The white balance calculation process is:
A first correction coefficient acquisition process for acquiring the white balance correction coefficient to the target pixel for each of the divided blocks;
Second correction for acquiring the white balance correction coefficient by interpolation based on the white balance correction coefficient of the peripheral target pixel and the distance from the non-target pixel to the peripheral target pixel for the non-target pixel for each of the divided blocks. Coefficient acquisition processing, and
14. The computer-readable device according to claim 11 , wherein a white balun correction coefficient acquired by the first correction coefficient acquisition process and the second correction coefficient acquisition process is set for each pixel. Recording medium. The computer-readable recording medium according to claim 14 , wherein the target pixel is a pixel located at a center of the divided block.