JP2010103700A - Imaging device and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device and an imaging method expanding a dynamic range by one shooting operation, without generating composite images through a plurality of shooting operations while changing light exposure. <P>SOLUTION: The imaging device includes, a light source color detection means, a saturation level setting means, a pixel output detector 60, and a pixel output correction processor 61. Based on the result of the light source color detection means, the saturation level setting means sets each predetermined saturation level to the output from each pixel with an RGB filter. While detecting the output from each pixel, the pixel output detector 60 determines whether the detected pixel output reaches or exceeds each predetermined saturation level or not. When the pixel output detector 60 determines that the output from a pixel with a G filter reaches or exceeds the predetermined saturation level, the pixel output correction processor 61 corrects the output from the pixel with the G filter, based on output from pixels with R and B filters around the pixel. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method such as a digital still camera and a digital video camera, and more particularly to an imaging apparatus and an imaging method that can expand a dynamic range of a captured image.

  Compared to the dynamic range of images taken with a conventional silver salt camera using a silver salt photographic film, the dynamic range of images taken with a digital still camera or a digital video camera having a solid-state image sensor such as a CCD is extremely narrow. When the dynamic range is narrow, a phenomenon called “blackout” occurs in the dark part of the subject, and conversely, a phenomenon called “overexposure” occurs in the bright part of the subject, and the image quality deteriorates.

Therefore, in order to expand the dynamic range of an image captured by a solid-state imaging device such as a CCD, for example, the same subject is shot multiple times with different exposure amounts to obtain a plurality of images with different exposure amounts. A technique for adding these images to generate a composite image with an expanded dynamic range has been known (see, for example, Patent Document 1).
JP 2000-92378 A

  By the way, in order to expand the dynamic range, in the method of performing photographing a plurality of times while changing the exposure amount as in the above-mentioned Patent Document 1, when photographing a moving object, the subject is doubled. Therefore, the image may not be synthesized correctly.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus and an imaging method capable of expanding the dynamic range by one shooting without changing the exposure amount and performing a plurality of shootings to combine images. .

  In order to achieve the above object, according to the first aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and the front side of each pixel is provided. An image sensor in which a plurality of color separation filters are arranged, a light source color detection unit that detects a light source color of a subject at the time of shooting, and a filter of a specific color is arranged based on the light source color detected by the light source color detection unit Saturation level setting means for setting each predetermined saturation level with respect to each output from the arranged pixels and pixels in which filters of other colors other than the specific color are arranged, and detecting an output from each pixel, The pixel output detection means for determining whether or not the detected pixel output has reached the predetermined saturation level or more, and the filter of the specific color is arranged by the pixel output detection means When it is determined that any pixel output among the outputs from the pixels in which the filters of the other colors around the pixel are arranged with respect to the element has reached the predetermined saturation level or more for the pixel Pixel output correction processing means for correcting the output from the pixel in which the filter of the specific color is arranged based on the output from the pixel in which the filter of the other color around the pixel is arranged, It is characterized by having prepared.

  According to the second aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are provided on the front side of each pixel. An image sensor arranged, a light source color detection unit for detecting a light source color of a subject at the time of shooting, a pixel in which a filter of a specific color is arranged based on the light source color detected by the light source color detection unit, and the specific color Saturation level setting means for setting each predetermined saturation level for each output from a pixel in which a filter of a color other than those is arranged, and detecting the output from each pixel, and the detected pixel output is Pixel output detection means for determining whether or not a predetermined saturation level or higher is reached, and the output from the pixel in which the filter of the specific color is arranged is detected by the pixel output detection means. When it is determined that the predetermined saturation level or higher is reached, the output from the pixel in which the filter of the specific color determined to have reached the predetermined saturation level or higher is output to the pixel. And pixel output correction processing means for performing correction based on an output from a pixel in which the surrounding color filters are arranged.

  According to a third aspect of the present invention, the light source color detection means includes the filter of the specific color and the surroundings in each of the regions obtained by dividing the screen corresponding to the light receiving surfaces of all the pixels of the image sensor into a plurality of vertical and horizontal areas. It is characterized in that the light source color of the subject at the time of photographing is determined according to the ratio of the output values from each pixel in which filters of other colors are arranged.

  The invention according to claim 4 further includes correction coefficient calculation means for calculating a correction coefficient for adjusting white balance according to a light source at the time of shooting, wherein the correction coefficient calculation means includes the filter of the specific color and its surroundings. The correction coefficient is calculated based on a ratio of output values output from the pixels in which the other color filters are arranged.

  According to a fifth aspect of the present invention, a plurality of pixel output ratios from each pixel in which the filter of the specific color and the filters of the other colors around it are arranged corresponding to a plurality of light source colors having different color temperatures. The pixel output ratio storage means further includes a saturation level setting means, wherein the saturation level setting means is configured to determine each of the predetermined levels based on a light source color detected by the light source color detection means and a pixel output ratio stored in the pixel output ratio storage means. It is characterized by setting a saturation level.

  According to a sixth aspect of the present invention, when the pixel output detection means has a pixel in which at least two or more filters of the same color are arranged in a processing unit when the output of each pixel is detected, the pixel output The detecting means calculates an average value of outputs of the pixels in which the plurality of filters of the same color are arranged, and uses the calculated average value as a pixel output value in the processing unit.

  According to the seventh aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electric signal, and a plurality of color separation filters are provided on the front side of each pixel. In an imaging method of an imaging device including an image sensor arranged, a light source color detection step for detecting a light source color of a subject at the time of shooting, and a filter of a specific color based on the light source color detected by the light source color detection step A saturation level setting step for setting each predetermined saturation level for each output from the arranged pixel and a pixel in which a filter of a color other than the specific color is arranged; and detecting an output from each pixel A pixel output detection step for determining whether or not the detected pixel output has reached or exceeded each of the predetermined saturation levels, and the pixel output detection step, One of the outputs from the pixels where the other color filters around the pixel where the constant color filter is arranged is equal to or higher than the predetermined saturation level for the pixel When it is determined that the pixel has reached the pixel, the output from the pixel in which the filter of the specific color is arranged is corrected based on the output from the pixel in which the filter of another color around the pixel is arranged And a pixel output correction processing step.

  According to an eighth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are provided on the front side of each pixel. In an imaging method of an imaging device including an image sensor arranged, a light source color detection step for detecting a light source color of a subject at the time of shooting, and a filter of a specific color based on the light source color detected by the light source color detection step A saturation level setting step for setting each predetermined saturation level for each output from the arranged pixel and a pixel in which a filter of a color other than the specific color is arranged; and detecting an output from each pixel A pixel output detection step for determining whether or not the detected pixel output has reached or exceeded each of the predetermined saturation levels, and the pixel output detection step, When it is determined that the output from the pixel in which the constant color filter is arranged has reached the predetermined saturation level or higher for the pixel, the specific color determined to have reached the predetermined saturation level or higher is determined. And a pixel output correction processing step for performing correction on the output from the pixel in which the filter is arranged based on the output from the pixel in which the filter of the other color around the pixel is arranged. Yes.

  According to the present invention, when the output from the pixel in which the filter of the specific color is arranged reaches or exceeds a predetermined saturation level, the output from the pixel in which another color filter around the pixel is arranged is output. Based on this, the pixel output in the region above the saturation level is corrected to predictively interpolate, and the dynamic range is expanded to synthesize images without using a special sensor or by taking multiple shots with different exposures. Without this, the dynamic range can be expanded by a single shooting.

  Furthermore, by changing the saturation determination level according to the light source color of the shooting environment, the output balance from the pixels where the filter of the specific color is arranged and the pixels where the filter of the other color is arranged is larger than before the correction. Since it is possible to prevent the shift, a sense of incongruity due to a shift in hue (hue) can be reduced.

Hereinafter, the present invention will be described based on the illustrated embodiments.
<Embodiment 1>
FIG. 1A is a front view showing a digital still camera (hereinafter referred to as “digital camera”) as an example of an imaging apparatus according to Embodiment 1 of the present invention, and FIG. 1 (c) is a rear view thereof, and FIG. 2 is a block diagram showing an outline of a system configuration in the digital camera shown in FIGS. 1 (a), (b), and (c).

(Appearance structure of digital camera)
As shown in FIGS. 1A, 1B, and 1C, a release button (shutter button) 2, a power button 3, a shooting / playback switching dial 4 are provided on the upper surface side of the digital camera 1 according to the present embodiment. In the front (front) side of the digital camera 1, a lens barrel unit 6 having a photographing lens system 5, a strobe light emitting unit (flash) 7, and an optical viewfinder 8 are provided.

  On the back side of the digital camera 1, there are a liquid crystal monitor (LCD) 9, an eyepiece 8 a of the optical viewfinder 8, a wide-angle zoom (W) switch 10, a telephoto zoom (T) switch 11, and a menu (MENU) button 12. , A confirmation button (OK button) 13 and the like are provided. Further, a memory card storage unit 15 for storing a memory card 14 (see FIG. 2) for storing captured image data is provided inside the side surface of the digital camera 1.

(Digital camera system configuration)
As shown in FIG. 2, the digital camera 1 includes a CCD 20 serving as a solid-state imaging device on which a subject image incident through a photographing lens system 5 of a lens barrel unit 6 forms an image on a light receiving surface, and electrical signals output from the CCD 20. An analog front-end unit (hereinafter referred to as “AFE unit”) 21 that processes (analog RGB image signal) into a digital signal, a signal processing unit 22 that processes a digital signal output from the AFE unit 21, and temporarily stores data SDRAM 23 for controlling, ROM 24 for storing a control program, a motor driver 25 for driving the lens barrel unit 6 and the like.

  The lens barrel unit 6 includes a photographic lens system 5 having a zoom lens, a focus lens, and the like, an aperture unit 26, and a mechanical shutter unit 27. The drive units of the photographic lens system 5, the aperture unit 26, and the mechanical shutter unit 27 are as follows. It is driven by the motor driver 25. The motor driver 25 is driven and controlled by a drive signal from a control unit (CPU) 28 of the signal processing unit 22.

  As shown in FIG. 3, the CCD 20 has a Bayer array RGB primary color filter (hereinafter referred to as “RGB filter”) arranged on a plurality of pixels 20 a constituting the CCD 20. A signal (analog RGB image signal) is output.

  The AFE unit 21 is sampled by a TG (timing signal generating unit) 30 that drives the CCD 20, a CDS (correlated double sampling unit) 31 that samples an electrical signal (analog RGB image signal) output from the CCD 20, and the CDS 31. An AGC (analog gain control unit) 32 that adjusts the gain of the signal, and an A / D conversion unit 33 that converts the signal gain-adjusted by the AGC 32 into a digital signal (hereinafter referred to as “RAW-RGB data”).

  The signal processing unit 22 outputs a screen horizontal synchronization signal (HD) and a screen vertical synchronization signal (VD) to the TG 30 of the AFE unit 21, and an A / D conversion unit 33 of the AFE unit 21 in accordance with these synchronization signals. CCD interface (hereinafter referred to as “CCD I / F”) 34 that captures RAW-RGB data output from the CPU, a memory controller 35 that controls the SDRAM 23, and a YUV format that can display and record the captured RAW-RGB data. A YUV conversion unit 36 for converting to image data, a resizing processing unit 37 for changing the image size in accordance with the size of image data to be displayed or recorded, a display output control unit 38 for controlling display output of the image data, and an image A data compression unit 39 for recording data by JPEG formation and the like, and writing image data to the memory card 14; A digital camera based on a control program stored in the ROM 24 based on a media interface (hereinafter referred to as “media I / F”) 40 for reading image data written in the memory card 14 and operation input information from the operation unit 41. 1 is provided with a control unit (CPU) 28 for performing overall system control and the like.

  The operation unit 41 includes a release button 2, a power button 3, a photographing / playback switching dial 4, and a wide-angle zoom provided on the external surface of the digital camera 1 (see FIGS. 1A, 1 </ b> B, and 1 </ b> C). The switch 10, the telephoto zoom switch 11, the menu button 12, the confirm button 13, and the like, and a predetermined operation instruction signal is input to the control unit 28 by the photographer's operation.

  The SDRAM 23 stores the RAW-RGB data captured by the CCD I / F 34, and stores the YUV data (YUV format image data) converted by the YUV conversion unit 36. Further, the SDRAM 23 stores the RAW-RGB data. The compressed image data such as JPEG formation is stored.

  Note that YUV of the YUV data is luminance data (Y), color difference (difference (U) between luminance data and blue (B) component data, and difference (V) between luminance data and red (R) component data). It is a format that expresses color with information.

(Digital camera monitoring and still image shooting)
Next, the monitoring operation and still image shooting operation of the digital camera 1 will be described. In the still image shooting mode, the digital camera 1 performs a still image shooting operation while executing a monitoring operation as described below.

  First, when the photographer turns on the power button 3 and sets the photographing / playback switching dial 4 to the photographing mode, the digital camera 1 is activated in the recording mode. When the control unit 28 detects that the power button 3 is turned on and the shooting / playback switching dial 4 is set to the shooting mode, the control unit 28 outputs a control signal to the motor driver 25 to switch the lens barrel unit 6. The camera 20 is moved to a photographing position and the CCD 20, the AFE unit 21, the signal processing unit 22, the SDRAM 23, the ROM 24, the liquid crystal monitor 9 and the like are activated.

  Then, by directing the photographic lens system 5 of the lens barrel unit 6 toward the subject, a subject image incident through the photographic lens system 5 is formed on the light receiving surface of each pixel of the CCD 20. Then, an electrical signal (analog RGB image signal) corresponding to the subject image output from the CCD 20 is input to the A / D conversion unit 33 via the CDS 31 and the AGC 32, and 12 bits (bit) by the A / D conversion unit 33. To RAW-RGB data.

  This RAW-RGB data is taken into the CCD I / F 34 of the signal processing unit 22 and stored in the SDRAM 23 via the memory controller 35. The RAW-RGB data read from the SDRAM 23 is converted into YUV data (YUV signal) in a format that can be displayed by the YUV converter 36, and then the YUV data is stored in the SDRAM 23 via the memory controller 35. The

  The YUV data read from the SDRAM 23 via the memory controller 35 is sent to the liquid crystal monitor (LCD) 9 via the display output control unit 38, and a captured image is displayed. At the time of monitoring in which a photographed image is displayed on the liquid crystal monitor (LCD) 9 described above, one frame is read out in a time of 1/30 second by thinning out the number of pixels by the CCD I / F 34.

  In this monitoring operation, the photographed image is only displayed on the liquid crystal monitor (LCD) 9 functioning as an electronic viewfinder, and the release button 2 is not yet pressed (including half-pressed).

  The photographer can confirm the photographed image by displaying the photographed image on the liquid crystal monitor (LCD) 9. In addition, it can output as a TV video signal from the display output control part 38, and can display a picked-up image on external TV (television) via a video cable.

  The CCD I / F 34 of the signal processing unit 22 calculates an AF (automatic focus) evaluation value, an AE (automatic exposure) evaluation value, and an AWB (auto white balance) evaluation value from the captured RAW-RGB data.

  The AF evaluation value is calculated by, for example, the output integrated value of the high frequency component extraction filter or the integrated value of the luminance difference between adjacent pixels. When in the in-focus state, the edge portion of the subject is clear, so the high frequency component is the highest. By utilizing this, at the time of AF operation (at the time of focus detection operation), an AF evaluation value at each focus lens position in the photographing lens system 5 is acquired, and the point where the maximum is obtained is used as the focus detection position for AF. The action is executed.

  The AE evaluation value and the AWB evaluation value are calculated from the integrated values of the RGB values in the RAW-RGB data. For example, the screen corresponding to the light receiving surfaces of all the pixels of the CCD 20 is equally divided into 1024 areas (horizontal 32 divisions and vertical 32 divisions), and the RGB integration of each area is calculated.

  Then, the control unit 28 reads the calculated RGB integrated value, and in the AE process, calculates the luminance of each area of the screen and determines an appropriate exposure amount from the luminance distribution. Based on the determined exposure amount, exposure conditions (the number of electronic shutters of the CCD 20, the aperture value of the aperture unit 26, etc.) are set. In the AWB process, the subject color and the light source color are determined from the RGB distribution, and the AWB control value according to the light source color is determined. By this AWB process, white balance is adjusted when the YUV conversion unit 36 performs conversion to YUV data. The AE process and the AWB process described above are continuously performed during the monitoring operation.

  When the release button 2 is pressed (half-pressed to full-press) and the still image shooting operation is started during the monitoring operation described above, the AF operation and the still image recording process that are the focus position detection operations are performed. .

  That is, when the release button 2 is pressed (half-pressed to full-pressed), the focus lens of the photographing lens system 5 is moved by a drive command from the control unit 28 to the motor driver 25, and is referred to as so-called hill-climbing AF, for example. The contrast evaluation AF operation is executed.

  When the AF (focusing) target range is the entire region from infinity to close, the focus lens of the photographing lens system 5 moves to each focus position from close to infinity, or from infinity to close, and CCDI / The control unit 28 reads out the AF evaluation value at each focus position calculated in F34. Then, the focus lens is moved to the in-focus position with the point where the AF evaluation value at each focus position is maximized as the in-focus position, and in-focus.

  Then, when the AE process described above is performed and the exposure is completed, the mechanical shutter unit 27 opened by the drive command from the control unit 28 to the motor driver 25 is closed, and the analog RGB image signal for still image is output from the CCD 20. Is output. As in the monitoring, the A / D conversion unit 33 of the AFE unit 21 converts the data into RAW-RGB data.

  The RAW-RGB data is taken into the CCD I / F 34 of the signal processing unit 22, converted into YUV data by a YUV conversion unit 36 described later, and stored in the SDRAM 23 via the memory controller 35. The YUV data is read from the SDRAM 23, converted into a size corresponding to the number of recorded pixels by the resizing processing unit 37, and compressed into image data such as JPEG format by the data compression unit 39.

  The compressed image data such as JPEG format is written back to the SDRAM 23, read out from the SDRAM 23 through the memory controller 35, and stored in the memory card 14 through the media I / F 40.

(Principle of dynamic range expansion in the present invention)
A Bayer-arranged RGB filter (see FIG. 3) is arranged on each pixel constituting the CCD 20 of the digital camera 1, but a normal RGB filter for light having a wide wavelength band such as sunlight. Have different luminance sensitivities for each color.

  For example, as shown in FIG. 4, an RGB filter (a, b, c in FIG. 4) has a pixel sensitivity of a G (green) filter that is about twice that of an R (red) filter and a B (blue) filter. In the case of a CCD 20 having a large wavelength band such as sunlight, when the same amount of light having a wide wavelength band is incident on the RGB filter, the G filter (hatched portion of c in FIG. 4) is output for each pixel output of the R and B filters. ) Reaches the saturation level A first. In FIG. 4, f is the pixel sensitivity characteristic of the G filter, g is the pixel sensitivity characteristic of the R and B filters, and the pixel sensitivity characteristic of the G filter is about twice the pixel sensitivity characteristics of the R and B filters. It has the sensitivity of.

  By the way, in a conventional digital camera having a solid-state imaging device such as a CCD in which an RGB filter is arranged, a saturation level corresponding to the pixel output of a high-sensitivity G filter, such as the RGB filters of a, b, and c in FIG. The dynamic range is set according to A. For this reason, even when the pixel output of the G filter reaches the saturation level A, the pixel output of the R and B filters is about ½ of the saturation level A.

  On the other hand, in the present invention, even if the pixel output of the G filter exceeds the saturation level A, as in the RGB filters of d and e in FIG. From within the pixel output levels of the R and B filters, the pixel sensitivity characteristics of the R and B filters (g in FIG. 4) and the pixel sensitivity characteristics of the G filter (f in FIG. 4) Based on this, the pixel output level of the G filter is corrected so as to perform prediction interpolation (dotted line portion), and the dynamic range is expanded by this prediction interpolation (correction).

  As described above, in the present embodiment, the pixel sensitivity characteristic of the G filter has a sensitivity about twice that of each pixel sensitivity characteristic of the R and B filters with respect to light having a wide wavelength band such as sunlight. ing. Therefore, the maximum value of the dynamic range expansion degree in the present embodiment is about twice that of a normal photographing operation in which the dynamic range expansion processing operation is not performed.

  In this embodiment, the pixel sensitivity characteristic of the G filter has a sensitivity that is about twice that of the pixel sensitivity characteristics of the R and B filters, and based on this, the maximum value of the degree of expansion of the dynamic range is doubled. However, by changing the pixel sensitivity characteristics of the RGB filter, the maximum value of the degree of expansion of the dynamic range can be set to a predetermined value of 2 times or more, or a predetermined value of 2 times or less.

(Dynamic range expansion processing by the YUV converter 36)
The YUV conversion unit 36 of the digital camera 1 according to the present embodiment has a dynamic range expansion processing function for expanding the dynamic range described above.

  As shown in FIG. 5, the YUV conversion unit 36 includes a dynamic range expansion prediction interpolation unit (hereinafter referred to as “D range expansion prediction interpolation unit”) 50, a bit compression conversion unit 51, a white balance control unit 52, and synchronization described later. A unit 53, a tone conversion unit 54, an RGB-YUV conversion unit 55, an image size converter unit 56, a luminance histogram generation unit 57, and an edge enhancement unit 58.

  As illustrated in FIG. 6, the D range expansion prediction interpolation unit 50 includes a pixel output detection unit 60, a pixel output correction processing unit 61, and a bit extension processing unit 62.

  The pixel output detection unit 60 detects the pixel output of each pixel provided with the RGB filter from the input RAW-RGB data, and also outputs the pixel output of the pixel provided with the highest sensitivity G filter (hereinafter referred to as “G filter”). It is determined whether or not the “pixel output of” has reached a predetermined saturation level or higher.

  When the pixel output detection unit 60 determines that the pixel output of the G filter has reached a predetermined saturation level or higher, the pixel output correction processing unit 61 provides R and B filters around the pixel where the G filter is disposed. Correction (predictive interpolation) processing of the pixel output of the G filter that has reached a predetermined saturation level or higher by the pixel output of the selected pixel (hereinafter referred to as “pixel output of the R and B filters”) (details will be described later).

  When the pixel output detection unit 60 determines that the pixel output of the G filter has not reached the saturation level, the bit extension processing unit 62 performs the pixel output of the G filter and the pixel output of the R and B filters. Only bit expansion is performed from 12 bits to 14 bits without correcting the output level.

  The dynamic range expansion processing operation in this embodiment will be described below.

  During the monitoring operation described above, for example, when there is an extremely bright part of the background of the subject to be photographed, the menu button 12 (see FIG. 1C) is pressed by the photographer's judgment, For example, a shooting setting screen as shown in FIG. 7 is displayed on the liquid crystal monitor (LCD) 9.

  Then, by selecting “ON” in the “dynamic range expansion” item from this display screen, a control signal is output from the control unit 28 to the YUV conversion unit 36, and a processing operation for expanding the dynamic range is executed.

  In the present embodiment, as described above, it is assumed that the pixel sensitivity characteristics of the G filter have sensitivity that is about twice as high as the pixel sensitivity characteristics of the R and B filters. However, if the light source color of the subject at the time of shooting changes, the ratio of the pixel sensitivity of the R and B filters to the pixel sensitivity of the G filter changes. For example, under extremely red or blue light sources, the pixel sensitivity of the R and B filters may be saturated more than the pixel sensitivity of the G filter. Under such circumstances, if dynamic range expansion processing is performed, accurate gradation and color reproduction cannot be obtained. In such a case, according to the photographer's judgment, the item “dynamic range expansion” described above is used. Do not select “ON”.

  The RAW-RGB data read from the SDRAM 23 is input to the pixel output detection unit 60 of the D range expansion prediction interpolation unit 50.

  The pixel output detection unit 60 detects the pixel output of each pixel provided with the RGB filter from the input RAW-RGB data, and also outputs the pixel output of the pixel provided with the G filter (hereinafter referred to as “G filter pixel output”). ) And the peripheral pixel outputs of the pixels provided with the R and B filters (hereinafter referred to as “R and B filter pixel outputs”) have reached a predetermined saturation level or higher. Determine. Further, the pixel output detection unit 60 corrects the pixel output of the G filter (predictive interpolation) when either the pixel output of the G filter or the pixel output of the R or B filter has reached a predetermined saturation level or higher. A correction coefficient to be described later is calculated.

  The pixel output correction processing unit 61 performs correction processing of the pixel output of the G filter by multiplying the pixel output of the G filter by the correction coefficient calculated by the pixel output detection unit 60.

  When the pixel output detection unit 60 calculates the correction coefficient for the pixel output of the G filter, in this embodiment, for each pixel of the CCD 20 having the RGB filter, as shown in FIG. A processing unit (minimum unit) is 2 × 2 pixels (two G filter pixels, one R and B filter pixels). The correction coefficient (K) of the pixel output of the G filter and the pixel output (Ge) of the G filter after the correction process are calculated from the following expressions (1) and (2), respectively.

K = {l * f (Ro) + m * f (Go) + n * f (Bo)} / 3 Formula (1)
Ge = K × Go (2)
Here, l, m, and n are coefficients set from the ratios of the RGB filters, and Go is the pixel output of the G filter before correction processing. Further, f (Ro), f (Go), and f (Bo) are coefficients set by the following equation 1 (Equation (3) to Equation (5)).

  Where Ro is the pixel output of the R filter, TR is the saturation judgment level of the pixel output of the R filter, Go is the pixel output of the G filter before correction processing, TG is the saturation judgment level of the pixel output of the G filter, and Bo is the B filter The pixel output TB is a saturation judgment level of the B filter pixel output.

  The saturation determination levels TR, TG, and TB in the equations (3) to (5) correspond to, for example, predetermined saturation determination levels for each pixel output of the RGB filter under a certain light source shown in FIG. In FIG. 8, A (TG) is the saturation level (saturation determination level) of the pixel output of the G filter, TR is the saturation determination level of the pixel output of the R filter, and TB is the saturation determination level of the pixel output of the B filter. .

  As shown in FIG. 8, under this light source, the sensitivity of the pixel provided with the G filter is twice the sensitivity of each pixel provided with the R and B filters as described above. The pixel output first reaches the saturation point. For this reason, the pixel output value that has reached the saturation point is set as the saturation determination level TG of the pixel output of the G filter. Therefore, the saturation determination levels TB and TG of the pixel outputs of the R and B filters are ½ of the TG. Note that TR, TG, and TB depend on the characteristics (sensitivity ratio) of the solid-state imaging device (CCD and the like) having an RGB filter used in the imaging device (digital camera and the like) and the color temperature of the light source, and are shown in FIG. It is not limited to such a ratio.

  When the sensitivity ratio of each pixel output of the RGB filter is as shown in FIG. 8, the coefficients l and n in the above equation (1) are 3/2 and m is 0, respectively. A correction coefficient (K) is calculated. Then, the corrected pixel output (Ge) value of the G filter calculated from the equation (2) using the correction coefficient (K) is the pixel output of the two G filters in the processing unit (see FIG. 3). Replaced as a value.

  Since the pixel output value of the G filter exceeds 12 bits, it is replaced with 14-bit data here. Therefore, since the maximum value of each pixel output of the R and B filters is 4095 (12 bits), the maximum value of the pixel output of the G filter is 8190 and can be handled as 14-bit data.

  By the way, before the pixel output detection unit 60 calculates the correction coefficient of the pixel output of the G filter, it is necessary to complete the correction of the defective pixel. That is, when there is a defective pixel in each pixel provided with the RGB filter and there is a pixel that always outputs a saturated value, the correction coefficient becomes a large value, and as a result, the pixel output of the corrected G filter Is replaced with a large value, and a new defective pixel is generated. For this reason, in this embodiment, the CCD I / F 34 includes a defective pixel output removal processing unit (not shown) that removes the output of the defective pixel.

  In the above formulas (3) to (5), when each pixel output of the RGB filter is equal to or lower than a predetermined saturation determination level (TR, TG, TB), “1” is set. This is to prevent the pixel output (Ge) value of the subsequent G filter from becoming smaller than the pixel output (Go) of the G filter before correction.

  For example, as shown in FIG. 9A, the pixel output of the G filter in the processing unit (see FIG. 3) has reached the saturation level A or higher, and the pixel outputs of the surrounding R and B filters are also When it is high to some extent (saturation level A or less with respect to the pixel output of the G filter), as shown in FIG. 9B, the pixel output of the G filter is set to the saturation level based on the equations (1) to (5). It is possible to expand the dynamic range by correcting so as to expand beyond A.

  However, for example, as shown in FIG. 10A, even when the pixel output of the G filter in the processing unit (see FIG. 3) reaches the saturation level A or higher, the surrounding R and B filters Each pixel output may be extremely small relative to the pixel output of the G filter. In this case, in the formulas (3) to (5), the determination is made that “1” is set when each pixel output of the RGB filter is smaller than a predetermined saturation determination level (TR, TG, TB). If the pixel output of the G filter is not corrected, the pixel output of the G filter is reversed to the saturation level due to the influence of the R and B filters whose surrounding pixel outputs are extremely small as shown in FIG. A problem of being reduced to A or less occurs.

  Therefore, by performing comparison with predetermined saturation determination levels (TR, TG, TB) as in the above formulas (3) to (5), the calculation result is as shown in FIG. 9B. Only when the pixel output value of the G filter is larger, the dynamic range expansion is corrected. In FIGS. 9A and 9B and FIGS. 10A and 10B, A (TG) is the saturation level (saturation determination level) of the pixel output of the G filter, and TR is the pixel output of the R filter. The saturation determination level TB is the saturation determination level of the B filter pixel output.

  In this way, even when the pixel output of the G filter reaches the saturation level A or higher, if the pixel outputs of the surrounding R and B filters are extremely small relative to the pixel output of the G filter, the G filter By canceling the correction process for the pixel output, the pixel output of the G filter can be lowered and color misregistration can be prevented.

  Here, a method for setting the predetermined saturation determination levels (TR, TG, TB) will be described.

  FIG. 11 is a characteristic diagram showing an example of a ratio of each pixel output of the RGB filter with respect to a difference in color temperature (2000K (Kelvin) or lower to 9000K (Kelvin) or higher), which is a unit representing the color of the light source. As apparent from FIG. 11, the pixel output of the G filter hardly changes even when the color temperature changes, but the pixel output of the R filter increases as the color temperature decreases (red light source), and the pixel of the B filter The output increases as the color temperature increases (blue light source).

  As described above, the ratio of the pixel output values of the RGB filter depends on the characteristics of the solid-state imaging device (CCD or the like) itself and the color temperature of the light source. Therefore, when the CCD 20 with the RGB filter is incorporated in the digital camera 1, the ratio relationship of the pixel output values of the RGB filter with respect to different color temperatures as shown in FIG. 11 corresponding to the CCD 20 with the RGB filter is shown. The table shown (hereinafter referred to as “RGB output ratio table”) is stored in the ROM 24.

  Then, after determining the color temperature of the light source of the subject from the RGB distribution that is the AWB evaluation value acquired from the CCD I / F 34, the control unit 28 reads the RGB output ratio table from the ROM 24, and determines the prescribed saturation according to the ratio. Set levels (TR, TG, TB). In this way, by switching the specified saturation determination level according to the color of the light source, the saturation level determination can be performed more accurately, and correction of the pixel output of the G filter that has reached the saturation level or higher (predictive interpolation) Can be performed more accurately. In the present embodiment, the light source color detection means and the saturation level setting means in “Claims” correspond to the control unit 28, and the pixel output ratio storage means corresponds to the ROM 24.

  Then, the pixel output value data of the G filter corrected by the pixel output correction processing unit 61 of the D range expansion prediction interpolation unit 50, and the pixel output value data of the R and B filters that are only bit extended by the bit extension processing unit 62 Is output to the bit compression conversion unit 51.

  For example, the bit compression conversion unit 51 expands to 14 bits by conversion characteristics as shown in FIG. 12A (four-segment broken line approximation characteristics in which three nodes are designated and approximated by a straight line between them). Of the RGB filter pixel outputs, the G filter pixel output is compressed to 12 bits. In FIG. 12A, a is a 12-bit range, and b is a simple linear conversion characteristic (one-dot chain line portion) that halves the data of the maximum value 8190.

  In the conversion characteristic shown in FIG. 12A, since the maximum value of the pixel output of the G filter is 8190, compression is performed so that 8190 becomes 4095. Then, the pixel output values of the R and B filters are also compressed in accordance with the compression magnification of the pixel output of the G filter.

  As described above, in this embodiment, as an example of the case where the pixel output of the G filter whose maximum value is expanded to 8190 is compressed to 4095, three nodes as shown by the solid line in FIG. A conversion characteristic with In the present embodiment, the following two effects that cannot be obtained with a linear conversion characteristic without a simple node (b in FIG. 12A) are obtained.

  As a first effect, a larger number of bits can be assigned to data with high data reliability. In other words, when correction processing is performed on the pixel output of the G filter that has reached the saturation level or higher, as described above, the correction (predictive interpolation) is performed for the range in which the value of the pixel output of the G filter exceeds the specified value near the saturation level. ), And no correction is made within the specified range. Therefore, the accuracy of data differs between the range where correction is performed and the range where correction is not performed.

  For example, when correcting (predicting interpolation) the pixel output value of the G filter that is saturated according to the equation (1), the luminance level of the subject can be accurately reproduced in the correction range depending on the color of the main subject. There may not be. On the other hand, since the range where correction is not performed is data obtained by A / D conversion of actual data (analog RGB image signal) output from the CCD 20 in which the RGB filter is arranged, the reliability of this data is high. It will be a thing.

  That is, in the conversion characteristic (part indicated by the solid line) in this embodiment shown in FIG. 12A, for example, when the input 14-bit data is 1024, the output 12-bit data is 1024, and the original data is It is used as it is. On the other hand, for example, when the input 14-bit data is 3072, the output 12-bit data is 2560, and in this range, the bit allocation is smaller than the bit allocation before correction, so that some bit error occurs.

  In this way, by adopting a conversion characteristic (part indicated by a solid line) having three nodes as in this embodiment, instead of a characteristic for performing linear conversion without a simple node (part indicated by a one-dot chain line). Since the bit allocation can be gradually reduced, a larger number of bits can be allocated to data with high data reliability.

  As a second effect, gradations at low and medium luminance can be accurately stored. That is, when bit compression is performed with a simple linear conversion characteristic (b in FIG. 12A), the bit allocation becomes ¼ in a range where predictive interpolation on the low luminance side is not performed. For this reason, the image has no tone. On the other hand, when bit compression is performed with the conversion characteristic (solid line portion) in the present embodiment as shown in FIG. 12A, the data corresponding to the pixel output at the low luminance level below the saturation level is applied. Thus, by using a compression rate that gives substantially the same value before bit compression by the bit compression conversion unit 51 and after bit compression, it is possible to maintain good gradation at a low luminance level.

  In this embodiment, when the 14-bit data of the pixel output of the expanded G filter is reduced to 12 bits, three nodes are designated as shown in FIG. Although the bit compression is performed with the broken line approximation characteristic (conversion characteristic) of four sections, the number of sections is not particularly limited. For example, it may be a polygonal line approximate characteristic of two sections designating one node, but the above-mentioned two effects are reduced by changing the bit allocation in the vicinity of the node. Therefore, a broken line approximation characteristic (conversion characteristic) having three or more sections is preferable.

  Further, the conversion characteristic for reducing the 14-bit data of the pixel output of the expanded G filter to 12 bits may be a conversion characteristic based on a curve not having a plurality of nodes as shown in FIG. That is, with respect to the conversion characteristic having four sections in FIG. 12A, the conversion characteristic by this curve is obtained by setting the number of sections to 8192. In FIG. 12B, a is a 12-bit range.

  Further, by providing a lookup table having numeric data after reduction conversion to 12 bits for 0 to 8192 of input 14-bit data, the pixel output of the expanded G filter with the conversion characteristics by this curve is provided. The 14-bit data can be successfully compressed to 12 bits.

  The pixel output data of the RGB filter that has been compression-converted from 14 bits to 12 bits by the bit compression conversion unit 51 is input to the white balance control unit 52.

  The white balance control unit 52 multiplies the input pixel output value data of the RGB filter by a correction coefficient for adjusting white balance. This correction coefficient is calculated by the control unit 28 based on the AWB evaluation value generated by the CCD I / F 34. The correction coefficient is a coefficient that makes the light source color white. Thus, in the present embodiment, the correction coefficient calculation means in “Claims” corresponds to the white balance control unit 52.

  Specifically, based on the ratio of the RGB filter pixel output values to different color temperatures as shown in FIG. 11, the R filter pixel output values include G / R (G filter pixel output as a correction coefficient). Value / R filter pixel output value), and the B filter pixel output value is multiplied by G / B (G filter pixel output value / B filter pixel output value) as a correction coefficient. Match the level of the pixel output value of the RGB filter.

  By the way, by the correction processing in the D-range expansion prediction interpolation unit 50 described above, the ratio of the pixel output values of the RGB filter is a value close to the ratio of the pixel output values of the original RGB filter as shown in d and e of FIG. It has become.

  Therefore, as a result of multiplying the output from the D range expansion prediction interpolation unit 50 by the correction coefficient by the white balance control unit 52, the color misregistration is small and the uncomfortable feeling is reduced.

  Then, the pixel output value data (12 bits) of the RGB filter with the white balance adjusted from the white balance control unit 52 is input to the synchronization unit 53. The synchronization unit 53 performs an interpolation calculation process on the RAW data having only one color data in one pixel of the Bayer array, and generates all RGB data for one pixel.

  All the RGB data (12 bits) generated by the synchronization unit 53 is input to the tone conversion unit 54. The tone conversion unit 54 performs γ conversion that converts 12-bit RGB data into 8-bit RGB data by using a conversion table as shown in FIG. 13, generates an 8-bit RGB value, and performs RGB-YUV conversion. To the unit 55.

  The RGB-YUV conversion unit 55 converts input RGB data (8 bits) into YUV data by matrix calculation and outputs the YUV data to the image size converter unit 56. The image size converter unit 56 reduces or enlarges the input YUV data (8 bits) to a desired image size, and outputs it to the luminance histogram generation unit 57 and the edge enhancement unit 58.

  The luminance histogram generation unit 57 generates a luminance histogram based on the input YUV data. The edge enhancement unit 58 performs processing such as edge enhancement according to the image on the input YUV data, and stores the processed data in the SDRAM 23 via the memory controller 35.

  As described above, in this embodiment, even when the pixel output of the G filter with high sensitivity in the processing unit exceeds the saturation level, the pixel is saturated based on the pixel output of the R and B filters with low sensitivity. As shown in FIG. 4, the pixel output of the G filter (d, e in FIG. 4) is corrected (predicted interpolation). Based on the pixel output of the G filter pixel outputs d and e in FIG. 4, the dynamic range can be doubled by one shooting.

  Therefore, even when there is a high-luminance portion in the background or the like in the captured image, it is possible to prevent the occurrence of overexposure and obtain good gradation.

  Furthermore, in the present embodiment, as described above, the predetermined saturation determination levels (TR, TG, TB) are set in accordance with the color temperature of the light source in the shooting environment, so that the expressions (1) and (2) The calculation result of becomes more accurate. As a result, it is possible to prevent the output balance of each pixel output of the RGB filter before and after correcting the pixel output of the saturated G filter, thereby reducing a sense of incongruity due to a hue (hue) shift. Can do.

  FIG. 14A shows an example of a histogram generated by the luminance histogram generation unit 57 when the dynamic range expansion processing in the present embodiment described above is performed when the pixel output of the G filter exceeds the saturation level. It is. As is apparent from this histogram, by performing the dynamic range expansion processing, whiteout in the maximum luminance portion (255) hardly occurs, and reproduction is performed with good gradation.

  On the other hand, FIG. 14B is generated by the luminance histogram generation unit 57 when the dynamic range expansion processing in the present embodiment is not performed when the pixel output of the G filter exceeds the saturation level. It is an example of the done histogram. As can be seen from this histogram, when the dynamic range expansion process is not performed, the maximum luminance portion (255) has a frequency and whiteout occurs.

  In the description of the first embodiment and FIGS. 4 and 8, the saturation level A in FIG. 4 that is a predetermined saturation level determination value and 4095 that is the 12-bit maximum value after correction, or the saturation in FIG. Although it has been described that the determination level TR and the corrected 12-bit maximum value 4095 match each other, the present invention is not limited to this. For example, in a CCD having an RGB filter whose output linearity (linearity) is not good in a high-luminance part near where the output is completely saturated, for example, 4032 which is a value smaller than 4095 which is completely saturated is set to a predetermined saturation level. As a value (saturation level A in FIG. 4), a pixel output exceeding that value may be subjected to correction processing.

  Also, depending on the system configuration of the digital camera, even a high-luminance subject may not reach 4095, which is the maximum value of 12 bits. In that case as well, the predetermined saturation level may be set to a value lower than 4095.

  Thus, even when less than 4095 is set to a predetermined saturation level, the output of the bit compression conversion unit 51 can be set to 4095 by switching the conversion curve of FIG. The dynamic range can be expanded without changing the process.

  In the present embodiment, for example, when the values of the pixel output of the G filter and the pixel output of the R filter (or B filter) reach the saturation level or more, they are calculated from the equations (1) to (5). Since the corrected pixel output value of the G filter becomes inaccurate and bit compression is performed at the compression rate used for the pixel output of the G filter without correcting the pixel output value of the R filter (or B filter), the hue May change.

Therefore, when the values of the pixel output of the G filter and the pixel output of the R filter (or B filter) have reached at least the saturation level, the dynamic range expansion processing by the correction described above may not be performed. preferable. Alternatively, the fact that the pixel output values of a plurality of pixels (G filter pixel output and R filter (or B filter)) have reached a saturation level or higher assumes that the brightness of the processing unit area is extremely bright, A predetermined value for the pixel output value of the G filter, for example,
G filter pixel output = 4096 x 1.8 = 7372 (14 bits)
You may set it.

  In this embodiment, as shown in FIG. 5, 14-bit RAW-RGB data (RGB filter pixel output data) output from the D-range expansion prediction interpolation unit 50 is converted into 12 bits by the bit compression conversion unit 51. The white balance control unit 52 and the synchronization unit 53 are configured to perform 12-bit data processing, but in addition to this, a bit compression conversion unit 51 is provided after the synchronization unit 53, The 14-bit data output from the synchronization unit 53 may be reduced to 12-bit data.

<Embodiment 2>
In the first embodiment, the correction (prediction interpolation) of the pixel output of the G filter is performed when any of the pixel output of the G filter and the pixel output of the R and B filters has reached the saturation level or more. In this embodiment, only when the pixel output of the G filter is saturated, the correction (predictive interpolation) processing of the pixel output of the G filter by the D-range expansion prediction interpolation unit 50 (see FIG. 6) is performed. .

  In the present embodiment, in the pixel output correction processing unit 61 of the D-range expansion prediction interpolation unit 50 (see FIG. 6), the pixel output of the G filter is saturated as defined by the following formula 2 (equation (6)). The correction process is performed only when the level is exceeded.

  However, Ge is the pixel output of the G filter after correction, Go is the pixel output of the G filter before correction processing, K is the correction coefficient, and TG is the saturation determination level of the pixel output of the G filter.

  In this way, by performing the correction process according to the equation (6), for example, only the pixel output of the R and B filters is large depending on the color of the subject, and the ratio of the pixel outputs of the RGB filter is as shown in FIG. Even when it is extremely deviated, it is possible to prevent the pixel output value of the G filter from deviating.

<Embodiment 3>
In the present embodiment, it is assumed that a white balance correction coefficient is used to determine the prescribed saturation determination levels (TR, TG, TB).

  In the first embodiment, the color temperature is estimated from the AWB evaluation value, the pixel output ratio as shown in FIG. 11 stored in the ROM 24 corresponding to the estimated color temperature is read, and the prescribed saturation determination level (TR , TG, TB). However, this method may not be able to completely cope with the light source color of the shooting environment. That is, FIG. 11 shows the ratio of the output of each pixel of the RGB filter in the light source having the color temperature of the black body radiation locus. In an artificial light source such as a fluorescent lamp, the distance from the locus of the black body radiation is shown. is there. For example, a white fluorescent lamp having poor color rendering properties has a color temperature of about 4000 K, but is shifted in the green direction from the black body radiation locus.

  As described above, the white balance correction coefficient calculation in the present embodiment is extremely far from the black body radiation locus from the average value of each pixel output of the RGB filter of each area obtained by dividing the screen by 1024 as described above. An AWB evaluation value of an area that can be determined not to be a color is removed. Then, the AWB evaluation values of the remaining areas are integrated, and the correction coefficient (WBr) for R filter pixel output is WBr = G / R, and the correction coefficient for pixel output (WBb) of the B filter is WBb = G / B Is calculated.

  Therefore, in the white balance control unit 52, the R filter pixel output is multiplied by the R filter pixel output correction coefficient (WBr = G / R), and the B filter pixel output is corrected by the B filter pixel output correction coefficient. (WBb = G / B) is multiplied. As a result, in the achromatic portion, correction is performed so that each pixel output of the RGB filter is the same as the pixel output of the G filter, and the white portion looks white.

  In the present embodiment, only the prescribed saturation level TG of the pixel output of the G filter is stored in the ROM 24. Since the prescribed saturation level TG has a small change with respect to the change in the light source color temperature as shown in FIG. 11, only one is stored.

Then, the control unit 41 uses the R filter pixel output correction coefficient (WBr) and the B filter pixel output correction coefficient (WBb) determined as described above, respectively. From 8), prescribed saturation levels (TR, TB) for output of R and B filter pixels are calculated.
TR = TG × WBr (7)
TB = TG × WBb (8)

  As in the present embodiment, by calculating the prescribed saturation level (TR, TB) from the white balance correction coefficient, it is possible to satisfactorily determine the saturation level corresponding to the light source even for an artificial light source deviated from the blackbody radiation track. It can be carried out.

<Embodiment 4>
In the first embodiment, as shown in FIG. 3, the processing unit (minimum unit) is 2 × 2 pixels for the CCD 20 having the RGB filter. However, in the present embodiment, as shown in FIG. Five pixels in the frame A (one G filter pixel, two R (R1, R2) and two B (B1, B2) filter pixels) are processing units (minimum units), and the processing unit is the above embodiment. In this example, the range is wider than in the case of 1. The configuration of the digital camera, the monitoring operation, the still image shooting operation, and the dynamic range expansion processing operation are the same as those in the first embodiment.

  When the pixel output of the G filter in the processing unit of the thick frame A shown in FIG. 15 has reached the saturation level or higher, the sensitivity of the G filter is about twice the sensitivity of the R and B filters as described above. Therefore, the correction coefficient (K) of the pixel output of the G filter and the pixel output (Ge) of the corrected G filter are calculated from the following equations (9) and (10). In this embodiment,

K = {l × f (Ra) + m × f (Ga) + n × f (Ba)} / 3 Equation (9)
Ge = K × Ga (10)
Here, l, m, and n are coefficients set from the sensitivity ratios of the RGB filters, and Ga is the pixel output of the G filter before correction. Further, f (Ra), f (Ga), and f (Ba) are coefficients set by the following Equation 3 (Expression (11) to Expression (13)).

  However, Ra is the average value of the pixel output of the R filter in the processing unit (see FIG. 15), TR is the saturation judgment level of the pixel output of the R filter, and Ga is G in the processing unit (see FIG. 15). The pixel output of the filter, TG is the saturation judgment level of the pixel output of the G filter, Ba is the average value of the pixel output of the B filter in the processing unit (see FIG. 15), and TB is the saturation judgment level of the pixel output of the B filter. It is.

  The TR, TG, and TB are the same as those in the equations (3) to (5). The coefficients l, m, and n are 3/2 and m are 0, respectively, if the sensitivity ratio of the pixel outputs of the R, G, and B filters is the same as in the first embodiment.

  Then, the pixel output correction processing unit 61 of the D range expansion prediction interpolation unit 50 shown in FIG. 6 sets the pixel output value of the G filter calculated by the equation (10) in the processing unit (see FIG. 15). The pixel output value of a certain G filter is replaced, and the same processing as in the first embodiment is performed.

  In this way, by widening the processing unit, it is possible to reduce the influence due to the sensitivity difference of the other R1, R2 filter pixels and B1, B2 filter pixels in the processing unit. More accurate dynamic range expansion prediction interpolation can be performed on the output.

<Embodiment 5>
In the present embodiment, as shown in FIG. 16, the processing unit (thick frame A) is wider than that in the fourth embodiment with respect to the CCD 20 having the RGB filter. The configuration of the digital camera, the monitoring operation, the still image shooting operation, and the dynamic range expansion processing operation are the same as those in the first embodiment. However, formulas (9) to (13) are used as calculation formulas for predictive interpolation for the pixel output of the G filter, as in the fourth embodiment.

  If the processing unit is widened, processing is performed based on a wide range of luminance information, which is equivalent to applying a low-pass filter. Therefore, the edge portion of the luminance change is lost. Therefore, in the present embodiment, the size of the widened processing unit is partially changed using, for example, the AF evaluation value.

  That is, the CCD I / F 34 of the signal processing unit 22 shown in FIG. 2 calculates an AF evaluation value for performing AF as described above. This is an output of a high-pass filter (HPF), and a large value is output in a portion where there is a change in luminance in the screen of the captured image. Then, the control unit 28 reads out an AF evaluation value at the time of still image shooting, and discriminates between a portion having a luminance change and a portion having no luminance change in the screen. Then, the control unit 28 sets the D range expansion prediction interpolation unit 50 based on this determination data so that the processing unit is narrowed in a portion where there is a luminance change, and in a portion where there is no luminance change, As shown in FIG. 16, setting is performed so that the processing unit is in a wide range.

  Thus, even when the processing unit is further widened, accurate dynamic range expansion prediction interpolation can be performed without reducing the resolution by setting the processing unit to be narrower in a portion where there is a luminance change. .

  In each of the embodiments described above, the RGB three primary color filters are arranged as the color separation filters. However, the present invention is similarly applied to a configuration in which the complementary color filters are arranged as the color separation filters. Can do.

  In each of the above-described embodiments, the description has been made on the imaging device such as a digital camera having a photographing lens system and a solid-state imaging device such as a CCD. However, the present invention can be similarly applied to an image processing device. It is. For example, the present invention is similarly applied to an image processing apparatus in which analog data from a single-plate color image sensor is converted into digital signals and stored, and RAW data is input to output RGB data or YCbCr data. It is possible.

(A) is a front view which shows the digital camera as an example of the imaging device which concerns on Embodiment 1 of this invention, (b) is the top view, (c) is the back view. 1 is a block diagram showing an outline of a system configuration in a digital camera as an example of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram illustrating a pixel arrangement position and a processing unit of a CCD in which an RGB filter according to Embodiment 1 of the present invention is arranged. The figure for demonstrating the principle of the dynamic range expansion in Embodiment 1 of this invention. The block diagram which shows the structure of the YUV conversion part in Embodiment 1 of this invention. The block diagram which shows the structure of the D range expansion prediction interpolation part in Embodiment 1 of this invention. The figure which shows an example of the imaging | photography setting screen displayed on the liquid crystal monitor. The figure which showed the predetermined saturation determination level (TG, TR, TB) with respect to each pixel output of an RGB filter. (A) is a diagram showing a case where the pixel output of the G filter has reached the saturation level A or higher and each pixel output of the surrounding R and B filters is somewhat high, and (b) shows the pixel output of the G filter. The figure which shows the state which correct | amended so that it might expand beyond the saturation level A. (A) is a diagram showing a case where the pixel output of the surrounding R and B filters is extremely small when the pixel output of the G filter reaches or exceeds the saturation level, and (b) is the pixel output of the G filter. The figure which shows the state by which the pixel output of the G filter was reduced-corrected below the saturation level when the pixel output of the surrounding R and B filters was extremely small when the value reached the saturation level. The figure which shows the relationship between color temperature and the pixel output of an RGB filter. (A) is a figure which shows the conversion characteristic which reduces the 14-bit data of the pixel output of the extended G filter in Embodiment 1 of this invention to 12 bits, (b) is other of Embodiment 1 of this invention The figure which shows the conversion characteristic which reduces 14-bit data of the pixel output of the extended G filter in an example to 12 bits. The figure which shows the conversion table which converts 12-bit RGB data into 8-bit RGB data (gamma conversion). (A) is a figure which shows an example of the histogram at the time of performing the expansion process of the dynamic range in Embodiment 1 of this invention, (b) is the histogram at the time of not performing the expansion process of the dynamic range in this embodiment. The figure which shows an example. The figure which shows the pixel arrangement position and processing unit of CCD in which the RGB filter in Embodiment 4 of this invention is arrange | positioned. The figure which shows the pixel arrangement position and processing unit of CCD in which the RGB filter in Embodiment 5 of this invention is arrange | positioned.

Explanation of symbols

1 Digital camera (imaging device)
5 Shooting lens system (optical system)
6 Lens unit 9 LCD monitor 12 Menu button 20 CCD (image sensor)
21 Analog Front End 22 Signal Processor 23 SDRAM
28 Control Unit 34 CCD Interface 35 Memory Controller 36 YUV Conversion Unit 50 D Range Expansion Prediction Interpolation Unit 51 Bit Compression Conversion Unit 60 Pixel Output Detection Unit (Pixel Output Detection Unit)
61 Pixel output correction processing unit (pixel output correction processing means)
62-bit extension processing section

Claims (12)

An image sensor in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are disposed on the front side of each pixel;
Light source color detection means for detecting the light source color of the subject at the time of shooting;
Based on the light source color detected by the light source color detection means, each predetermined saturation level for each output from a pixel in which a filter of a specific color is arranged and a pixel in which a filter of a color other than the specific color is arranged Saturation level setting means for setting
Pixel output detection means for detecting an output from each of the pixels and determining whether the detected pixel output has reached or exceeded each of the predetermined saturation levels;
The pixel output detection means outputs one of the pixel outputs of the pixels from which the filter of the other color around the pixel is arranged with respect to the pixel in which the filter of the specific color is arranged. When it is determined that the pixel has reached the predetermined saturation level or higher, an output from the pixel in which the filter of the specific color is arranged is output from a pixel in which the filter of the other color around the pixel is arranged. An image pickup apparatus comprising: a pixel output correction processing unit that performs correction based on the output of. An image sensor in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are disposed on the front side of each pixel;
Light source color detection means for detecting the light source color of the subject at the time of shooting;
Based on the light source color detected by the light source color detection means, each predetermined saturation level for each output from a pixel in which a filter of a specific color is arranged and a pixel in which a filter of a color other than the specific color is arranged Saturation level setting means for setting
Pixel output detection means for detecting an output from each of the pixels and determining whether the detected pixel output has reached or exceeded each of the predetermined saturation levels;
When the pixel output detection means determines that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher for the pixel, the pixel output detection means has reached the predetermined saturation level or higher. Pixel output correction processing means for correcting the output from the pixel in which the filter of the specific color determined to be is corrected based on the output from the pixel in which the filter of the other color around the pixel is arranged An imaging device comprising:   In the light source color detection means, the filter of the specific color and the filters of the other colors around it are arranged for each region obtained by dividing the screen corresponding to the light receiving surface of all the pixels of the image sensor into a plurality of vertical and horizontal directions. The imaging apparatus according to claim 1, wherein a light source color of a subject at the time of photographing is determined according to a ratio of output values from each pixel.   It further comprises correction coefficient calculation means for calculating a correction coefficient for adjusting the white balance according to the light source at the time of shooting, and the correction coefficient calculation means includes the filter of the specific color and the filters of the other colors around it. The imaging apparatus according to claim 1, wherein the correction coefficient is calculated based on a ratio of output values output from each pixel. Pixel output ratio storage means for storing a plurality of pixel output ratios from each pixel in which the filter of the specific color and the filter of the other color around it are arranged corresponding to a plurality of light source colors having different color temperatures Prepared,
The saturation level setting means sets each predetermined saturation level based on a light source color detected by the light source color detection means and a pixel output ratio stored in the pixel output ratio storage means. Item 4. The imaging device according to any one of Items 1 to 3.   When the pixel output detection unit includes a pixel in which at least two or more filters of the same color are arranged in a processing unit when the output of each pixel is detected, the pixel output detection unit includes the plurality of same color filters. The imaging according to any one of claims 1 to 5, wherein an average value of an output of each pixel in which a filter is arranged is calculated, and the calculated average value is set as a pixel output value in the processing unit. apparatus. An imaging device including an imaging device in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are arranged on the front side of each pixel. In the imaging method of the apparatus,
A light source color detection step for detecting the light source color of the subject at the time of shooting;
Based on the light source color detected by the light source color detection step, each predetermined saturation level for each output from a pixel in which a filter of a specific color is arranged and a pixel in which a filter of a color other than the specific color is arranged A saturation level setting step for setting
A pixel output detection step of detecting an output from each of the pixels and determining whether the detected pixel output has reached or exceeded each of the predetermined saturation levels;
In the pixel output detection step, any one of the outputs from the pixels in which the filter of the other color around the pixel is arranged with respect to the pixel in which the filter of the specific color is arranged, When it is determined that the pixel has reached the predetermined saturation level or higher, an output from the pixel in which the filter of the specific color is arranged is output from a pixel in which the filter of the other color around the pixel is arranged. And a pixel output correction processing step for performing correction based on the output of the imaging method. An imaging device including an imaging device in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are arranged on the front side of each pixel. In the imaging method of the apparatus,
A light source color detection step for detecting the light source color of the subject at the time of shooting;
Based on the light source color detected by the light source color detection step, each predetermined saturation level for each output from a pixel in which a filter of a specific color is arranged and a pixel in which a filter of a color other than the specific color is arranged A saturation level setting step for setting
A pixel output detection step of detecting an output from each of the pixels and determining whether the detected pixel output has reached or exceeded each of the predetermined saturation levels;
When the pixel output detection step determines that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher for the pixel, the pixel has reached the predetermined saturation level or higher. A pixel output correction processing step for correcting the output from the pixel in which the filter of the specific color determined to be is corrected based on the output from the pixel in which the filter of the other color around the pixel is arranged And an imaging method comprising:   In the light source color detection step, the filter of the specific color and the filters of the other colors around it are arranged for each region obtained by dividing the screen corresponding to the light receiving surface of all the pixels of the image sensor into a plurality of vertical and horizontal directions. The imaging method according to claim 7 or 8, wherein a light source color of a subject at the time of photographing is determined according to a ratio of output values from each pixel.   It further includes a correction coefficient calculation step for calculating a correction coefficient for adjusting white balance according to the light source at the time of shooting, and the correction coefficient calculation step includes arranging the filter of the specific color and the other color filters around it. 10. The imaging method according to claim 7, wherein the correction coefficient is calculated based on a ratio of output values output from each pixel. A pixel output ratio storing step of storing a plurality of pixel output ratios from each pixel in which the filter of the specific color and the filter of the other color surrounding it are arranged corresponding to a plurality of light source colors having different color temperatures; Including
The saturation level setting step sets each predetermined saturation level based on a light source color detected in the light source color detection step and a pixel output ratio stored in the pixel output ratio storage step. Item 10. The imaging method according to any one of Items 7 to 9.   When the pixel output detection step includes a pixel in which at least two or more filters of the same color are arranged in a processing unit when the output of each pixel is detected, the pixel output detection step includes the plurality of the same color The imaging according to any one of claims 7 to 11, wherein an average value of outputs of each pixel in which a filter is arranged is calculated, and the calculated average value is used as a pixel output value in the processing unit. Method.

Images