JP2009201094A - Imaging apparatus and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus which expands dynamic range at one shooting without compositing images by performing a plurality of times of shootings while changing exposure, and to provide an imaging method. <P>SOLUTION: The imaging apparatus includes: a saturation pixel correction processing part 61 which, when an output from a pixel where a G filter is placed reaches a saturation level or more, corrects a pixel output of the G filter based on desaturated outputs from pixels where R and B filters are placed, around the pixel where the G filter is placed; and a desaturated pixel correction processing part 62 for correcting output values of pixels where the R and B filters are placed. The desaturated pixel correction processing part 62 corrects each of output values of pixels where R and B filters are placed, in such a way that a ratio between the output value of the pixel where the G filter is placed, corrected by the saturated pixel correction processing part 61 and the output values of the pixels where the R and B filters are placed, around the pixel where the G filter is placed becomes equal to a pixel output ratio being stored in a pixel output ratio storage means in accordance with a color temperature detected by a color temperature detecting means. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method such as a digital still camera and a digital video camera, and more particularly to an imaging apparatus and an imaging method that can expand a dynamic range of a captured image.

  Compared to the dynamic range of images taken with a conventional silver salt camera using a silver salt photographic film, the dynamic range of images taken with a digital still camera or a digital video camera having a solid-state image sensor such as a CCD is extremely narrow. When the dynamic range is narrow, a phenomenon called “blackout” occurs in the dark part of the subject, and conversely, a phenomenon called “overexposure” occurs in the bright part of the subject, and the image quality deteriorates.

Therefore, in order to expand the dynamic range of an image captured by a solid-state imaging device such as a CCD, for example, the same subject is shot multiple times with different exposure amounts to obtain a plurality of images with different exposure amounts. A technique for adding these images to generate a composite image with an expanded dynamic range has been known (see, for example, Patent Document 1).
JP 2000-92378 A

  By the way, in order to expand the dynamic range, in the method of performing photographing a plurality of times while changing the exposure amount as in the above-mentioned Patent Document 1, when photographing a moving object, the subject is doubled. Therefore, the image may not be synthesized correctly.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus and an imaging method capable of expanding the dynamic range by one shooting without changing the exposure amount and performing a plurality of shootings to combine images. .

  In order to achieve the above object, according to the first aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and the front side of each pixel is provided. An image sensor in which color separation filters of a plurality of colors are arranged, and a pixel output detection unit that determines whether or not an output from any one of the pixels has reached a predetermined saturation level or more with respect to an output from each of the pixels. When the pixel output detection unit determines that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the color other than the specific color around the pixel A saturated pixel correction processing unit that corrects a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which the filter is disposed, and the specific color corrected by the saturated pixel correction processing unit A non-saturated pixel correction processing unit that corrects an output value of a pixel in which the other color filter that is not saturated is arranged around the pixel in which the filter is arranged, and a color that detects the color temperature of the subject at the time of shooting Temperature detection means, and pixel output ratio storage means for storing a pixel output ratio for each color of the color separation filter corresponding to a color temperature, and the non-saturated pixel correction processing means is the saturated pixel correction processing means. The color detected by the color temperature detecting means is the ratio between the corrected output value of the pixel in which the filter of the specific color is arranged and the output value of the pixel in which the filter of the other color around the pixel is arranged According to the temperature, each output value of the pixel in which the filter of the other color is arranged is corrected so as to be the same as the pixel output ratio stored in the pixel output ratio storage unit.

  According to a second aspect of the present invention, the color temperature detection unit includes the filter of the specific color and the surroundings in each of the areas obtained by dividing the screen corresponding to the light receiving surfaces of all the pixels of the image sensor into a plurality of vertical and horizontal areas. It is characterized in that the color temperature of the light source of the subject at the time of shooting is determined according to the ratio of the output values from each pixel in which filters of other colors are arranged.

  The invention according to claim 3 further includes correction coefficient calculation means for calculating a correction coefficient for adjusting white balance according to a light source at the time of photographing, wherein the correction coefficient calculation means includes the filter of the specific color and its surroundings. The correction coefficient is calculated based on a ratio of output values output from the pixels in which the other color filters are arranged.

  According to a fourth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are provided on the front side of each pixel. A pixel output detecting unit that determines whether or not an output from any of the pixels has reached a predetermined saturation level or more with respect to the output from each of the pixels; and the pixel output detecting unit. When it is determined that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the pixel in which a filter of a color other than the specific color is arranged around the pixel An image in which a saturated pixel correction processing unit that corrects a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from the image and a filter of the specific color corrected by the saturated pixel correction processing unit is arranged. The pixel output ratio for each color of the color separation filter corresponding to the color temperature, and a non-saturated pixel correction processing means for correcting an output value of a pixel in which the other color filters that are not saturated are arranged. A non-saturated pixel correction processing unit, wherein the non-saturated pixel correction processing unit includes an output value of the pixel in which the filter of the specific color corrected by the saturated pixel correction processing unit is arranged, and surroundings of the pixel The filter of the other color is arranged so that the ratio of the output value of the pixel in which the filter of the other color is arranged is the same as the pixel output ratio stored in the pixel output ratio storage means It is characterized by correcting each output value of the pixel.

  According to a fifth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are provided on the front side of each pixel. A pixel output detecting unit that determines whether or not an output from any of the pixels has reached a predetermined saturation level or more with respect to the output from each of the pixels; and the pixel output detecting unit. When it is determined that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the pixel in which a filter of a color other than the specific color is arranged around the pixel An image in which a saturated pixel correction processing unit that corrects a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from the image and a filter of the specific color corrected by the saturated pixel correction processing unit is arranged. A non-saturated pixel correction processing unit that corrects an output value of a pixel around which a filter of the other color that is not saturated is arranged, and a plurality of screens corresponding to the light receiving surfaces of all the pixels of the image sensor vertically and horizontally A pixel output ratio calculating unit that calculates a ratio of output in each divided area, and the non-saturated pixel correction processing unit includes the filter of the specific color corrected by the saturated pixel correction processing unit The ratio of the output value of the pixel and the output value of the pixel in which the filter of the other color around the pixel is arranged is the same as the pixel output ratio calculated by the pixel output ratio calculating unit. It is characterized in that the output value of a pixel in which filters of other colors are arranged is corrected.

  The inventions according to claims 6 and 12 are characterized in that the processing unit when detecting the output of each pixel is 2 × 2 pixels in the horizontal and vertical directions.

  According to the seventh aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electric signal, and a plurality of color separation filters are provided on the front side of each pixel. In an imaging method of an imaging apparatus including an arranged imaging device, a determination processing step of detecting an output from each of the pixels and determining whether or not the detected pixel output reaches a predetermined saturation level; When it is determined in the determination processing step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel A saturation pixel correction processing step for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed; and the saturation pixel correction processing step. A non-saturated pixel correction processing step for correcting an output value of a pixel in which the filter of the other color that is not saturated is arranged around the pixel in which the filter of the specific color corrected in (i) is arranged; A color temperature detection step for detecting a color temperature of a subject; and a pixel output ratio storage step for storing a pixel output ratio for each color of the color separation filter corresponding to the color temperature, and the unsaturated pixel correction processing A step of calculating a ratio between an output value of the pixel in which the filter of the specific color corrected in the saturation pixel correction processing step is arranged and an output value of the pixel in which the filter of the other color around the pixel is arranged; Depending on the color temperature detected in the color temperature detection step, the filters of the other colors may be the same as the pixel output ratio stored in the pixel output ratio storage step. It is characterized by correcting the respective output values of the location pixel.

  In the invention according to claim 8, in the color temperature detection step, the filter of the specific color and the surroundings of the filter corresponding to the light receiving surface of all the pixels of the image sensor are divided into a plurality of regions divided vertically and horizontally. It is characterized in that the color temperature of the light source of the subject at the time of shooting is determined according to the ratio of the output values from each pixel in which filters of other colors are arranged.

  The invention according to claim 9 further includes a correction coefficient calculation step for calculating a correction coefficient for adjusting white balance according to a light source at the time of shooting, wherein the correction coefficient calculation step includes the filter of the specific color and its surroundings. The correction coefficient is calculated based on a ratio of output values output from the pixels in which the other color filters are arranged.

  According to a tenth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electric signal, and a plurality of color separation filters are provided on the front side of each pixel. In an imaging method of an imaging apparatus including an image sensor arranged, a determination processing step for determining whether an output from any pixel has reached a predetermined saturation level or higher with respect to an output from each pixel; When the determination processing step determines that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the color other than the specific color around the pixel A saturated pixel correction processing step for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which the filter is disposed; and the saturated pixel correction processing step A non-saturated pixel correction processing step for correcting an output value of a pixel in which the filter of the other color that is not saturated is arranged around the pixel in which the corrected filter of the specific color is arranged, and corresponds to the color temperature And a pixel output ratio storing step for storing a pixel output ratio for each color of the color separation filter, wherein the non-saturated pixel correction processing step includes the specific color corrected in the saturated pixel correction processing step. The ratio between the output value of the pixel in which the filter is disposed and the output value of the pixel in which the filter of the other color around the pixel is disposed is the same as the pixel output ratio stored in the pixel output ratio storing step. In other words, each output value of the pixel in which the filter of the other color is arranged is corrected.

  According to an eleventh aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are provided on the front side of each pixel. In an imaging method of an imaging apparatus including an image sensor arranged, a determination processing step for determining whether an output from any pixel has reached a predetermined saturation level or higher with respect to an output from each pixel; When the determination processing step determines that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the color other than the specific color around the pixel A saturated pixel correction processing step for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which the filter is disposed; and the saturated pixel correction processing step A non-saturated pixel correction processing step for correcting an output value of a pixel in which the filter of the other color that is not saturated is arranged around the pixel in which the corrected filter of the specific color is arranged; A pixel output ratio calculating step for calculating a ratio of output in each region obtained by dividing a screen corresponding to the light receiving surface of all pixels into a plurality of vertical and horizontal directions, and the unsaturated pixel correction processing step includes the saturated pixel correction processing step. In the pixel output ratio calculation step, the ratio between the output value of the pixel in which the filter of the specific color corrected in step S3 is arranged and the output value of the pixel in which the filter of the other color around the pixel is arranged is calculated. The output value of the pixel in which the filter of the other color is arranged is corrected so as to be the same as the pixel output ratio.

  According to the present invention, when it is determined that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the filter of another color that is not saturated is arranged around the output. Based on the output from the pixel, the pixel output in the region above the saturation level is corrected to expand the dynamic range, thereby changing the exposure amount and performing a plurality of shootings without synthesizing the image. The dynamic range of the image sensor can be expanded.

  Furthermore, even when the output of a pixel in which a filter of a specific color that has reached a saturation level or higher is corrected, the output balance with a pixel in which a filter of another color that has not reached the saturation level around that pixel is arranged Since it is possible to prevent the image from being lost, it is possible to eliminate a sense of incongruity due to a shift in hue (color).

Hereinafter, the present invention will be described based on the illustrated embodiments.
<Embodiment 1>
FIG. 1A is a front view showing a digital still camera (hereinafter referred to as “digital camera”) as an example of an imaging apparatus according to Embodiment 1 of the present invention, and FIG. 1 (c) is a rear view thereof, and FIG. 2 is a block diagram showing an outline of a system configuration in the digital camera shown in FIGS. 1 (a), (b), and (c).

(Appearance structure of digital camera)
As shown in FIGS. 1A, 1B, and 1C, a release button (shutter button) 2, a power button 3, a shooting / playback switching dial 4 are provided on the upper surface side of the digital camera 1 according to the present embodiment. In the front (front) side of the digital camera 1, a lens barrel unit 6 having a photographing lens system 5, a strobe light emitting unit (flash) 7, and an optical viewfinder 8 are provided.

  On the back side of the digital camera 1, there are a liquid crystal monitor (LCD) 9, an eyepiece 8 a of the optical viewfinder 8, a wide-angle zoom (W) switch 10, a telephoto zoom (T) switch 11, and a menu (MENU) button 12. , A confirmation button (OK button) 13 and the like are provided. Further, a memory card storage unit 15 for storing a memory card 14 (see FIG. 2) for storing captured image data is provided inside the side surface of the digital camera 1.

(Digital camera system configuration)
As shown in FIG. 2, the digital camera 1 includes a CCD 20 serving as a solid-state imaging device on which a subject image incident through a photographing lens system 5 of a lens barrel unit 6 forms an image on a light receiving surface, and electrical signals output from the CCD 20. An analog front-end unit (hereinafter referred to as “AFE unit”) 21 that processes (analog RGB image signal) into a digital signal, a signal processing unit 22 that processes a digital signal output from the AFE unit 21, and temporarily stores data SDRAM 23 for controlling, ROM 24 for storing a control program, a motor driver 25 for driving the lens barrel unit 6 and the like.

  The lens barrel unit 6 includes a photographic lens system 5 having a zoom lens, a focus lens, and the like, an aperture unit 26, and a mechanical shutter unit 27. The drive units of the photographic lens system 5, the aperture unit 26, and the mechanical shutter unit 27 are as follows. It is driven by the motor driver 25. The motor driver 25 is driven and controlled by a drive signal from a control unit (CPU) 28 of the signal processing unit 22.

  As shown in FIG. 3, the CCD 20 has a Bayer array RGB primary color filter (hereinafter referred to as “RGB filter”) arranged on a plurality of pixels 20 a constituting the CCD 20. A signal (analog RGB image signal) is output.

  The AFE unit 21 is sampled by a TG (timing signal generating unit) 30 that drives the CCD 20, a CDS (correlated double sampling unit) 31 that samples an electrical signal (analog RGB image signal) output from the CCD 20, and the CDS 31. An AGC (analog gain control unit) 32 that adjusts the gain of the signal, and an A / D conversion unit 33 that converts the signal gain-adjusted by the AGC 32 into a digital signal (hereinafter referred to as “RAW-RGB data”).

  The signal processing unit 22 outputs a screen horizontal synchronization signal (HD) and a screen vertical synchronization signal (VD) to the TG 30 of the AFE unit 21, and an A / D conversion unit 33 of the AFE unit 21 in accordance with these synchronization signals. CCD interface (hereinafter referred to as “CCD I / F”) 34 that captures RAW-RGB data output from the CPU, a memory controller 35 that controls the SDRAM 23, and a YUV format that can display and record the captured RAW-RGB data. A YUV conversion unit 36 for converting to image data, a resizing processing unit 37 for changing the image size in accordance with the size of image data to be displayed or recorded, a display output control unit 38 for controlling display output of the image data, and an image A data compression unit 39 for recording data by JPEG formation and the like, and writing image data to the memory card 14; A digital camera based on a control program stored in the ROM 24 based on a media interface (hereinafter referred to as “media I / F”) 40 for reading image data written in the memory card 14 and operation input information from the operation unit 41. 1 is provided with a control unit (CPU) 28 for performing overall system control and the like.

  The operation unit 41 includes a release button 2, a power button 3, a photographing / playback switching dial 4, and a wide-angle zoom provided on the external surface of the digital camera 1 (see FIGS. 1A, 1 </ b> B, and 1 </ b> C). The switch 10, the telephoto zoom switch 11, the menu button 12, the confirm button 13, and the like, and a predetermined operation instruction signal is input to the control unit 28 by the photographer's operation.

  The SDRAM 23 stores the RAW-RGB data captured by the CCD I / F 34, and stores the YUV data (YUV format image data) converted by the YUV conversion unit 36. Further, the SDRAM 23 stores the RAW-RGB data. The compressed image data such as JPEG formation is stored.

  Note that YUV of the YUV data is luminance data (Y), color difference (difference (U) between luminance data and blue (B) component data, and difference (V) between luminance data and red (R) component data). It is a format that expresses color with information.

(Digital camera monitoring and still image shooting)
Next, the monitoring operation and still image shooting operation of the digital camera 1 will be described. In the still image shooting mode, the digital camera 1 performs a still image shooting operation while executing a monitoring operation as described below.

  First, when the photographer turns on the power button 3 and sets the photographing / playback switching dial 4 to the photographing mode, the digital camera 1 is activated in the recording mode. When the control unit 28 detects that the power button 3 is turned on and the shooting / playback switching dial 4 is set to the shooting mode, the control unit 28 outputs a control signal to the motor driver 25 to switch the lens barrel unit 6. The camera 20 is moved to a photographing position and the CCD 20, the AFE unit 21, the signal processing unit 22, the SDRAM 23, the ROM 24, the liquid crystal monitor 9 and the like are activated.

  Then, by directing the photographic lens system 5 of the lens barrel unit 6 toward the subject, a subject image incident through the photographic lens system 5 is formed on the light receiving surface of each pixel of the CCD 20. Then, an electrical signal (analog RGB image signal) corresponding to the subject image output from the CCD 20 is input to the A / D conversion unit 33 via the CDS 31 and the AGC 32, and 12 bits (bit) by the A / D conversion unit 33. To RAW-RGB data.

  This RAW-RGB data is taken into the CCD I / F 34 of the signal processing unit 22 and stored in the SDRAM 23 via the memory controller 35. The RAW-RGB data read from the SDRAM 23 is converted into YUV data (YUV signal) in a format that can be displayed by the YUV converter 36, and then the YUV data is stored in the SDRAM 23 via the memory controller 35. The

  The YUV data read from the SDRAM 23 via the memory controller 35 is sent to the liquid crystal monitor (LCD) 9 via the display output control unit 38, and a captured image is displayed. At the time of monitoring in which a photographed image is displayed on the liquid crystal monitor (LCD) 9 described above, one frame is read out in a time of 1/30 second by thinning out the number of pixels by the CCD I / F 34.

  In this monitoring operation, the photographed image is only displayed on the liquid crystal monitor (LCD) 9 functioning as an electronic viewfinder, and the release button 2 is not yet pressed (including half-pressed).

  The photographer can confirm the photographed image by displaying the photographed image on the liquid crystal monitor (LCD) 9. In addition, it can output as a TV video signal from the display output control part 38, and can display a picked-up image on external TV (television) via a video cable.

  The CCD I / F 34 of the signal processing unit 22 calculates an AF (automatic focus) evaluation value, an AE (automatic exposure) evaluation value, and an AWB (auto white balance) evaluation value from the captured RAW-RGB data.

  The AF evaluation value is calculated by, for example, the output integrated value of the high frequency component extraction filter or the integrated value of the luminance difference between adjacent pixels. When in the in-focus state, the edge portion of the subject is clear, so the high frequency component is the highest. By utilizing this, at the time of AF operation (at the time of focus detection operation), an AF evaluation value at each focus lens position in the photographing lens system 5 is acquired, and the point where the maximum is obtained is used as the focus detection position for AF. The action is executed.

  The AE evaluation value and the AWB evaluation value are calculated from the integrated values of the RGB values in the RAW-RGB data. For example, the screen corresponding to the light receiving surfaces of all the pixels of the CCD 20 is equally divided into 1024 areas (horizontal 32 divisions and vertical 32 divisions), and the RGB integration of each area is calculated.

  Then, the control unit 28 reads the calculated RGB integrated value, and in the AE process, calculates the luminance of each area of the screen and determines an appropriate exposure amount from the luminance distribution. Based on the determined exposure amount, exposure conditions (the number of electronic shutters of the CCD 20, the aperture value of the aperture unit 26, etc.) are set. In the AWB process, the subject color and the light source color are determined from the RGB distribution, and the AWB control value according to the color of the light source is determined. By this AWB process, white balance is adjusted when the YUV conversion unit 36 performs conversion to YUV data. The AE process and AWB process described above are continuously performed during the monitoring.

  When the release button 2 is pressed (half-pressed to full-press) and the still image shooting operation is started during the monitoring operation described above, the AF operation and the still image recording process that are the focus position detection operations are performed. .

  That is, when the release button 2 is pressed (half-pressed to full-pressed), the focus lens of the photographing lens system 5 is moved by a drive command from the control unit 28 to the motor driver 25, and is referred to as so-called hill-climbing AF, for example. The contrast evaluation AF operation is executed.

  When the AF (focusing) target range is the entire region from infinity to close, the focus lens of the photographing lens system 5 moves to each focus position from close to infinity, or from infinity to close, and CCDI / The control unit 28 reads out the AF evaluation value at each focus position calculated in F34. Then, the focus lens is moved to the in-focus position with the point where the AF evaluation value at each focus position is maximized as the in-focus position, and in-focus.

  Then, the AE process described above is performed, and when the exposure is completed, the mechanical shutter unit 27 is closed by a drive command from the control unit 28 to the motor driver 25, and an analog RGB image signal for a still image is output from the CCD 20. As in the monitoring, the A / D conversion unit 33 of the AFE unit 21 converts the data into RAW-RGB data.

  The RAW-RGB data is taken into the CCD I / F 34 of the signal processing unit 22, converted into YUV data by a YUV conversion unit 36 described later, and stored in the SDRAM 23 via the memory controller 35. The YUV data is read from the SDRAM 23, converted into a size corresponding to the number of recorded pixels by the resizing processing unit 37, and compressed into image data such as JPEG format by the data compression unit 39.

  The compressed image data such as JPEG format is written back to the SDRAM 23, read out from the SDRAM 23 through the memory controller 35, and stored in the memory card 14 through the media I / F 40.

(Principle of dynamic range expansion in the present invention)
A Bayer-arranged RGB filter (see FIG. 3) is arranged on each pixel constituting the CCD 20 of the digital camera 1, but a normal RGB filter for light having a wide wavelength band such as sunlight. Have different luminance sensitivities for each color.

  For example, as shown in FIG. 4, an RGB filter (a, b, c in FIG. 4) has a sensitivity of the G (green) filter that is about twice that of an R (red) filter and a B (blue) filter. In the case of the CCD 20 having the same, when the same amount of light having a wide wavelength band such as sunlight is incident on the RGB filter, the G filter (shaded portion in FIG. 4c) is applied to each pixel output of the R and B filters. The pixel output first reaches the saturation level A. In FIG. 4, f is the pixel sensitivity characteristic of the G filter, g is the pixel sensitivity characteristic of the R and B filters, and the pixel sensitivity characteristic of the G filter is about twice the pixel sensitivity characteristics of the R and B filters. It has the sensitivity of.

  By the way, in a conventional digital camera having a solid-state imaging device such as a CCD in which an RGB filter is arranged, a saturation level corresponding to the pixel output of a high-sensitivity G filter, such as the RGB filters of a, b, and c in FIG. The dynamic range is set according to A. For this reason, even when the pixel output of the G filter reaches the saturation level A, the pixel output of the R and B filters is about ½ of the saturation level A.

  On the other hand, in the present invention, even if the pixel output of the G filter exceeds the saturation level A, as in the RGB filters of d and e in FIG. From within the pixel output levels of the R and B filters, the pixel sensitivity characteristics of the R and B filters (g in FIG. 4) and the pixel sensitivity characteristics of the G filter (f in FIG. 4) Based on this, the pixel output level of the G filter is corrected so as to perform prediction interpolation (dotted line portion), and the dynamic range is expanded by this prediction interpolation (correction).

  As described above, in the present embodiment, the pixel sensitivity characteristic of the G filter has a sensitivity about twice that of each pixel sensitivity characteristic of the R and B filters with respect to light having a wide wavelength band such as sunlight. ing. Therefore, the maximum value of the dynamic range expansion degree in the present embodiment is about twice that of a normal photographing operation in which the dynamic range expansion processing operation is not performed.

  In this embodiment, the pixel sensitivity characteristic of the G filter has a sensitivity that is about twice that of the pixel sensitivity characteristics of the R and B filters, and based on this, the maximum value of the degree of expansion of the dynamic range is doubled. However, by changing the pixel sensitivity characteristics of the R, G, and B filters, the maximum value of the degree of expansion of the dynamic range can be set to a predetermined value that is two times or more, or a predetermined value that is two times or less.

(Dynamic range expansion processing by the YUV converter 36)
The YUV conversion unit 36 of the digital camera 1 according to the present embodiment has a dynamic range expansion processing function for expanding the dynamic range described above.

  As shown in FIG. 5, the YUV conversion unit 36 includes a dynamic range expansion prediction interpolation unit (hereinafter referred to as “D range expansion prediction interpolation unit”) 50, a bit compression conversion unit 51, a white balance control unit 52, and synchronization described later. A unit 53, a tone conversion unit 54, an RGB-YUV conversion unit 55, an image size converter unit 56, a luminance histogram generation unit 57, and an edge enhancement unit 58.

  As illustrated in FIG. 6, the D range expansion prediction interpolation unit 50 includes a luminance level determination unit 60, a saturated pixel correction processing unit 61, a non-saturated pixel correction processing unit 62, and a bit extension processing unit 63.

  The luminance level determination unit 60 detects the pixel output of each pixel provided with the RGB filter from the input RAW-RGB data, and also outputs the pixel output of the pixel provided with the highest sensitivity G filter (hereinafter referred to as “G filter”). It is determined whether or not the “pixel output of” has reached a predetermined saturation level or higher.

  When the luminance level determination unit 60 determines that the pixel output of the G filter has reached a predetermined saturation level or higher, the saturated pixel correction processing unit 61 provides R and B filters around the pixel where the G filter is disposed. Correction (prediction interpolation processing) of the pixel output of the G filter that has reached a predetermined saturation level or more is performed by the pixel output of the selected pixel (hereinafter referred to as “R, B filter pixel output”) (details will be described later). Based on the pixel output value of the G filter corrected by the saturated pixel correction processing unit 61, the non-saturated pixel correction processing unit 62 corrects the pixel output of the R and B filters around the pixel where the G filter is arranged. (Details will be described later).

  When the luminance level determination unit 60 determines that the pixel output of the G filter has not reached the saturation level, the bit extension processing unit 63 performs the pixel output of the G filter and the pixel output of the R and B filters. Only bit expansion is performed from 12 bits to 14 bits without correcting the output level.

  The dynamic range expansion processing operation in this embodiment will be described below.

  For example, when there is an extremely bright part of the background of the subject to be photographed and the photographer wants to expand the dynamic range, the photographer presses the menu button 12 (see FIG. 1C), for example, A shooting setting screen as shown in FIG. 7 is displayed on a liquid crystal monitor (LCD) 9.

  Then, as shown in FIG. 7, with respect to the item “dynamic range doubling” selected by pressing the menu button 12, the confirmation button (see FIG. 1C) 13 is pressed to set “dynamic range doubling”. decide. As a result, the dynamic range expansion processing operation is turned ON, and a control signal for doubling the dynamic range expansion rate is output from the control unit 28 to the luminance level determination unit 60.

  Note that the dynamic range expansion ratio in the present embodiment can be arbitrarily set in a range of 1.1 to 2 times, with the above-mentioned double being the maximum value.

  When a control signal for doubling the dynamic range expansion rate is output from the control unit 28 to the luminance level determination unit 60 and the dynamic range expansion processing operation is executed, the luminance level of the D range expansion prediction interpolation unit 50 is The determination unit 60 determines whether or not the pixel output of the G filter has reached a predetermined saturation level or more based on the input RAW-RGB data.

  In the case where this determination process is performed, in the present embodiment, as shown in FIG. 3, 2 × 2 pixels in the thick frame A (2 pixels for the G filter) for each pixel of the CCD 20 in which the RGB filter is installed. , R and B filters are each one pixel) as a processing unit (minimum unit).

  When at least one pixel output of two G filter pixels in the processing unit (thick frame A) reaches a predetermined saturation level or higher, the sensitivity of the G filter is R, as described above. Since it is about twice the sensitivity of the B filter, the pixel output value (G) of the G filter is calculated from the following equation (1).

  G = {(R + B) / 2} × 2 Formula (1)

  The saturated pixel correction processing unit 61 calculates the average value of the pixel outputs of the R and B filters and doubles the average value of the pixel outputs of the R filter and the B filter as shown in the equation (1). The calculated pixel output values of the G filter are replaced with the pixel output values of the two G filters in the processing unit (2 × 2 pixels). Since the pixel output value of the G filter exceeds 12 bits, it is replaced here with 14-bit data. Therefore, since the maximum value of the pixel output of the R and B filters is 4095 (12 bits), the maximum value of the pixel output of the G filter is 8190, so that it can be handled as 14-bit data.

  In the equation (1), the calculation of the pixel output value of the G filter is performed with 2 × 2 pixels. However, the present invention is not limited to this, and the R and B filters around the pixel of the correction target G filter are not limited thereto. It is also possible to use n (two or more) pixels, calculate the average value in the same manner as in equation (1), and double the average value.

  By the way, when the pixel output of the G filter that has reached the predetermined saturation level or higher is corrected by the saturated pixel correction processing unit 61 as described above, if only the pixel output of the saturated G filter is corrected, In some cases, the balance with the pixel outputs of the R and B filters that are not saturated is lost, and the hue (hue) is shifted. Therefore, the non-saturated pixel correction processing unit 62 corrects the R and B filter pixel outputs as follows according to the G filter pixel output corrected by the saturated pixel correction processing unit 61.

  FIG. 8 is a characteristic diagram showing an example of a ratio of each pixel output of the RGB filter with respect to a difference in color temperature of the light source (2000K (Kelvin) or less to 9000K (Kelvin) or more). As apparent from FIG. 8, the pixel output of the G filter hardly changes even when the color temperature changes, but the pixel output of the R filter increases as the color temperature decreases (red light source), and the pixel of the B filter The output increases as the color temperature increases (blue light source).

  The ratio relationship between the color temperature and each pixel output value of the RGB filter as shown in FIG. 8 depends on the CCD in which the RGB filter is arranged. Therefore, when the CCD 20 with the RGB filter is incorporated in the digital camera 1, the ratio relationship of the pixel output values of the RGB filter with respect to different color temperatures as shown in FIG. The table shown (hereinafter referred to as “RGB output ratio table”) is stored in the ROM 24.

  Then, the control unit 28 determines the color temperature of the subject at the time of shooting from the RGB distribution that is the AWB evaluation value acquired from the CCD I / F 34, and then reads the RGB output ratio table from the ROM 24, and the unsaturated pixel correction processing unit To 62. Thus, in this embodiment, the pixel output ratio storage means in the “claims” corresponds to the ROM 24, and the color temperature detection means corresponds to the control unit 28.

  Then, the non-saturated pixel correction processing unit 62 corrects the pixel output value of the G filter corrected by the saturated pixel correction processing unit 61 with respect to the color temperature of the subject at the time of shooting, and the R and B filter pixels around the pixels of the G filter. The pixel output values of the R and B filters are corrected so that the ratio with the output value is the same as the pixel output ratio in the RGB output ratio table.

  When the luminance level determination unit 60 determines that the pixel output of the G filter has not reached the saturation level, the bit expansion processing unit 63 outputs the pixel output of the G filter and the pixel output of the R and B filters. Thus, only bit expansion is performed from 12 bits to 14 bits without correcting the output level.

  By the way, before the luminance level determination unit 60 determines whether or not the pixel output of the G filter has reached the saturation level or higher, the correction of the defective pixel needs to be completed. That is, if there is a defective pixel in the pixel provided with the G filter and there is a pixel that always outputs a saturated value, the pixel provided with the G filter in the same processing unit is replaced with a large value. A defective pixel will be generated.

  Further, when a pixel in which the R and B filters are arranged has a defective pixel, the conversion of the pixel provided with the G filter according to the equation (1) becomes an incorrect value. For this reason, in this embodiment, the CCD I / F 34 includes a defective pixel output removal processing unit (not shown) that removes the output of the defective pixel.

  Then, pixel output value data of the RGB filter corrected by the saturated pixel correction processing unit 61 and the non-saturated pixel correction processing unit 62 of the D range expansion prediction interpolation unit 50 or only bit-extended by the bit expansion processing unit 63, respectively. The pixel output value data of the RGB filter is output to the bit compression conversion unit 51.

  The bit compression conversion unit 51 expands to 14 bits by, for example, conversion characteristics as shown in FIG. 9A (four-segment broken line approximation characteristics in which three nodes are designated and approximated by a straight line). Of the RGB filter pixel outputs, the G filter pixel output is compressed to 12 bits. In FIG. 9A, a is a 12-bit range, and b is a simple linear conversion characteristic (one-dot chain line portion) that halves the data of the maximum value 8190.

  In the conversion characteristic shown in FIG. 9A, since the maximum value of the pixel output of the G filter is 8190, compression is performed so that 8190 becomes 4095. Then, the pixel output values of the R and B filters are also compressed in accordance with the compression magnification of the pixel output of the G filter.

  As described above, in this embodiment, as an example of the case where the pixel output of the G filter whose maximum value is expanded to 8190 is compressed to 4095, three nodes as shown by the solid line in FIG. A conversion characteristic with In the present embodiment, the following two effects that cannot be obtained with a linear conversion characteristic without a simple node (b in FIG. 9A) are obtained.

  As a first effect, a larger number of bits can be assigned to data with high data reliability. That is, when predictive interpolation processing is performed on the pixel output of the G filter that has reached the saturation level or higher, as described above, the range in which the value of the pixel output of the G filter becomes equal to or higher than the specified value near the saturation level is corrected (predicted). Interpolation) is performed, and correction is not performed within the range below this specified value. Therefore, the accuracy of data differs between the range where correction is performed and the range where correction is not performed.

  That is, for example, when correcting (predicting interpolation) the pixel output value of the G filter that is saturated by the equation (1), the luminance level of the subject is accurately reproduced within the correction range depending on the color of the main subject. It may not be possible. On the other hand, since the range where correction is not performed is data obtained by A / D conversion of actual data (analog RGB image signal) output from the CCD 20 in which the RGB filter is arranged, the reliability of this data is high. It will be a thing.

  That is, in the conversion characteristic (part indicated by the solid line) in this embodiment shown in FIG. 9A, for example, when the input 14-bit data is 1024, the output 12-bit data is 1024, and the original data is It is used as it is. On the other hand, for example, when the input 14-bit data is 3072, the output 12-bit data is 2560, and in this range, the bit allocation is smaller than the bit allocation before correction, so that some bit error occurs.

  In this way, by adopting a conversion characteristic (part indicated by a solid line) having three nodes as in this embodiment, instead of a characteristic for performing linear conversion without a simple node (part indicated by a one-dot chain line). Since the bit allocation can be gradually reduced, a larger number of bits can be allocated to data with high data reliability.

  As a second effect, gradations at low and medium luminance can be accurately stored. That is, when bit compression is performed with a simple linear conversion characteristic (b in FIG. 9A), the bit allocation becomes ¼ in a range where predictive interpolation on the low luminance side is not performed. For this reason, the image has no tone. On the other hand, when bit compression is performed with the conversion characteristic (solid line portion) in the present embodiment as shown in FIG. 9A, the data corresponding to the pixel output at the low luminance level below the saturation level is obtained. Thus, by using a compression rate that gives substantially the same value before bit compression by the bit compression conversion unit 51 and after bit compression, it is possible to maintain good gradation at a low luminance level.

  In this embodiment, when the 14-bit data of the pixel output of the expanded G filter is reduced to 12 bits, three nodes are designated as shown in FIG. Although the bit compression is performed with the broken line approximation characteristic (conversion characteristic) of four sections, the number of sections is not particularly limited. For example, it may be a polygonal line approximate characteristic of two sections designating one node, but the above-mentioned two effects are reduced by changing the bit allocation in the vicinity of the node. Therefore, a broken line approximation characteristic (conversion characteristic) having three or more sections is preferable.

  Further, the conversion characteristic for reducing the 14-bit data of the pixel output of the expanded G filter to 12 bits may be a conversion characteristic by a curve not having a plurality of nodes as shown in FIG. That is, in contrast to the conversion characteristic having four sections in FIG. 9A, the conversion characteristic based on this curve is obtained by setting the number of sections to 8192. In FIG. 9B, a is a 12-bit range.

  Further, by providing a lookup table having numeric data after reduction conversion to 12 bits for 0 to 8192 of input 14-bit data, the pixel output of the expanded G filter with the conversion characteristics by this curve is provided. The 14-bit data can be successfully compressed to 12 bits.

  The pixel output data of the RGB filter that has been compression-converted from 14 bits to 12 bits by the bit compression conversion unit 51 is input to the white balance control unit 52.

  The white balance control unit 52 multiplies the input pixel output value data of the RGB filter by a correction coefficient for adjusting white balance. This correction coefficient is calculated by the control unit 28 based on the AWB evaluation value generated by the CCD I / F 34. The correction coefficient is a coefficient that makes the light source color white. Thus, in the present embodiment, the correction coefficient calculation means in “Claims” corresponds to the white balance control unit 52.

  Specifically, based on the ratio of the RGB filter pixel output values to different color temperatures as shown in FIG. 8, the R filter pixel output values include G / R (G filter pixel output as a correction coefficient). Value / R filter pixel output value), and the B filter pixel output value is multiplied by G / B (G filter pixel output value / B filter pixel output value) as a correction coefficient. Match the level of the pixel output value of the RGB filter.

  By the way, with the correction processing in the saturated pixel correction processing unit 61 and the unsaturated pixel correction processing unit 62 of the D range expansion prediction interpolation unit 50 described above, the ratio of the pixel output values of the RGB filter is determined by the RGB filter determined from the light source color. This is the ratio of the pixel output value.

  Therefore, the output obtained by multiplying the output from the D-range expansion prediction interpolation unit 50 by the correction coefficient by the white balance control unit 52 has substantially the same RGB filter level and a value close to an achromatic color. Note that an image region in which all the pixel output values of the RGB filter are saturated is overexposed and whitened. In addition, an image region where only the pixel output value of the high-sensitivity G filter is saturated is also likely to have a high luminance and a large amount of incident light, so even if the image region to be corrected for pixel output is an achromatic color There is less discomfort.

  Then, the pixel output value data (12 bits) of the RGB filter with the white balance adjusted from the white balance control unit 52 is input to the synchronization unit 53. The synchronization unit 53 performs an interpolation calculation process on the RAW data having only one color data in one pixel of the Bayer array, and generates all RGB data for one pixel.

  All the RGB data (12 bits) generated by the synchronization unit 53 is input to the tone conversion unit 54. The tone conversion unit 54 performs γ conversion that converts 12-bit RGB data into 8-bit RGB data using a conversion table as shown in FIG. 10 to generate an 8-bit RGB value, and performs RGB-YUV conversion. To the unit 55.

  The RGB-YUV conversion unit 55 converts input RGB data (8 bits) into YUV data by matrix calculation and outputs the YUV data to the image size converter unit 56. The image size converter unit 56 reduces or enlarges the input YUV data (8 bits) to a desired image size, and outputs it to the luminance histogram generation unit 57 and the edge enhancement unit 58.

  The luminance histogram generation unit 57 generates a luminance histogram based on the input YUV data. The edge enhancement unit 58 performs processing such as edge enhancement according to the image on the input YUV data, and stores the processed data in the SDRAM 23 via the memory controller 35.

  As described above, in the present embodiment, even when the pixel output of the G filter with high sensitivity in the processing unit exceeds the saturation level, the pixel output of the R and B filters that have not reached the surrounding saturation level. As shown in FIG. 4, the pixel output of the G filter (d and e in FIG. 4) is corrected (predicted interpolation) by correcting the pixel output of the saturated G filter (predictive interpolation processing) based on Based on the expanded region (the one-dot chain line portion of the pixel output of the G filter of d and e in FIG. 4), the dynamic range can be doubled by one shooting.

  Therefore, even when there is a high-luminance portion in the background or the like in the captured image, it is possible to prevent the occurrence of overexposure and obtain good gradation.

  Furthermore, in this embodiment, as described above, the pixel output value of the G filter corrected by the saturated pixel correction processing unit 61 with respect to the color temperature of the subject at the time of shooting by the unsaturated pixel correction processing unit 62 and the G filter The pixel output values of the R and B filters are corrected so that the ratio of the pixel output values of the R and B filters around the pixels is the same as the pixel output ratio of the RGB output ratio table.

  Thereby, even when the pixel output of the G filter that has reached the saturation level or higher is corrected, the output balance with the pixel outputs of the R and B filters that have not reached the saturation level around the pixel of the G filter is prevented from being lost. Therefore, a sense of incongruity due to a shift in hue (hue) can be eliminated.

  FIG. 11A shows an example of a histogram generated by the luminance histogram generation unit 57 when the dynamic range expansion process in the present embodiment described above is performed when the pixel output of the G filter exceeds the saturation level. It is. As is apparent from this histogram, by performing the dynamic range expansion processing, whiteout in the maximum luminance portion (255) hardly occurs, and reproduction is performed with good gradation.

  On the other hand, FIG. 11B is generated by the luminance histogram generation unit 57 when the dynamic range expansion process in the present embodiment is not performed when the pixel output of the G filter exceeds the saturation level. It is an example of the done histogram. As is apparent from this histogram, it can be seen that, by not performing the dynamic range expansion process, the maximum luminance portion (255) has a frequency and whiteout occurs.

  In the description of the first embodiment and FIG. 4, the saturation level A in FIG. 4 that is a predetermined saturation level determination value and the corrected 12-bit maximum value 4095 are described so as to coincide with each other. However, the present invention is not limited to this. For example, in a CCD having an RGB filter whose output linearity (linearity) is not good in a high-luminance part near where the output is completely saturated, for example, 4032 which is a value smaller than 4095 which is completely saturated is set to a predetermined saturation level. With respect to the value (saturation level A in FIG. 4), the pixel output exceeding that value may be subject to correction processing (predictive interpolation processing).

  Also, depending on the system configuration of the digital camera, even a high-luminance subject may not reach 4095, which is the maximum value of 12 bits. In that case as well, the predetermined saturation level may be set to a value lower than 4095.

  As described above, even when the predetermined saturation level is less than 4095, the output of the bit compression conversion unit 51 can be set to 4095 by switching the conversion curve of FIG. 10 according to the characteristics, and the subsequent processing can be performed. The dynamic range can be expanded without change.

<Embodiment 2>
In the first embodiment, the ratio between the pixel output value of the corrected G filter and the pixel output values of the R and B filters around the pixel of the G filter in accordance with the color temperature of the subject at the time of shooting is the RGB output. The pixel output values of the R and B filters are corrected so as to be the same as the pixel output ratio of the ratio table. However, as a simple configuration, according to the sensitivity difference of the pixel output of the RGB filter under a specific light source The ratio may be measured in advance and stored in the ROM 24 (see FIG. 2), and the pixel output values of the R and B filters may be corrected so as to be the same as the ratio measured in advance. Other configurations are the same as those of the digital camera of the first embodiment.

  For example, FIGS. 12A and 12B show examples of pixel outputs of two RGB filters having different sensitivity characteristics under a specific light source.

  In the digital camera (imaging apparatus) having the pixel output characteristics of the RGB filter of FIG. 12A, the pixel output ratio of the RGB filter is stored in advance. In the digital camera (imaging device) having the pixel output characteristics of the RGB filter shown in FIG. 12B, the pixel output ratio of the RGB filter is stored in advance.

  For example, in a digital camera (imaging device) in which the pixel output ratio of the RGB filter as shown in FIG. 12 (a) is stored in advance, R, so as to be the same as the ratio measured in advance. The pixel output value of the B filter is corrected.

  Since the normal white balance control range is about 2000 to 8000K (Kelvin), the pixel output ratio of the RGB filter measured under a light source having a color temperature of about 4000 to 5000K (Kelvin) near the center is stored in advance. It is desirable to keep it.

  As described above, in this embodiment, a ratio according to the sensitivity difference of the pixel output of the RGB filter under a specific light source is measured and stored in advance, and R is set to be the same as the ratio measured in advance. Since the configuration is such that the pixel output value of the B filter is corrected, the configuration can be made simpler than in the first embodiment.

<Embodiment 3>
FIG. 13 is a block diagram illustrating a configuration of the D range expansion prediction interpolation unit 50a in the present embodiment, and further includes a pixel output ratio setting unit 64. Other configurations are the same as those of the first embodiment, and a duplicate description is omitted.

  As described in the first embodiment, the AWB evaluation value generated by the CCD I / F 34 is obtained by equally dividing the screen corresponding to the light receiving surface of all the pixels of the CCD 20 into 1024 areas (horizontal 32 divisions and vertical 32 divisions). Calculated from RGB integration in the area. And the control part 28 (refer FIG. 2) calculates the ratio of the pixel output of the RGB filter in each said area from the produced | generated AWB evaluation value. The calculated pixel output ratio of the RGB filter in each area is set in the pixel output ratio setting unit 64 of the D range expansion prediction interpolation unit 50a. Thus, in the present embodiment, the pixel output ratio calculation means in “Claims” corresponds to the control unit 28.

  When the dynamic range expansion processing operation is executed as in the first embodiment, the luminance level determination unit 60 of the D range expansion prediction interpolation unit 50a outputs the pixel output of the G filter based on the input RAW-RGB data. It is determined whether or not has reached a predetermined saturation level or higher. If it is determined in this determination that the pixel output of the G filter has reached a predetermined saturation level or higher, the saturation pixel correction processing unit 61 performs a predetermined output based on the pixel output of the R and B filters around the pixel of the G filter. Correction (predictive interpolation processing) of the pixel output of the G filter that has reached the saturation level or higher is performed.

  At this time, pixel position information of the G filter that has reached the saturation level or higher is output from the luminance level determination unit 60 to the pixel output ratio setting unit 64. The pixel output ratio setting unit 64 determines in which area of the 1024 areas (horizontal 32 divisions and vertical 32 divisions) the G filter pixels that have reached the saturation level or more are included. The ratio of pixel output values is output to the unsaturated pixel correction processing unit 62.

  In the non-saturated pixel correction processing unit 62, the ratio between the pixel output value of the G filter corrected by the saturated pixel correction processing unit 61 and the pixel output value of the surrounding R and B filters is set in the pixel output ratio setting unit 64. The pixel output values of the R and B filters are corrected so that the pixel output ratio is the same. The following processing is the same as in the first embodiment.

  As described above, in this embodiment, the ratio between the pixel output value of the G filter corrected by the saturated pixel correction processing unit 61 and the pixel output value of the R and B filters around the pixel where the G filter is arranged is the pixel. By correcting the pixel output values of the R and B filters so as to be the same as the pixel output ratio set in the output ratio setting unit 64, the ratio of the corrected pixel output values of the RGB filter is changed to its peripheral region. It can be set to a value close to the hue (hue). Therefore, even when the dynamic range is expanded, it is possible to eliminate a sense of incongruity due to a hue (hue) shift.

  In the present embodiment, the screen corresponding to the light receiving surface of all the pixels of the CCD 20 is equally divided into 1024 areas (horizontal 32 divisions and vertical 32 divisions), and the pixels of the G filter that reach the saturation level or more are in the 1024 areas. Although it is determined which area is included (horizontal 32 divisions, vertical 32 divisions), the screen corresponding to the light receiving surface of all the pixels of the CCD 20 is divided into finer areas such as one pixel unit. It is better not to.

  This is because the pixel output of the RGB filter has a slight sensitivity difference for each pixel, and there are some pixels that output a specific output regardless of incident light, such as defective pixels. For this reason, for example, if the integration target at the time of generating the AWB evaluation value is 10 pixels or more, a CCD provided with an RGB filter having a variation that can absorb the variation in characteristics of each pixel by averaging is used. It is desirable to make it.

  In each of the embodiments described above, the luminance level determination unit 60 determines whether or not the pixel output of the G filter with the highest sensitivity has reached the saturation level or higher. The determination may be made other than the pixel output. For example, it may be determined whether or not to perform the above-described correction (predictive interpolation process) by determining whether or not the pixel output of the R filter has reached a predetermined level or more.

  In the RGB filter having the Bayer array as shown in FIG. 3, if the pixel output of the R filter reaches, for example, 70% of the saturation level, it can be determined that the pixel output of the G filter has reached the saturation level or more. it can. Similarly, it may be determined whether the correction (predictive interpolation process) is performed by determining whether the pixel output of the R filter has reached a predetermined level or more.

  Further, since the ratio of the pixel output of the RGB filter to the color temperature changes as shown in FIG. 8, it is determined whether or not the average value of the pixel output of the R filter and the pixel output of the B filter has reached a predetermined level. By determining, it may be determined whether to execute the above-described correction (predictive interpolation process).

  In each of the embodiments described above, the RGB three primary color filters are arranged as the color separation filters. However, the present invention is similarly applied to a configuration in which the complementary color filters are arranged as the color separation filters. Can do.

  In each of the above-described embodiments, the description has been made on the imaging device such as a digital camera having a photographing lens system and a solid-state imaging device such as a CCD. However, the present invention can be similarly applied to an image processing device. It is. For example, the present invention is similarly applied to an image processing apparatus in which analog data from a single-plate color image sensor is converted into digital signals and stored, and RAW data is input to output RGB data or YCbCr data. It is possible.

(A) is a front view which shows the digital camera as an example of the imaging device which concerns on Embodiment 1 of this invention, (b) is the top view, (c) is the back view. 1 is a block diagram showing an outline of a system configuration in a digital camera as an example of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram illustrating a pixel arrangement position and a processing unit of a CCD in which an RGB filter according to Embodiment 1 of the present invention is arranged. The figure for demonstrating the principle of the dynamic range expansion in Embodiment 1 of this invention. The block diagram which shows the structure of the YUV conversion part in Embodiment 1 of this invention. The block diagram which shows the structure of the D range expansion prediction interpolation part in Embodiment 1 of this invention. The figure which shows an example of the imaging | photography setting screen displayed on the liquid crystal monitor. The figure which shows the relationship between color temperature and the pixel output of an RGB filter. (A) is a figure which shows the conversion characteristic which compresses the 14-bit data of the pixel output of the extended G filter in Embodiment 1 of this invention to 12 bits, (b) is other of Embodiment 1 of this invention The figure which shows the conversion characteristic which compresses 14 bits data of the pixel output of the extended G filter in an example to 12 bits. The figure which shows the conversion table which converts 12-bit RGB data into 8-bit RGB data (gamma conversion). (A) is a figure which shows the histogram at the time of performing the expansion process of the dynamic range in Embodiment 1 of this invention, (b) shows the histogram at the time of not performing the expansion process of the dynamic range in this embodiment. Figure. (A), (b) is a figure which shows an example of the pixel output of the R, G, B filter from which a sensitivity characteristic each differs under a specific light source. The block diagram which shows the structure of the D range expansion prediction interpolation part in Embodiment 3 of this invention.

Explanation of symbols

1 Digital camera (imaging device)
5 Shooting lens system (optical system)
6 Lens unit 9 LCD monitor 12 Menu button 20 CCD (image sensor)
21 Analog Front End 22 Signal Processor 23 SDRAM
28 Control Unit 34 CCD Interface 35 Memory Controller 36 YUV Conversion Unit 50, 50a D Range Expansion Prediction Interpolation Unit 51 Bit Compression Conversion Unit 60 Luminance Level Determination Unit (Pixel Output Detection Unit)
61 Saturated pixel correction processing unit (saturated pixel correction processing means)
62 Unsaturated pixel correction processing unit (unsaturated pixel correction processing means)
63-bit extension processing unit 64 pixel output ratio setting unit

Claims (12)

An image sensor in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are disposed on the front side of each pixel;
Pixel output detection means for determining whether the output from any pixel reaches or exceeds a predetermined saturation level with respect to the output from each pixel;
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel are detected. Saturated pixel correction processing means for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed;
Non-saturated pixel correction processing for correcting an output value of a pixel in which the other color filter that is not saturated is arranged around the pixel in which the filter of the specific color corrected by the saturated pixel correction processing unit is arranged Means,
Color temperature detecting means for detecting the color temperature of the subject at the time of shooting;
Pixel output ratio storage means for storing a pixel output ratio for each color of the color separation filter corresponding to a color temperature,
The non-saturated pixel correction processing means outputs an output value of a pixel in which the filter of the specific color corrected by the saturated pixel correction processing means and a pixel in which the filter of the other color around the pixel is arranged. The filters of the other colors are arranged so that the ratio to the output value is the same as the pixel output ratio stored in the pixel output ratio storage unit according to the color temperature detected by the color temperature detection unit An image pickup apparatus that corrects each output value of a pixel that has been corrected.   In the color temperature detection means, the filter of the specific color and the filters of the other colors around it are arranged for each region obtained by dividing the screen corresponding to the light receiving surface of all the pixels of the image sensor into a plurality of vertical and horizontal directions. The imaging apparatus according to claim 1, wherein the color temperature of the light source of the subject at the time of shooting is determined according to a ratio of output values from each pixel.   It further comprises correction coefficient calculation means for calculating a correction coefficient for adjusting the white balance according to the light source at the time of shooting, and the correction coefficient calculation means includes the filter of the specific color and the filters of the other colors around it. The imaging apparatus according to claim 1, wherein the correction coefficient is calculated based on a ratio of output values output from the respective pixels. An image sensor in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are disposed on the front side of each pixel;
Pixel output detection means for determining whether the output from any pixel reaches or exceeds a predetermined saturation level with respect to the output from each pixel;
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel are detected. Saturated pixel correction processing means for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed;
Non-saturated pixel correction processing for correcting an output value of a pixel in which the other color filter that is not saturated is arranged around the pixel in which the filter of the specific color corrected by the saturated pixel correction processing unit is arranged Means,
Pixel output ratio storage means for storing a pixel output ratio for each color of the color separation filter corresponding to a color temperature,
The non-saturated pixel correction processing means outputs an output value of a pixel in which the filter of the specific color corrected by the saturated pixel correction processing means and a pixel in which the filter of the other color around the pixel is arranged. Correcting each output value of the pixel in which the filter of the other color is arranged so that the ratio with the output value is the same as the pixel output ratio stored in the pixel output ratio storage unit An imaging device. An image sensor in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are disposed on the front side of each pixel;
Pixel output detection means for determining whether the output from any pixel reaches or exceeds a predetermined saturation level with respect to the output from each pixel;
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel are detected. Saturated pixel correction processing means for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed;
Non-saturated pixel correction processing for correcting an output value of a pixel in which the other color filter that is not saturated is arranged around the pixel in which the filter of the specific color corrected by the saturated pixel correction processing unit is arranged Means,
A pixel output ratio calculating means for calculating a ratio of output in each region obtained by dividing a screen corresponding to the light receiving surface of all pixels of the image sensor into a plurality of vertical and horizontal directions,
The non-saturated pixel correction processing means outputs an output value of a pixel in which the filter of the specific color corrected by the saturated pixel correction processing means and a pixel in which the filter of the other color around the pixel is arranged. Imaging that corrects an output value of a pixel in which the filter of the other color is arranged so that a ratio to an output value is the same as a pixel output ratio calculated by the pixel output ratio calculating unit apparatus.   The processing unit when the pixel output detection unit detects the output of the pixel is a size of 2 × 2 pixels in the horizontal and vertical directions, according to claim 1. Imaging device. An imaging device including an imaging device in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are arranged on the front side of each pixel. In the imaging method of the apparatus,
A determination processing step for detecting an output from each of the pixels and determining whether the detected pixel output has reached a predetermined saturation level or higher;
When it is determined in the determination processing step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel A saturated pixel correction processing step for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed;
Non-saturated pixel correction processing for correcting an output value of a pixel in which the filter of the other color that is not saturated is arranged around the pixel in which the filter of the specific color corrected in the saturated pixel correction processing step is arranged Steps,
A color temperature detection step for detecting the color temperature of the subject at the time of shooting;
A pixel output ratio storing step for storing a pixel output ratio for each color of the color separation filter corresponding to a color temperature,
The non-saturated pixel correction processing step includes an output value of a pixel in which the filter of the specific color corrected in the saturated pixel correction processing step and a pixel in which the filter of the other color around the pixel is disposed. The filters of the other colors are arranged so that the ratio with the output value is the same as the pixel output ratio stored in the pixel output ratio storage step according to the color temperature detected in the color temperature detection step. An imaging method characterized by correcting each output value of a pixel obtained.   In the color temperature detection step, the filter of the specific color and the filters of the other colors around it are arranged for each region obtained by dividing the screen corresponding to the light receiving surface of all the pixels of the image sensor into a plurality of vertical and horizontal directions. 8. The imaging method according to claim 7, wherein the color temperature of the light source of the subject at the time of shooting is determined according to the ratio of output values from each pixel. A correction coefficient calculating step for calculating a correction coefficient for adjusting the white balance according to the light source at the time of shooting;
The correction coefficient calculating step calculates the correction coefficient based on a ratio of output values output from each pixel in which the filter of the specific color and the other color filters around it are arranged. The imaging method according to claim 7 or 8. An imaging device including an imaging device in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are arranged on the front side of each pixel. In the imaging method of the apparatus,
A determination processing step for determining whether the output from any of the pixels has reached a predetermined saturation level or higher with respect to the output from each of the pixels;
When it is determined in the determination processing step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel A saturated pixel correction processing step for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed;
Non-saturated pixel correction processing for correcting an output value of a pixel in which the filter of the other color that is not saturated is arranged around the pixel in which the filter of the specific color corrected in the saturated pixel correction processing step is arranged Steps,
A pixel output ratio storing step for storing a pixel output ratio for each color of the color separation filter corresponding to a color temperature, and
The non-saturated pixel correction processing step includes an output value of a pixel in which the filter of the specific color corrected in the saturated pixel correction processing step and a pixel in which the filter of the other color around the pixel is disposed. Each output value of the pixel in which the filter of the other color is arranged is corrected so that the ratio with the output value is the same as the pixel output ratio stored in the pixel output ratio storing step. Imaging method. An imaging device including an imaging device in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are arranged on the front side of each pixel. In the imaging method of the apparatus,
A determination processing step for determining whether the output from any of the pixels has reached a predetermined saturation level or higher with respect to the output from each of the pixels;
When it is determined in the determination processing step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, other colors other than the specific color around the pixel A saturated pixel correction processing step for correcting a pixel output determined to be equal to or higher than the predetermined saturation level based on an output from a pixel in which a filter is disposed;
Non-saturated pixel correction processing for correcting an output value of a pixel in which the filter of the other color that is not saturated is arranged around the pixel in which the filter of the specific color corrected in the saturated pixel correction processing step is arranged Steps,
A pixel output ratio calculating step for calculating a ratio of output in each region obtained by dividing a screen corresponding to the light receiving surface of all pixels of the image sensor into a plurality of vertical and horizontal directions,
The non-saturated pixel correction processing step includes an output value of a pixel in which the filter of the specific color corrected in the saturated pixel correction processing step and a pixel in which the filter of the other color around the pixel is disposed. Imaging that corrects an output value of a pixel in which the filter of the other color is arranged so that a ratio to an output value is the same as a pixel output ratio calculated in the pixel output ratio calculating step Method.   The processing unit when the pixel output detection step detects the output of each pixel is a size of 2 × 2 pixels in the horizontal and vertical directions. Imaging method.

Images