JP2010068331A - Imaging device and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device and imaging method that magnify a dynamic range by one time shooting without combining images by changing light exposure and carrying out the shooting at a plurality of times. <P>SOLUTION: The imaging method generates third YUV data by combining first YUV data, luminance data, and color difference data. The first YUV data are generated from a pixel output obtained by reading out RAW-RGB data from SDRAM 23 and correcting them with a D-range magnification correction portion in a YUV conversion portion 36. The luminance data are generated by introducing a second YUV data generated from the pixel output which has reached the saturation region without carrying out correction processing at the D-range magnification correction portion into a YUV combining portion 37, and by taking them out of the first YUV data with the YUV combining portion 37. The color difference data are taken out of the second YUV data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method such as a digital still camera and a digital video camera, and more particularly to an imaging apparatus and an imaging method that can expand a dynamic range of a captured image.

  Compared to the dynamic range of images taken with a conventional silver salt camera using a silver salt photographic film, the dynamic range of images taken with a digital still camera or a digital video camera having a solid-state image sensor such as a CCD is extremely narrow. When the dynamic range is narrow, a phenomenon called “blackout” occurs in the dark part of the subject, and conversely, a phenomenon called “overexposure” occurs in the bright part of the subject, and the image quality deteriorates.

Therefore, in order to expand the dynamic range of an image captured by a solid-state imaging device such as a CCD, for example, the same subject is shot multiple times with different exposure amounts to obtain a plurality of images with different exposure amounts. A technique for adding these images to generate a composite image with an expanded dynamic range has been known (see, for example, Patent Document 1).
JP 2000-92378 A

  By the way, in order to expand the dynamic range, in the method of performing photographing a plurality of times while changing the exposure amount as in the above-mentioned Patent Document 1, when photographing a moving object, the subject is doubled. Therefore, the image may not be synthesized correctly.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus and an imaging method capable of expanding the dynamic range by one shooting without changing the exposure amount and performing a plurality of shootings to combine images. .

  In order to achieve the above object, an imaging apparatus according to claim 1, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a front side of each pixel In addition, an image sensor having a plurality of color separation filters disposed therein and an output from each of the pixels is detected, and an output from a pixel in which a specific color filter is disposed based on the detected pixel output, Peripheral pixel saturation determination means for determining whether or not a saturation region where the input / output characteristics are not linear is reached, and the output from the pixel in which the filter of the specific color is arranged by the peripheral pixel saturation determination means When it is determined that the filter has reached the output, the output from the pixel having the filter of the specific color is compensated based on the output from the pixel having the filter other than the specific color around it. The saturated pixel output correction processing means, the first YUV data generated from the pixel output corrected by the saturated pixel output correction processing means, and the saturation without performing the correction processing by the saturated pixel output correction processing means. YUV composition that takes in the second YUV data generated from the pixel output reaching the area, and generates third YUV data that combines the luminance data extracted from the first YUV data and the color difference data extracted from the second YUV data And means.

  The imaging apparatus according to claim 2, wherein a processing unit when the output of each pixel is detected by the peripheral pixel saturation determination unit is a size of 2 × 2 pixels in the horizontal and vertical directions. .

  The image pickup apparatus according to claim 3, wherein an operation selection is performed for selecting and executing the operation to be corrected when the saturation region is reached with respect to an output of the pixel in which the filter of the specific color is arranged. It is characterized by having means.

  The imaging device according to claim 4, wherein the saturation region of the pixel output that is output from the saturation pixel output correction processing unit when an output of a pixel in which the filter of the specific color has reached the saturation region Compression of pixel output data once expanded from the first number of bits that can express the pixel value to the second number of bits that can be expressed beyond the saturation level of the pixel output to the first number of bits again Conversion means, wherein the bit compression conversion means compresses the data corresponding to the pixel output in the saturation region or lower than the compression rate for the data corresponding to the pixel output in the saturation region or less. It is characterized by.

  The imaging apparatus according to claim 5, for data corresponding to a pixel output at a low luminance level below the saturation region, is substantially the same value before bit compression by the bit compression conversion unit and after bit compression. It is characterized by using such a compression rate.

  The imaging apparatus according to claim 6, wherein when there is a defective pixel in the processing unit, a pixel in which a filter of the same color as the defective pixel around the defective pixel is arranged instead of the defective pixel. It is characterized by use.

  The imaging method according to claim 7, wherein a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are provided on the front side of each pixel. In the imaging method of the imaging apparatus including the imaging device in which is arranged, an output from each of the pixels is detected, and an output from a pixel in which a specific color filter around the pixel is arranged is detected based on the detected pixel output. The peripheral pixel saturation determination processing step for determining whether or not the saturation region where the input / output characteristics become non-linear is reached, and the output from the pixel in which the filter of the specific color is arranged by the peripheral pixel saturation determination processing step. When it is determined that the saturation region has been reached, the filter of the specific color is arranged based on the output from the pixel where the filter other than the specific color is arranged around it. A saturated pixel output correction processing step for correcting an output from the pixel, first YUV data generated from the pixel output corrected by the saturated pixel output correction processing step, and the correction processing in the saturated pixel output correction processing step. The second YUV data generated from the pixel output reaching the saturation region without performing the processing, and the third YUV obtained by combining the luminance data extracted from the first YUV data and the color difference data extracted from the second YUV data And a YUV composition processing step for generating data.

  The imaging method according to claim 8, wherein a processing unit when detecting an output of each pixel in the peripheral pixel saturation determination processing step is a size of 2 × 2 pixels in the horizontal and vertical directions. Yes.

  The imaging method according to claim 9, wherein an operation to be corrected when the saturation region is reached is selected by an operation selection unit with respect to an output from a pixel in which the filter of the specific color is arranged. It is characterized by being executed.

  The imaging method according to claim 10, wherein the saturation region of the pixel output output from the saturation pixel output correction processing unit when an output of the pixel in which the filter of the specific color is arranged reaches the saturation region. Compression of pixel output data once expanded from the first number of bits that can represent the pixel to the second number of bits that can be expressed beyond the saturation region of the pixel output to the first number of bits again A conversion processing step, wherein the bit compression conversion processing step reduces the compression rate for the data corresponding to the pixel output in the saturation region or less than the compression rate for the data corresponding to the pixel output in the saturation region or more. It is characterized by compression.

  12. The imaging method according to claim 11, wherein the data corresponding to the pixel output at a low luminance level below the saturation region has substantially the same value before bit compression and after bit compression in the bit compression conversion processing step. It compresses using the compression rate which becomes.

  The imaging method according to claim 12, wherein when there is a defective pixel in the processing unit, a pixel in which a filter of the same color as the defective pixel around the defective pixel is arranged instead of the defective pixel. It is characterized by use.

  According to the first and seventh aspects of the present invention, when it is determined that the output from the pixel in which the filter of the specific color is arranged has reached the saturation region, the surrounding filters of other colors that are not saturated Based on the output from the pixel where the pixel is arranged, the pixel output in the saturation region or higher is corrected to expand the dynamic range, so that the exposure amount is changed and the image is synthesized multiple times without changing the exposure amount. The dynamic range can be expanded by shooting.

  Further, according to the first and seventh aspects of the present invention, the first YUV data generated from the corrected pixel output and the second YUV generated from the pixel output reaching the saturation region without performing the correction process. The data is taken in and the luminance data (Y data) taken out from the first YUV data and the color difference data (UV data) taken out from the second YUV data are combined to generate third YUV data. As a result, the dynamic range is expanded only for the luminance data of the first YUV data generated from the corrected pixel output, and the original color difference data is used for the color, thereby preventing a hue shift when the dynamic range is expanded. Accurate color reproduction.

  According to the third and ninth aspects of the present invention, the operation selecting means selects an operation for correcting the output of the pixel in which the filter of the specific color is disposed when the saturation region is reached. This can be done at the discretion of the photographer.

  According to the fourth and tenth aspects of the present invention, compression is performed by reducing the compression rate for data corresponding to the pixel output in the saturation region or lower than the compression rate for the data corresponding to the pixel output in the saturation region or higher. Therefore, it is possible to maintain good gradation in the saturation region and below.

  According to the fifth and eleventh aspects of the present invention, for data corresponding to a pixel output at a low luminance level below the saturation region, compression is performed so that the values are substantially the same before bit compression and after bit compression. By using the rate, the gradation at a low luminance level can be maintained satisfactorily.

  According to the invention described in claims 6 and 12, when there is a defective pixel in the processing unit, a pixel in which a filter of the same color as the defective pixel around the defective pixel is arranged instead of the defective pixel. By using this, it is not necessary to use defective pixels when determining whether or not the output of the pixel in which the filter of the specific color is arranged reaches the saturation region. The output can be corrected with high accuracy when it reaches the saturation region.

Hereinafter, the present invention will be described based on the illustrated embodiments.
<Embodiment 1>

  FIG. 1A is a front view showing a digital still camera (hereinafter referred to as “digital camera”) as an example of an imaging apparatus according to Embodiment 1 of the present invention, and FIG. 1 (c) is a rear view thereof, and FIG. 2 is a block diagram showing an outline of a system configuration in the digital camera shown in FIGS. 1 (a), (b), and (c).

(Appearance structure of digital camera)
As shown in FIGS. 1A, 1B, and 1C, a release button (shutter button) 2, a power button 3, a shooting / playback switching dial 4 are provided on the upper surface side of the digital camera 1 according to the present embodiment. In the front (front) side of the digital camera 1, a lens barrel unit 6 having a photographing lens system 5, a strobe light emitting unit (flash) 7, and an optical viewfinder 8 are provided.

  On the back side of the digital camera 1, there are a liquid crystal monitor (LCD) 9, an eyepiece 8 a of the optical viewfinder 8, a wide-angle zoom (W) switch 10, a telephoto zoom (T) switch 11, and a menu (MENU) button 12. , A confirmation button (OK button) 13 and the like are provided. Further, a memory card storage unit 15 for storing a memory card 14 (see FIG. 2) for storing captured image data is provided inside the side surface of the digital camera 1.

(Digital camera system configuration)
As shown in FIG. 2, the digital camera 1 includes a CCD 20 serving as a solid-state imaging device on which a subject image incident through a photographing lens system 5 of a lens barrel unit 6 forms an image on a light receiving surface, and electrical signals output from the CCD 20. An analog front-end unit (hereinafter referred to as “AFE unit”) 21 that processes (analog RGB image signal) into a digital signal, a signal processing unit 22 that processes a digital signal output from the AFE unit 21, and temporarily stores data SDRAM 23 for controlling, ROM 24 for storing a control program, a motor driver 25 for driving the lens barrel unit 6 and the like.

  The lens barrel unit 6 includes a photographic lens system 5 having a zoom lens, a focus lens, and the like, an aperture unit 26, and a mechanical shutter unit 27. The drive units of the photographic lens system 5, the aperture unit 26, and the mechanical shutter unit 27 are as follows. It is driven by the motor driver 25. The motor driver 25 is driven and controlled by a drive signal from a control unit (CPU) 28 of the signal processing unit 22.

  In the CCD 20, RGB primary color filters (see FIG. 7: hereinafter referred to as “RGB filters”) are arranged on a plurality of pixels constituting the CCD 20, and electrical signals (analog RGB image signals) corresponding to the RGB three primary colors are output. The

  The AFE unit 21 is sampled by a TG (timing signal generating unit) 30 that drives the CCD 20, a CDS (correlated double sampling unit) 31 that samples an electrical signal (analog RGB image signal) output from the CCD 20, and the CDS 31. An AGC (analog gain controller) 32 that adjusts the gain of the image signal, and an A / D converter 33 that converts the image signal gain-adjusted by the AGC 32 into a digital signal (hereinafter referred to as “RAW-RGB data”) are provided. Yes.

  The signal processing unit 22 outputs a screen horizontal synchronization signal (HD) and a screen vertical synchronization signal (VD) to the TG 30 of the AFE unit 21, and an A / D conversion unit 33 of the AFE unit 21 in accordance with these synchronization signals. Camera interface (hereinafter referred to as “camera I / F”) 34 that captures RAW-RGB data output from the camera, a memory controller 35 that controls the SDRAM 23, and a YUV format that can display and record the captured RAW-RGB data A YUV conversion unit 36 for converting the image data into the image data, a YUV composition unit 37 to be described later, a resize processing unit 38 for changing the image size in accordance with the size of the image data to be displayed and recorded, and a display output of the image data. A display output control unit 39 for recording, a data compression unit 40 for recording image data by JPEG formation, and the like. Based on the operation input information from the media interface (hereinafter referred to as “media I / F”) 41 for reading the image data written to the re-card 14 or the image data written to the memory card 14 and the control stored in the ROM 24. A control unit (CPU) 28 that performs system control and the like of the entire digital camera 1 based on a program is provided.

  The operation unit 42 includes a release button 2, a power button 3, a shooting / playback switching dial 4, and a wide-angle zoom provided on the external surface of the digital camera 1 (see FIGS. 1A, 1B, and 1C). The switch 10, the telephoto zoom switch 11, the menu button 12, the confirm button 13, and the like, and a predetermined operation instruction signal is input to the control unit 28 by the photographer's operation.

  In the SDRAM 23, the RAW-RGB data captured by the camera I / F 34 is stored, and YUV data (YUV format image data) converted by the YUV conversion unit 36 and synthesized by the YUV synthesis unit 37 are stored. YUV data (details will be described later) and the like are stored, and further, image data and the like such as JPEG formation compressed by the data compression unit 40 are stored.

  The YUV of the YUV data is color information based on luminance data (Y), color difference (difference (U) between luminance data and blue (B) data, and difference (V) between luminance data and red (R)). Is a format that expresses

  As shown in FIG. 3, the YUV conversion unit 36 includes a dynamic range expansion correction unit (hereinafter referred to as “D range expansion correction unit”) 50, a bit compression conversion unit 51, a white balance control unit 52, and a synchronization unit 53, which will be described later. A tone curve conversion unit 54, an RGB-YUV conversion unit 55, an image size converter unit 56, and an edge enhancement unit 57.

(Digital camera monitoring operation, normal still image shooting operation)
Next, the monitoring operation of the digital camera 1 and the normal still image shooting operation will be described. The digital camera 1 performs a monitoring operation and a still image shooting operation described below in the still image shooting mode.

  First, when the photographer turns on the power button 3 and sets the photographing / playback switching dial 4 to the photographing mode, the digital camera 1 is activated in the recording mode. When the control unit 28 detects that the power button 3 is turned on and the shooting / playback switching dial 4 is set to the shooting mode, the control unit 28 outputs a control signal to the motor driver 25 to switch the lens barrel unit 6. The camera 20 is moved to a photographing position and the CCD 20, the AFE unit 21, the signal processing unit 22, the SDRAM 23, the ROM 24, the liquid crystal monitor 9 and the like are activated.

  Then, by directing the photographic lens system 5 of the lens barrel unit 6 toward the subject, a subject image incident through the photographic lens system 5 is formed on the light receiving surface of each pixel of the CCD 20. Then, an electrical signal (analog RGB image signal) corresponding to the subject image output from the CCD 20 is input to the A / D conversion unit 33 via the CDS 31 and the AGC 32, and 12 bits (bit) by the A / D conversion unit 33. To RAW-RGB data.

  This RAW-RGB data is taken into the camera I / F 34 of the signal processing unit 22 and stored in the SDRAM 23 via the memory controller 35. The RAW-RGB data read from the SDRAM 23 is converted into YUV data (YUV signal) in a format that can be displayed by the YUV converter 36, and then the YUV data is stored in the SDRAM 23 via the memory controller 35. The

  The YUV data read from the SDRAM 23 via the memory controller 35 is sent to the liquid crystal monitor (LCD) 9 via the display output control unit 39, and a photographed image (moving image) is displayed. At the time of monitoring in which a photographed image is displayed on the liquid crystal monitor (LCD) 9 described above, one frame is read out in a time of 1/30 seconds by thinning out the number of pixels by the camera I / F 34.

  In this monitoring operation, the photographed image is only displayed on the liquid crystal monitor (LCD) 9 functioning as an electronic viewfinder, and the release button 2 is not yet pressed (including half-pressed).

  The photographer can confirm the photographed image by displaying the photographed image on the liquid crystal monitor (LCD) 9. In addition, it can output as a TV video signal from the display output control part 39, and a picked-up image (moving image) can also be displayed on external TV (television) via a video cable.

  Then, the camera I / F 34 of the signal processing unit 22 calculates an AF (autofocus) evaluation value, an AE (automatic exposure) evaluation value, and an AWB (auto white balance) evaluation value from the captured RAW-RGB data.

  The AF evaluation value is calculated by, for example, the output integrated value of the high frequency component extraction filter or the integrated value of the luminance difference between adjacent pixels. When in the in-focus state, the edge portion of the subject is clear, so the high frequency component is the highest. By utilizing this, at the time of AF operation (at the time of focus detection operation), an AF evaluation value at each focus lens position in the photographing lens system 5 is acquired, and the point where the maximum is obtained is used as the focus detection position for AF. The action is executed.

  The AE evaluation value and the AWB evaluation value are calculated from the integrated values of the RGB values in the RAW-RGB data. For example, the screen corresponding to the light receiving surfaces of all the pixels of the CCD 20 is equally divided into 256 areas (16 horizontal divisions and 16 vertical divisions), and the RGB integration of each area is calculated.

  Then, the control unit 28 reads the calculated RGB integrated value, and in the AE process, calculates the luminance of each area of the screen and determines an appropriate exposure amount from the luminance distribution. Based on the determined exposure amount, exposure conditions (the number of electronic shutters of the CCD 20, the aperture value of the aperture unit 26, etc.) are set. In the AWB process, an AWB control value that matches the color of the light source of the subject is determined from the RGB distribution. By this AWB process, white balance is adjusted when the YUV conversion unit 36 performs conversion to YUV data. The AE process and AWB process described above are continuously performed during the monitoring.

  When the still image shooting operation in which the release button 2 is pressed (half-pressed to full-press) is started during the monitoring operation described above, the AF operation and the still image recording process that are the focus position detection operations are performed. .

  That is, when the release button 2 is pressed (half-pressed to full-pressed), the focus lens of the photographing lens system 5 is moved by a drive command from the control unit 28 to the motor driver 25, and is referred to as so-called hill-climbing AF, for example. The contrast evaluation AF operation is executed.

  When the AF (focusing) target range is the entire region from infinity to close, the focus lens of the photographing lens system 5 moves to each focus position from close to infinity or from infinity to close to the camera I. The control unit 28 reads out the AF evaluation value at each focus position calculated in / F34. Then, the focus lens is moved to the in-focus position with the point where the AF evaluation value at each focus position is maximized as the in-focus position, and in-focus (AF processing).

  Then, the AE process described above is performed, and when the exposure is completed, the mechanical shutter unit 27 is closed by a drive command from the control unit 28 to the motor driver 25, and an analog RGB image signal for a still image is output from the CCD 20. As in the monitoring, the A / D conversion unit 33 of the AFE unit 21 converts the data into RAW-RGB data.

  The RAW-RGB data is captured by the camera I / F 34 of the signal processing unit 22, converted to YUV data by the YUV conversion unit 36, and stored in the SDRAM 23 via the memory controller 35. The YUV data is read from the SDRAM 23, converted into a size corresponding to the number of recorded pixels by the resizing processing unit 38, and compressed to image data in the JPEG format or the like by the data compression unit 40. The compressed image data such as JPEG format is written back to the SDRAM 23, read out from the SDRAM 23 via the memory controller 35, and recorded on the memory card 14 via the media I / F 41.

  The YUV conversion unit 36 of the digital camera 1 according to the present embodiment has a dynamic range expansion processing function for expanding the dynamic range.

(Principle of dynamic range expansion in the present invention)
A Bayer array RGB filter (see FIG. 6) is generally disposed on each pixel constituting the CCD 20 of the digital camera 1, but normal RGB is not used for light having a wide wavelength band such as sunlight. Filters have different sensitivities to luminance for each color.

  For example, as shown in FIG. 4, an RGB filter (a, b, c in FIG. 4) has a sensitivity of the G (green) filter that is about twice that of an R (red) filter and a B (blue) filter. In the case of the CCD 20 having the same, when the same amount of light having a wide wavelength band such as sunlight is incident on the RGB filter, the G filter (shaded portion in FIG. 4c) is applied to each pixel output of the R and B filters. The pixel output first reaches the saturation level A. In FIG. 3, f is the pixel sensitivity characteristic of the G filter, g is the pixel sensitivity characteristic of the R and B filters, and the pixel sensitivity characteristic of the G filter is about twice the pixel sensitivity characteristics of the R and B filters. It has the sensitivity of.

  By the way, in a digital camera having a solid-state image sensor such as a CCD having a conventional RGB filter, a saturation level A corresponding to the pixel output of a highly sensitive G filter, such as the RGB filters of a, b, and c in FIG. In addition, the dynamic range is set. That is, even when the pixel output of the G filter reaches the saturation level A, the pixel output of the R and B filters is about ½ of the saturation level A.

  On the other hand, in the present invention, even if the pixel output of the G filter exceeds the saturation level A, as in the RGB filters of d and e in FIG. From within the pixel output levels of the R and B filters, the pixel sensitivity characteristics of the R and B filters (g in FIG. 4) and the pixel sensitivity characteristics of the G filter (f in FIG. 4) Based on this, the pixel output level of the G filter is corrected (one-dot chain line portion), and the dynamic range is expanded by this correction.

  As described above, in the present embodiment, the pixel sensitivity characteristic of the G filter has a sensitivity about twice that of each pixel sensitivity characteristic of the R and B filters with respect to light having a wide wavelength band such as sunlight. ing. Therefore, the maximum value of the dynamic range expansion degree in this embodiment is about twice that in the normal still image shooting mode in which the dynamic range expansion processing operation is not performed.

  In this embodiment, the pixel sensitivity characteristic of the G filter has a sensitivity that is about twice that of the pixel sensitivity characteristics of the R and B filters, and based on this, the maximum value of the degree of expansion of the dynamic range is doubled. However, by changing the pixel sensitivity characteristics of the RGB filter, the maximum value of the degree of expansion of the dynamic range can be set to a predetermined value of 2 times or more, or a predetermined value of 2 times or less.

(Configuration of D range expansion correction unit 50)
The D range expansion correction unit 50 of the YUV conversion unit 36 shown in FIG. 3 has a dynamic range expansion processing function for expanding the dynamic range described above. As shown in FIG. 5, the D range expansion correction unit 50 includes a luminance level determination unit 60, a pixel output correction processing unit 61, and a bit expansion processing unit 62.

  The luminance level determination unit 60 detects the pixel output of each pixel provided with the RGB filter from the input RAW-RGB data, and outputs the pixel output of the pixel provided with the G filter (hereinafter, “G filter pixel output”). Each of the pixel outputs of the pixels provided with the R and B filters in the vicinity thereof (hereinafter referred to as “R and B filter pixel outputs”) is set in advance. Judgment is made based on whether or not the G pixel saturation determination threshold is reached. Then, the luminance level determination unit 60 calculates a correction coefficient to be described later for correcting the pixel output of the G filter when it is determined that the pixel output of the G filter having the highest sensitivity has reached the saturation region. To do.

  The pixel output correction processing unit 61 performs correction processing of the pixel output of the G filter by multiplying the pixel output of the G filter by the correction coefficient calculated by the luminance level determination unit 60.

  When the luminance level determination unit 60 determines that the pixel output of the G filter has reached the saturation region as described above, the bit extension processing unit 62 outputs the output level to the pixel output of the R and B filters. Only bit expansion is performed from 12 bits to 14 bits without correction. The bit expansion processing unit 62 corrects the output level for each pixel output of the RGB filter when the luminance level determination unit 60 determines that each pixel output of the RGB filter has not reached the saturation level. Only the bit expansion is performed from 12 bits to 14 bits without performing it.

  In calculating the correction coefficient of the pixel output of the G filter by the luminance level determination unit 60, in this embodiment, for each pixel of the CCD 20 having the RGB filter, as shown in FIG. X2 pixels (two G filter pixels, one R and B filter pixels) are set as a processing unit (minimum unit). The correction coefficient (K) of the pixel output of the G filter and the pixel output (Ge) of the G filter after the correction process (predictive interpolation process) are calculated from the following expressions (1) and (2), respectively.

K = {l * f (Ro) + m * f (Go) + n * f (Bo)} / 3 Formula (1)
Ge = K × Go (2)
Here, l, m, and n are coefficients set from the sensitivity ratios of the RGB filters, and Go is the pixel output of the G filter before correction processing. Further, f (Ro), f (Go), and f (Bo) are coefficients set by the following equation 1 (Equation (3) to Equation (5)).

  Where Ro is the pixel output of the R filter, TR is the G pixel saturation determination level corresponding to the pixel output of the R filter, Go is the pixel output of the G filter before correction processing, TG is the saturation determination level of the pixel output of the G filter, Bo is a pixel output of the B filter, and TB is a G pixel saturation determination level corresponding to the pixel output of the B filter.

  The G pixel saturation determination levels TR, TG, and TB in the equations (3) to (5) correspond to, for example, predetermined saturation determination levels for the respective pixel outputs of the RGB filter shown in FIG. In FIG. 7, A (TG) is the saturation level (saturation determination level) of the pixel output of the G filter, TR is the saturation determination level of the pixel output of the R filter, and TB is the G pixel saturation corresponding to the pixel output of the B filter. It is a judgment level.

  In this embodiment, since the sensitivity of the pixel provided with the G filter is twice the sensitivity of the pixel provided with the R filter and the B filter as described above, the pixel of the G filter first reaches the saturation level A. Reach. Therefore, the output value that has reached the saturation level A is set as the saturation determination level TG of the pixel output of the G filter, and the G pixel saturation determination levels TR and TB corresponding to the pixel outputs of the R filter and the B filter are 1/2 of TG. Was set to the value of Note that TR, TG, and TB depend on the sensitivity ratio of a solid-state imaging device (such as a CCD) having an RGB filter used in an imaging device (such as a digital camera) and are limited to the ratios shown in FIG. is not.

  As in the above formulas (1) to (5), the ratios of the pixel outputs of the R, G, and B filters and the G pixel saturation determination levels TR, TG, and TB are calculated, and the RGB values are calculated as these calculated values. A coefficient for correcting the pixel output of the G filter is calculated by multiplying the coefficient set from the sensitivity ratio of the filter and averaging, and the corrected correction coefficient (K) is multiplied by the pixel output of the G filter to obtain a corrected result. The pixel output of the G filter is calculated. Then, the value of the pixel output (Ge) of the G filter after correction calculated from the equation (2) by the correction coefficient (K) is the pixel output of the two G filters in the processing unit (see FIG. 6). Replaced as a value.

  Since the pixel output value of the G filter exceeds 12 bits, it is replaced with 14-bit data here. Therefore, since the maximum value of each pixel output of the R and B filters is 4095 (12 bits), the maximum value of the pixel output of the G filter is 8190 and can be handled as 14-bit data.

  By the way, when the luminance level determination unit 60 determines whether or not the pixel output of the G filter has reached the saturation region, it is necessary not to use the defective pixel for this determination. In other words, if the pixel provided with the G filter has a defective pixel and the value of the saturation region is always output, the output of the pixel provided with the G filter in the same processing unit (see FIG. 6) is replaced with a large value. Therefore, a new defective pixel is generated.

  Further, when there is a defective pixel in at least one of the pixels provided with the R and B filters in the processing unit (see FIG. 6), the pixel of the G filter calculated by the above equation (1) The output correction coefficient (K) becomes an incorrect value, and the value of the pixel output (Ge) of the G filter after the correction process calculated by the equation (2) also becomes an incorrect value.

  For this reason, in the present embodiment, when the luminance level determination unit 60 detects the pixel output of each pixel provided with the RGB filter and determines that the value of the saturation region is always output, the processing unit (FIG. 6)), it is determined that there is a defective pixel. When it is determined that there is a defective pixel in this way, a pixel that is not a defective pixel in the processing unit and a pixel that is provided with a filter of the same color as the defective pixel around the processing unit (not a defective pixel) Instead, the pixel output of each pixel is detected as described above.

  Accordingly, when the luminance level determination unit 60 determines whether or not the pixel output of the G filter has reached the saturation region, it is possible to prevent the use of a defective pixel, so that the pixel in which the G filter is arranged When it is determined that the output reaches the saturation region, high-accuracy correction processing can be performed using the equations (1) to (5).

  Next, the still image shooting operation in the above-described dynamic range expansion mode in the present embodiment will be described with reference to the flowcharts shown in FIGS. First, the first process in which the RAW-RGB data (A in FIG. 3) is input to the D range expansion correction unit 50 of the YUV conversion unit 36 will be described with reference to the flowchart shown in FIG. Next, referring to the flowchart shown in FIG. 9, the RAW-RGB data (B in FIG. 3) is input to the white balance control unit 52 without being input to the D range expansion correction unit 50 side of the YUV conversion unit 36. The input second process will be described.

  RAW-RGB data (A in FIG. 3) input to the D range expansion correction unit 50 and RAW-RGB data input to the white balance control unit 52 without being input to the D range expansion correction unit 50 side. (B in FIG. 3) is the same RAW-RGB data read from the SDRAM 23.

(First process: flowchart of FIG. 8)
In the first process, for example, when the photographer has an extremely bright part of the background of the subject, the photographer presses the menu (MENU) button 12 (see FIG. 1C), for example, A shooting setting screen as shown in FIG. 10 is displayed on the liquid crystal monitor (LCD) 9, and the control signal is sent from the control unit 28 to the YUV conversion unit 36 by selecting the item “dynamic range double” from this display screen. Is output, and the dynamic range expansion mode for expanding the dynamic range by a factor of two is executed.

  Then, when the photographer presses the release button 2 (half press to full press) from the monitoring operation described above, the camera I / F 34 of the signal processing unit 22 passes through the shooting lens system 5, the CCD 20, and the AFE unit 21. RAW-RGB data is captured (step S1).

  Then, the camera I / F 34 calculates an AE (automatic exposure) evaluation value, an AF (automatic focus) evaluation value, and an AWB (auto white balance) evaluation value from the captured RAW-RGB data. AE processing is performed based on the calculated AE evaluation value, and exposure at the time of photographing is determined (step S2). Further, the control unit 28 performs AF processing based on the calculated AF evaluation value, and moves the focus lens of the photographic lens system 5 to the in-focus position for focusing (step S3).

  Then, by performing exposure for photographing (recording) based on the exposure condition determined in step S2 (step S4), a subject image is formed on the light receiving surface of each pixel of the CCD 20. Then, an electrical signal (analog RGB image signal) corresponding to the subject image output from the CCD 20 is output to the A / D converter 33 via the CDS 31 and the AGC 32, and 12 bits (bit) is output by the A / D converter 33. To RAW-RGB data. This RAW-RGB data is taken into the camera I / F 34 of the signal processing unit 22 and stored in the SDRAM 23 via the memory controller 35. The processing so far is the same as the normal still image shooting operation described above.

  Then, the RAW-RGB data (A in FIG. 3) read from the SDRAM 23 is input to the D range expansion correction unit 50 of the YUV conversion unit 36, and the luminance level determination unit 60 of the D range expansion correction unit 50 as described above. The pixel output of each pixel provided with the RGB filter is detected (step S5), and it is determined whether or not the pixel output is saturated depending on whether or not a preset saturation determination value has been reached (step S6). If it is determined that there is a pixel output reaching the saturation region (step S6; YES), it is determined whether or not the pixel output reaching the saturation region or more is only the pixel output of the G filter having the highest sensitivity. Determination is made (step S7).

  When it is determined in step S7 that the pixel output reaching the saturation region is only the pixel output of the G filter (step S7; YES), the luminance level determination unit 60 determines whether the expression (1) or (3) to (3) to A correction coefficient for correcting the pixel output of the G filter is calculated from equation (5) (step S8). Then, the pixel output correction processing unit 61 multiplies the pixel output of the G filter by the correction coefficient calculated in step S7 as in the above equation (2), and corrects the pixel output of the G filter (step S9). .

  Then, the bit compression conversion unit 51 converts the pixel output data of the G filter expanded to 14 bits by the D range expansion correction unit (luminance level determination unit 60, pixel output correction processing unit 61) 50 into, for example, FIG. ) Is subjected to bit compression into 12-bit data (step S10) using the conversion characteristics shown in FIG. 4 (four-section broken line approximation characteristics in which three nodes are designated and approximated by straight lines between them) (step S10). In FIG. 11A, a is a 12-bit range, and b is a simple linear conversion characteristic (dotted line portion) when the maximum value 8190 data is halved. In the conversion characteristic shown in FIG. 11A, since the maximum value of the pixel output of the G filter is 8190, compression is performed so that 8190 becomes 4095. Then, the pixel output values of the R and B filters are also compressed in accordance with the compression magnification of the pixel output of the G filter.

  The RGB filter pixel output data compressed and converted from 14 bits to 12 bits by the bit compression conversion unit 51 is input to the white balance control unit 52 and subjected to white balance (AWB) processing (step S11). Specifically, the white balance control unit 52 amplifies the input RAW-RGB data. At this time, the control unit 28 calculates a correction value for adjusting the white balance based on the AWB evaluation value calculated by the camera I / F 34 and outputs the calculated correction value to the white balance control unit 52. The white balance control unit 52 adjusts the white balance based on the input correction value.

  Then, the pixel output data (12 bits) of the RGB filter with the white balance adjusted from the white balance control unit 52 is input to the synchronization unit 53. The synchronizer 53 performs interpolation calculation processing on RAW data having only one color data per pixel, such as a Bayer array, and generates all RGB data for one pixel. All the RGB data (12 bits) generated by the synchronization unit 53 is input to the tone curve conversion unit 54. The tone curve conversion unit 54 performs, for example, γ conversion for converting 12-bit RGB data into 8-bit RGB data using a conversion table as shown in FIG. -Output to the YUV converter 55.

  The RGB-YUV converter 55 converts input RGB data (8 bits) into YUV data (hereinafter referred to as “first YUV data”) by matrix calculation (step S12). Then, the image size converter unit 56 reduces or enlarges the first YUV data to a desired image size and outputs it to the edge enhancement unit 57 and the luminance histogram generation unit 58.

  The edge enhancement unit 57 performs processing such as edge enhancement according to the image on the input first YUV data, and stores the first YUV data in the SDRAM 23 via the memory controller 35 (step S13). The luminance histogram generation unit 58 generates a luminance histogram based on the input first YUV data.

(Second process: flowchart of FIG. 9)
In the second process, the RAW-RGB data obtained in the first process (steps S1 to S4 in FIG. 8) is read from the SDRAM 23 (step S21), and the RAW-RGB data (B in FIG. 3) is read. The signal is input to the white balance control unit 52 without passing through the D range expansion correction unit 50 and the bit compression conversion unit 51, and the white balance (AWB) process is performed in the same manner as in the first process (step S22). In the RAW-RGB data (B in FIG. 3), as described in the first processing, the pixel output of the G filter reaches the saturation region.

  Then, the pixel output data (12 bits) of the RGB filter with the white balance adjusted from the white balance control unit 52 is input to the synchronization unit 53. The synchronizer 53 performs interpolation calculation processing on RAW data having only one color data per pixel, such as a Bayer array, and generates all RGB data for one pixel. All the RGB data (12 bits) generated by the synchronization unit 53 is input to the tone curve conversion unit 54. The tone curve conversion unit 54 performs, for example, γ conversion that converts 12-bit RGB data into 8-bit RGB data using a conversion table as shown in FIG. 12 to generate an 8-bit RGB value, The data is output to the RGB-YUV converter 55.

  The RGB-YUV converter 55 converts the input RGB data (8 bits) into YUV data (hereinafter referred to as “second YUV data”) by matrix calculation (step S23). Then, the image size converter unit 56 reduces or enlarges the second YUV data to a desired image size and outputs it to the edge enhancement unit 57 and the luminance histogram generation unit 58.

  The edge enhancement unit 57 performs processing such as edge enhancement according to the image on the input second YUV data, and stores the second YUV data in the SDRAM 23 via the memory controller 35 (step S24).

  The YUV synthesis unit 37 extracts only Y data (luminance data) from the first YUV data read from the SDRAM 23, and extracts only UV data (color difference data) from the second YUV data read from the SDRAM 23. YUV data (hereinafter referred to as “third YUV data”) is synthesized (step S25). The synthesized third YUV data is stored in the SDRAM 23 via the memory controller 35.

  The third YUV data is read from the SDRAM 23, converted into a size corresponding to the number of recorded pixels by the resizing processing unit 38, and compressed into image data such as JPEG format by the data compression unit 40 (step S26). . The compressed image data in JPEG format or the like is written back to the SDRAM 23, then read from the SDRAM 23 via the memory controller 35, and recorded on the memory card 14 via the media I / F 41 (step S27).

  By the way, in step S10 of the first process described above (the flowchart of FIG. 9), when the pixel output data of the G filter expanded to 14 bits is bit-compressed to 12-bit data, the maximum value is expanded to 8190. As an example of compressing the pixel output of the G filter to a maximum value of 4095, a conversion characteristic having three nodes as shown in FIG. As a result, the following two effects that cannot be obtained with a linear conversion characteristic without a simple node (b in FIG. 11A) are obtained.

  As a first effect, a larger number of bits can be assigned to data with high data reliability. That is, when performing correction processing on the pixel output of the G filter that has reached the saturation region or more, as described above, correction is performed for the range in which the value of the pixel output of the G filter is equal to or greater than the specified value near the saturation region, Correction is not performed in the range below this specified value. Therefore, the accuracy of data differs between the range where correction is performed and the range where correction is not performed.

  That is, for example, when correcting the pixel output value of the saturated G filter according to the equations (1) to (5), the luminance level of the subject is accurately reproduced in the correction range depending on the color of the main subject. It may not be possible. On the other hand, the range where correction is not performed is data obtained by A / D conversion of actual data (analog RGB image signal) output from the CCD 20 having the RGB filter, and thus the reliability of this data is high. Become.

  That is, in the conversion characteristics in this embodiment shown in FIG. 11A, for example, when the input 14-bit data is 1024, the output 12-bit data is 1024, and the original data is used as it is. On the other hand, for example, when the input 14-bit data is 3072, the output 12-bit data is 2560, and in this range, the bit allocation is smaller than the bit allocation before correction, so that some bit error occurs.

  In this way, bit allocation is gradually increased by adopting a conversion characteristic having three nodes as in this embodiment instead of a characteristic for performing linear conversion without simple nodes (b in FIG. 11A). Therefore, it is possible to allocate more bits to data with high data reliability.

  As a second effect, gradations at low and medium luminance can be accurately stored. That is, when bit compression is performed with a simple linear conversion characteristic (b in FIG. 11A), the bit allocation becomes 1/4 in a range where correction on the low luminance side is not performed. For this reason, the image has no tone. On the other hand, when bit compression is performed with the conversion characteristics in the present embodiment as shown in FIG. 11A, the bit corresponding to the pixel output at the low luminance level below the saturation region is a bit. By using a compression ratio that gives substantially the same value before bit compression by the compression conversion unit 51 and after bit compression, it is possible to maintain good gradation at a low luminance level.

  In this embodiment, when the 14-bit data of the pixel output of the expanded G filter is reduced to 12 bits, three nodes are designated as shown in FIG. 11A and approximated by a straight line between them. Although the bit compression is performed with the broken line approximation characteristic (conversion characteristic) of four sections, the number of sections is not particularly limited. For example, it may be a polygonal line approximate characteristic of two sections designating one node, but the above-mentioned two effects are reduced by changing the bit allocation in the vicinity of the node. Therefore, a broken line approximation characteristic (conversion characteristic) having three or more sections is preferable.

  Further, the conversion characteristic for compressing the 14-bit data of the expanded G filter pixel output to 12 bits may be a conversion characteristic by a curve having no plurality of nodes as shown in FIG. That is, in contrast to the conversion characteristic having four sections in FIG. 11A, a conversion characteristic with this curve is obtained by setting the number of sections to 8192. In FIG. 11B, a is a 12-bit range.

  Further, by providing a lookup table having numerical data after compression conversion to 12 bits for 0 to 8192 of input 14-bit data, the pixel output of the expanded G filter with the conversion characteristics by this curve is provided. The 14-bit data can be successfully compressed to 12 bits.

  As described above, by performing the still image shooting operation in the dynamic range expansion mode of the present embodiment, even when the pixel output of the G filter with high sensitivity reaches the saturation region, the sensitivity is higher than that of the G filter. By correcting the pixel output of the saturated G filter based on the pixel output of the low R and B filters, the pixel output of the G filter (d and e in FIG. 4) is corrected as shown in FIG. Based on the corrected extended region (the portion indicated by the alternate long and short dash line of the pixel output of the G filter of d and e in FIG. 4), the dynamic range can be doubled by one shooting.

  Therefore, even when there is a high-luminance portion in the background or the like in the captured image, it is possible to prevent the occurrence of overexposure and obtain good gradation.

  Further, in the present embodiment, one RAW-RGB data is read from the SDRAM 23, and the first YUV data generated from the pixel output corrected by the pixel output correction processing unit 61 by the first processing, and the second The second YUV data generated from the pixel output reaching the saturation region without performing the correction process in the pixel output correction processing unit 61 by the processing is taken into the YUV synthesizing unit 37, and the YUV synthesizing unit 37 extracts the first YUV data from the first YUV data. The data (luminance data) and the UV data (color difference data) extracted from the second YUV data are combined to generate third YUV data.

  Thereby, the dynamic range is expanded only for the Y data (luminance data) of the first YUV data generated from the pixel output corrected by the pixel output correction processing unit 61, and the original UV data (color difference data) is used as the color. Accordingly, it is possible to prevent a hue shift when expanding the dynamic range and to perform accurate color reproduction.

  For example, the D range expansion correction unit 50 performs correction processing of the pixel output of the G filter determined to be saturated using the above formulas (1) to (5), and performs dynamic range expansion processing. Sometimes, for example, when the main subject is magenta, the pixel output of the R and B filters is large. For this reason, the pixel output of the G filter is calculated to a value larger than the original value accordingly, and as a result, accurate color reproduction may not be possible.

  Even in such a situation, the dynamic range is expanded only for the Y data (luminance data) of the first YUV data by performing the still image shooting operation in the dynamic range expansion mode of the present embodiment, and the color is the original. By using this UV data (color difference data), it is possible to prevent hue shift when expanding the dynamic range and to accurately reproduce colors.

  FIG. 13A shows an example of a histogram generated by the luminance histogram generation unit 58 when the dynamic range expansion process in the present embodiment is performed when the pixel output of the G filter exceeds the saturation region. It is. As is apparent from this histogram, when the dynamic range expansion process in the present embodiment described above is performed, an appropriate gradation is ensured for the high-luminance portion, leaving pixels up to the maximum luminance portion (255). And the occurrence of whiteout can be prevented.

  On the other hand, FIG. 13B is generated by the luminance histogram generation unit 58 when the dynamic range expansion process in the present embodiment is not performed when the pixel output of the G filter is in the saturation region. It is an example of a histogram. As is clear from this histogram, when the dynamic range expansion process in this embodiment is not performed, a large number of pixels exist in the maximum luminance portion (255), and overexposure occurs.

  In the description of Embodiment 1 and FIG. 4 described above, the saturation level A in FIG. 4 that is the saturation determination value and 4095 that is the 12-bit maximum value after correction are the same. It is not limited to this. For example, in a CCD having an RGB filter whose output linearity (linearity) is not good in a high-luminance part near where the output is completely saturated, for example, 4032 which is smaller than 4095 which is completely saturated is set as a saturation determination value ( At the saturation level A) in FIG. 4, the pixel output exceeding the value may be the target of the correction process. Even in this case, the pixel of the R and B filters with respect to the saturation determination threshold (in this case, for example, 4032/2 = 2016) for determining that the pixel output of the G filter exceeds the saturation determination value of 4032 The saturation of the pixel output of the G filter is determined depending on whether or not the output exceeds.

  Also, depending on the system configuration of the digital camera, even a high-luminance subject may not reach 4095, which is the maximum value of 12 bits. In that case as well, the saturation determination value may be set to a value lower than 4095.

  As described above, even when the saturation region is less than 4095, the output of the bit compression conversion unit 51 can be set to 4095 by switching the conversion curve shown in FIG. The dynamic range can be expanded without change.

  In the present embodiment, as described above, it is assumed that the pixel sensitivity characteristics of the G filter have a sensitivity that is about twice as high as the pixel sensitivity characteristics of the R and B filters. Under the blue light source or the blue light source, the pixel sensitivity of the R and B filters may be saturated rather than the pixel sensitivity of the G filter. If the dynamic range expansion process described above is performed under such circumstances, accurate gradation and color reproduction cannot be obtained. In such a case, the dynamic range expansion process described above is performed at the discretion of the photographer. Do not.

  In the present embodiment, as shown in FIG. 3, the 14-bit RAW-RGB data output from the D range expansion correction unit 50 is compressed to 12 bits by the bit compression conversion unit 51, and the white balance control unit 52, the synchronization unit 53 is configured to perform 12-bit data processing, but in addition to this, a bit compression conversion unit 51 is provided after the synchronization unit 53 and is output from the synchronization unit 53. The bit data may be compressed into 12-bit data.

<Embodiment 2>
In the first embodiment, as shown in FIG. 6, 2 × 2 pixels are used as the processing unit (minimum unit) for the CCD 20 having the RGB filter. However, in this embodiment, as shown in FIG. 5 pixels in frame A (one G filter pixel (Gb) in the center, two R (R1, R2) filter pixels in the vertical direction in the figure, and two B (B1, B2) in the horizontal direction in the figure In this example, the pixel of the filter is a processing unit (minimum unit), and the processing unit is in a wider range than in the first embodiment. The configuration of the digital camera, the monitoring operation, the still image shooting operation, and the dynamic range expansion processing operation are the same as those in the first embodiment.

  In the present embodiment, the correction coefficient calculation unit 60 calculates the average value of the pixel outputs of the two R1 and R2 filters around the pixel of one G filter in the processing unit (see FIG. 14), and the two B1 and B2 An average value of the pixel output of the filter is calculated, and each calculated average value is set as a value of each pixel output of the R and B filters in this processing unit.

  When the pixel output of the G filter in the processing unit of the thick frame A shown in FIG. 14 reaches the saturation region, the sensitivity of the G filter is about twice the sensitivity of the R and B filters as described above. Therefore, the correction coefficient (K) of the G filter pixel output and the corrected pixel output (Ge) of the G filter are calculated from the following equations (6) and (7). In this embodiment,

K = {l * f (Ra) + m * f (Ga) + n * f (Ba)} / 3 ... Formula (6)
Ge = K × Ga (7)
Here, l, m, and n are coefficients set from the ratios of the RGB filters, and Ga is the pixel output of the G filter before correction. Further, f (Ra), f (Ga), and f (Ba) are coefficients set by the following formula 2 (Expression (8) to Expression (10)).

  However, Ra is an average value of the pixel output of the R filter in the processing unit (see FIG. 14), TR is a G pixel saturation determination level corresponding to the pixel output of the R filter, and Ga is the processing unit (see FIG. 14). G filter pixel output, TG is the saturation determination level of the G filter pixel output, Ba is the average value of the B filter pixel output in the processing unit (see FIG. 14), and TB is the B filter pixel output. G pixel saturation determination level corresponding to.

  The TR, TG, and TB are the same as those in the equations (3) to (5). The coefficients l, m, and n are 3/2 and m are 0, respectively, if the sensitivity ratio of the pixel outputs of the R, G, and B filters is the same as in the first embodiment.

  Then, the pixel output correction processing unit 61 of the D range expansion correction unit 50 shown in FIG. 6 has the pixel output value of the G filter calculated by the equation (7) in the processing unit (see FIG. 14). The pixel output value of the G filter is replaced, and the same processing as in the first embodiment is performed.

  In this way, by widening the processing unit, it is possible to alleviate the influence due to the sensitivity difference of the other R1, R2 filter pixels and B1, B2 filter pixels in the processing unit. It is possible to perform more accurate correction on the pixel output of the G filter that has reached.

<Embodiment 3>
In this embodiment, as shown in FIG. 15, for a CCD 20 having an RGB filter, 5 × 5 pixels (13 G filter pixels (Gr , Gb), and six R, B filters) as processing units. The configuration of the digital camera, the monitoring operation, the still image shooting operation, and the dynamic range expansion processing operation are the same as those in the first embodiment.

  If the processing unit is wider than those in the first and second embodiments, processing is performed based on a wide range of luminance information, which is equivalent to applying a low-pass filter. Therefore, the edge portion of the luminance change is lost. Therefore, in the present embodiment, the size of the widened processing unit is partially changed using, for example, the AF evaluation value.

  That is, the camera I / F 34 of the signal processing unit 22 shown in FIG. 2 calculates an AF evaluation value for performing AF as described above. This is an output of a high-pass filter (HPF), and a large value is output in a portion where there is a change in luminance in the screen of the captured image. Then, the control unit 28 reads out an AF evaluation value at the time of still image shooting, and discriminates between a portion having a luminance change and a portion having no luminance change in the screen. Then, the control unit 28 sets the D range expansion correction unit 50 based on the determination data so that the processing unit is narrowed in the portion where the luminance change occurs (for example, the processing unit in the first and second embodiments). In a portion where there is no change in luminance, the setting is performed so that the processing unit is in a wide range as shown in FIG.

  As described above, even when the processing unit is further widened, the pixel output of the G filter that has reached the saturation region without reducing the resolution can be obtained by performing the setting so that the processing unit is narrowed in the portion where the luminance changes. Therefore, accurate correction can be performed.

  In each of the embodiments described above, the RGB three primary color filters are arranged as the color separation filters. However, the present invention is similarly applied to a configuration in which the complementary color filters are arranged as the color separation filters. Can do.

(A) is a front view which shows the digital camera as an example of the imaging device which concerns on Embodiment 1 of this invention, (b) is the top view, (c) is the back view. 1 is a block diagram showing an outline of a system configuration in a digital camera as an example of an imaging apparatus according to Embodiment 1 of the present invention. The block diagram which shows the structure of the YUV conversion part in Embodiment 1 of this invention. The figure for demonstrating the principle of the dynamic range expansion in Embodiment 1 of this invention. The block diagram which shows the structure of the YUV conversion part in Embodiment 1 of this invention. The figure which shows arrangement | positioning and a processing unit of RGB filter in Embodiment 1 of this invention. The figure which showed G pixel saturation determination level (TG, TR, TB) corresponding to each pixel output of RGB filter. 6 is a flowchart showing a still image shooting operation in a dynamic range expansion mode (first process) according to the first embodiment of the present invention. 5 is a flowchart showing a still image shooting operation in a dynamic range expansion mode (second processing) according to the first embodiment of the present invention. Will be described with reference to FIG. The figure which shows an example of the imaging | photography setting screen displayed on the liquid crystal monitor. (A) is a figure which shows the conversion characteristic which compresses the 14-bit data of the pixel output of the extended G filter in Embodiment 1 of this invention to 12 bits, (b) is other of Embodiment 1 of this invention The figure which shows the conversion characteristic which compresses 14 bits data of the pixel output of the extended G filter in an example to 12 bits. The figure which shows the conversion table which converts 12-bit RGB data into 8-bit RGB data (gamma conversion). (A) is a figure which shows an example of the histogram at the time of performing the expansion process of the dynamic range in Embodiment 1 of this invention, (b) is the histogram at the time of not performing the expansion process of the dynamic range in this embodiment. The figure which shows an example. The figure which shows arrangement | positioning and a process unit of RGB filter in Embodiment 2 of this invention. The figure which shows arrangement | positioning and a processing unit of RGB filter in Embodiment 3 of this invention.

Explanation of symbols

1 Digital camera (imaging device)
5 Shooting lens system (optical system)
6 Lens unit 9 LCD monitor 12 Menu button (operation selection means)
20 CCD (imaging device)
21 Analog Front End 22 Signal Processor 23 SDRAM
28 Control Unit 34 Camera Interface 35 Memory Controller 36 YUV Conversion Unit 37 YUV Composition Unit (YUV Composition Unit)
50 D range expansion correction unit 51 Bit compression conversion unit (bit compression conversion means)
60 Luminance level determination unit (peripheral pixel saturation determination means)
61 Pixel output correction processing unit (saturated pixel output correction processing means)
62-bit extension processing section

Claims (12)

An image sensor in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are disposed on the front side of each pixel;
Whether the output from each pixel detects the output from the pixel, and whether the output from the pixel in which a filter of a specific color is arranged based on the detected pixel output reaches a saturation region where the input / output characteristics are not linear Peripheral pixel saturation determination means for determining
When it is determined by the peripheral pixel saturation determination means that the output from the pixel in which the filter of the specific color is arranged has reached the saturation region, from the pixel in which a filter other than the specific color in the vicinity is arranged Saturated pixel output correction processing means for correcting the output from the pixel in which the filter of the specific color is arranged based on the output of
Generated from the first YUV data generated from the pixel output corrected by the saturated pixel output correction processing means and the pixel output reaching the saturation region without performing the correction processing by the saturated pixel output correction processing means YUV combining means for generating third YUV data obtained by combining the luminance data extracted from the first YUV data and the color difference data extracted from the second YUV data. An imaging device.   The imaging apparatus according to claim 1, wherein a processing unit when the output of each pixel is detected by the peripheral pixel saturation determination unit is a size of 2 × 2 pixels in the horizontal and vertical directions.   An operation selection means for selecting and executing the operation to be corrected when the output of the pixel in which the filter of the specific color is arranged reaches the saturation region or more is provided. The imaging apparatus according to claim 1 or 2. From the first number of bits that can represent the saturated region of the pixel output that is output from the saturated pixel output correction processing means when the output of the pixel in which the filter of the specific color has reached the saturated region Bit compression conversion means for compressing the pixel output data once expanded to the second number of bits that can be expressed beyond the saturation region of the pixel output to the first number of bits again,
The bit compression conversion means compresses data by reducing a compression rate for data corresponding to a pixel output in the saturation region or lower than a compression rate for data corresponding to a pixel output in the saturation region or higher. Item 4. The imaging device according to any one of Items 1 to 3.   For data corresponding to a pixel output at a low luminance level below the saturation region, a compression rate is used so that the bit compression conversion means has substantially the same value before bit compression and after bit compression. The imaging device according to claim 4.   2. If there is a defective pixel in the processing unit, a pixel in which a filter of the same color as the defective pixel around the defective pixel is used instead of the defective pixel. The imaging device according to claim 5. An imaging device including an imaging device in which a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a plurality of color separation filters are arranged on the front side of each pixel. In the imaging method of the apparatus,
Whether the output from each pixel detects the output from the pixel, and whether the output from the pixel in which a filter of a specific color is arranged based on the detected pixel output reaches a saturation region where the input / output characteristics are not linear Surrounding pixel saturation determination processing step for determining
When it is determined in the peripheral pixel saturation determination processing step that the output from the pixel in which the filter of the specific color is arranged has reached the saturation region, from the pixel in which the filter other than the specific color in the vicinity is arranged A saturated pixel output correction processing step for correcting the output from the pixel in which the filter of the specific color is arranged based on the output of
Generated from the first YUV data generated from the pixel output corrected by the saturated pixel output correction processing step and the pixel output reaching the saturation region without performing the correction processing in the saturated pixel output correction processing step YUV synthesis processing step of generating third YUV data by taking in the second YUV data to be generated and synthesizing the luminance data extracted from the first YUV data and the color difference data extracted from the second YUV data. An imaging method.   The imaging method according to claim 7, wherein a processing unit when detecting an output of each pixel in the peripheral pixel saturation determination processing step is a size of 2 × 2 pixels in the horizontal and vertical directions.   The operation for correcting the output from the pixel in which the filter of the specific color is arranged when the saturation region is reached is performed by selecting an operation by an operation selection unit. Item 9. The imaging method according to Item 7 or 8. From the first number of bits that can represent the saturated region of the pixel output that is output from the saturated pixel output correction processing means when the output of the pixel in which the filter of the specific color has reached the saturated region A bit compression conversion processing step for compressing the pixel output data once expanded to the second number of bits that can be expressed beyond the saturation region of the pixel output to the first number of bits again,
In the bit compression conversion processing step, compression is performed by reducing a compression rate for data corresponding to a pixel output in the saturation region or lower than a compression rate for data corresponding to a pixel output in the saturation region or higher. The imaging method as described in any one of Claims 7 thru | or 9.   For data corresponding to pixel output at a low luminance level below the saturation region, compression is performed using a compression ratio that is substantially the same before and after bit compression in the bit compression conversion processing step. The imaging method according to claim 10.   8. If there is a defective pixel in the processing unit, a pixel in which a filter of the same color as the defective pixel around the defective pixel is used instead of the defective pixel. The imaging method according to claim 11.

Images