JP2009118158A - Imaging apparatus and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus which can expand dynamic range at one shooting, without performing compositing of a plurality of shootings by changing exposure, and to provide an imaging method. <P>SOLUTION: The imaging apparatus has a brightness level deciding part 60, which detects an output from each pixel in which RGB filters are placed, and decides whether the output from each pixel is reaching a predetermined saturation level or higher; a pixel output correction processing part 61 which, when it is decided that the output from a pixel where the G filter is placed, reaches the predetermined saturation level or higher by a brightness level decision part 60, predicts and interpolates pixel output in an area of the saturation level or higher to expand the dynamic range, on the basis of the output from unsaturated pixels where R, B filters are placed; and a control part which changes and controls the expansion ratio to the dynamic range, to be expanded by the pixel output correction processing part 61, on the basis of an expansion ratio set by the operation of a menu button. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method such as a digital still camera and a digital video camera, and more particularly to an imaging apparatus and an imaging method having a function capable of expanding a dynamic range of a captured image.

  Compared to the dynamic range of images taken with a conventional silver salt camera using a silver salt photographic film, the dynamic range of images taken with a digital still camera or a digital video camera having a solid-state image sensor such as a CCD is extremely narrow. When the dynamic range is narrow, a phenomenon called “blackout” occurs in the dark part of the subject, and conversely, a phenomenon called “overexposure” occurs in the bright part of the subject, and the image quality deteriorates.

Therefore, in order to expand the dynamic range of an image captured by a solid-state imaging device such as a CCD, for example, the same subject is shot multiple times with different exposure amounts to obtain a plurality of images with different exposure amounts. A technique for adding these images to generate a composite image with an expanded dynamic range has been known (see, for example, Patent Document 1).
JP 2000-92378 A

  By the way, in order to expand the dynamic range, in the method of performing photographing a plurality of times while changing the exposure amount as in the above-mentioned Patent Document 1, when photographing a moving object, the subject is doubled. Therefore, the image may not be synthesized correctly.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus and an imaging method capable of expanding the dynamic range by one shooting without changing the exposure amount and performing a plurality of shootings to combine images. .

  In order to achieve the above object, according to the first aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and the front side of each pixel is provided. An image sensor having a color separation filter of three primary color filters or a complementary color filter is arranged, and a pixel in which a filter of a specific color is arranged for light having a wide wavelength band is arranged with a filter of another color In an imaging device having higher luminance sensitivity than a pixel, the output from each pixel in which the color separation filter is arranged is detected based on an electrical signal output from the imaging element, and each pixel A pixel output detecting means for determining whether or not the output from the pixel reaches a predetermined saturation level or more, and an output from the pixel in which the filter of the specific color is arranged by the pixel output detecting means When it is determined that the predetermined saturation level or higher is reached, the pixel output in the region higher than the saturation level is determined based on the output from the pixels where the other color filters that are not saturated are arranged. A pixel output correction processing unit that expands a dynamic range by predictive interpolation, a dynamic range expansion rate setting unit that sets a dynamic range expansion rate, and a dynamic range that is expanded by the predictive interpolation processing by the pixel output correction processing unit And an enlargement ratio change control means for changing and controlling the enlargement ratio based on the enlargement ratio set by the dynamic range enlargement ratio setting means.

  According to a second aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a three primary color filter or a complementary color system is provided on the front side of each pixel. An image sensor with a filter color separation filter is arranged, and a pixel with a filter of a specific color for light having a wide wavelength band has higher luminance sensitivity than a pixel with a filter of another color. And detecting an output from each of the pixels in which the color separation filter is arranged based on an electrical signal output from the imaging element, and outputting from each of the pixels to a predetermined saturation A pixel output detecting means for determining whether or not the level has been reached, and an output from the pixel in which the filter of the specific color is arranged by the pixel output detecting means is less than or equal to the predetermined saturation level. When it is determined that the pixel value has reached the pixel output in the region above the saturation level based on the output from the pixels around which the filters of the same color as the specific color that are not saturated are arranged The pixel output correction processing means for expanding the dynamic range, the dynamic range expansion ratio setting means for setting the expansion ratio of the dynamic range, and the expansion ratio for the dynamic range expanded by the predictive interpolation process by the pixel output correction processing means And an enlargement rate change control means for performing change control based on the enlargement rate set by the dynamic range enlargement rate setting means.

  According to a third aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electric signal, and a three primary color filter or a complementary color system is provided on the front side of each pixel. An image sensor with a filter color separation filter is arranged, and a pixel with a filter of a specific color for light having a wide wavelength band has higher luminance sensitivity than a pixel with a filter of another color. And detecting an output from each pixel in which the color separation filter is disposed from an electric signal output from the image sensor, and an output from each pixel is equal to or higher than a predetermined saturation level. A pixel output detection means for determining whether or not the output from the pixel in which the filter of the specific color is arranged by the pixel output detection means reaches the predetermined saturation level or more. A first pixel output that predictively interpolates a pixel output in the region above the saturation level based on an output from a pixel in which the other color filters that are not saturated are arranged, When it is determined by the correction processing unit and the pixel output detection unit that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, a plurality of surrounding non-saturated Second pixel output correction processing means for predictively interpolating a pixel output in the region above the saturation level based on an output from a pixel in which a filter of the same color as the specific color is arranged; and the first pixel output correction processing On the basis of the prediction interpolation information respectively output from the first pixel output correction processing means and the second pixel output correction processing means. Predictive interpolation processing means for expanding the dynamic range by performing dynamic range expansion ratio setting means for setting the expansion ratio of the dynamic range, and expansion ratio for the dynamic range expanded by the predictive interpolation processing by the predictive interpolation processing means. And an enlargement ratio change control means for performing change control based on the enlargement ratio set by the dynamic range enlargement ratio setting means.

  The inventions according to claims 4 and 17 are characterized in that a processing unit for detecting the output of each pixel has a size of 2 × 2 pixels in the horizontal and vertical directions.

  According to a fifth aspect of the present invention, there is provided a histogram generating means for calculating a pixel occurrence frequency between a low luminance and a maximum high luminance corresponding to a dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means. The enlargement ratio change control means includes the histogram when it is determined by the pixel output detection means that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high luminance side to the low luminance side with respect to the histogram generated by the generation means, the pixel occurrence frequency is equal to or greater than a predetermined value. The dynamic range expanded by the pixel output correction processing means is set so that the position to become the new maximum high brightness position of the histogram. It is characterized by changing control that magnification.

  According to a sixth aspect of the present invention, there is provided a histogram generating means for calculating a pixel occurrence frequency between a low luminance and a maximum high luminance corresponding to a dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means. The enlargement ratio change control means includes the histogram when it is determined by the pixel output detection means that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high luminance side to the low luminance side with respect to the histogram generated by the generation means, the pixel occurrence frequency is equal to or greater than a predetermined value. To the dynamic range expanded by the predictive interpolation processing means so that the position to become the new maximum high brightness position of the histogram It is characterized by changing controlling a large rate.

  According to a seventh aspect of the invention, there is provided a histogram generating means for calculating a pixel occurrence frequency between a low luminance and a maximum high luminance corresponding to a dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means. The enlargement ratio change control means includes the histogram when it is determined by the pixel output detection means that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. The position of the histogram in which the value of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated by the generating unit is a predetermined ratio with respect to the value of the total luminance data number Change control of the enlargement ratio for the dynamic range expanded by the pixel output correction processing means so that it becomes a new maximum high brightness position It is characterized by a Rukoto.

  According to an eighth aspect of the present invention, there is provided a histogram generating means for calculating a pixel occurrence frequency between a low luminance and a maximum high luminance corresponding to a dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means. The enlargement ratio change control means includes the histogram when it is determined by the pixel output detection means that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. The position of the histogram in which the value of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated by the generating unit is a predetermined ratio with respect to the value of the total luminance data number The enlargement ratio with respect to the dynamic range enlarged by the predictive interpolation processing means is changed and controlled so as to be a new maximum high-luminance position. It is characterized by a door.

  The invention according to claim 9 is characterized in that the predetermined saturation level output from the pixel output correction processing means when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. Bit compression conversion means for compressing pixel output data once expanded from the first bit number to the second bit number to the first bit number again based on a preset bit compression conversion characteristic table The enlargement ratio change control means controls to change the enlargement ratio for the dynamic range expanded by the pixel output correction processing means based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed. Yes.

  According to a tenth aspect of the present invention, the predetermined saturation level output from the pixel output correction processing means when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. Bit compression conversion means for compressing pixel output data once expanded from the first bit number to the second bit number to the first bit number again based on a preset bit compression conversion characteristic table The enlargement ratio change control means is characterized in that, based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed, the enlargement ratio for the dynamic range expanded by the predictive interpolation processing means is changed and controlled. .

  In the invention according to claim 11, the enlargement ratio change control unit is configured to output the pixel output correction processing unit from the pixel output correction processing unit when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. The pixel output correction processing means multiplies the pixel output data that is output from the first bit number below the predetermined saturation level to the second bit number by a predetermined coefficient to multiply the data. It is characterized by changing and controlling the enlargement ratio for the expanded dynamic range.

  In the twelfth aspect of the present invention, the enlargement ratio change control unit is configured to output the pixel output correction processing unit from the pixel output correction processing unit when the output of the pixel in which the filter of the specific color is arranged reaches the predetermined saturation level or more. The output is expanded by the predictive interpolation processing means by multiplying the pixel output data once expanded from the first bit number to the second bit number below the predetermined saturation level by a predetermined coefficient. It is characterized by changing and controlling the enlargement ratio with respect to the dynamic range.

  According to a thirteenth aspect of the present invention, in order to select and execute the operation of performing the predictive interpolation when the output of the pixel in which the filter of the specific color is arranged reaches the predetermined saturation level or more. It is characterized by comprising the operation selection means.

According to the fourteenth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a three primary color filter or a complementary color system is provided on the front side of each pixel. An image sensor with a filter color separation filter is arranged, and a pixel with a filter of a specific color for light having a wide wavelength band has higher luminance sensitivity than a pixel with a filter of another color. In the imaging method of the imaging apparatus, the output from each pixel in which the color separation filter is arranged is detected based on the electrical signal output from the imaging element, and the output from each pixel is A pixel output detecting step for determining whether or not a predetermined saturation level or more is reached, and an output from the pixel in which the filter of the specific color is arranged by the pixel output detecting step is the predetermined output When it is determined that the sum level has been reached, the pixel output in the region above the saturation level is predicted and interpolated based on the output from the pixel in which the other color filters that are not saturated are arranged. A pixel output correction process that expands the dynamic range,
An enlargement ratio for changing and controlling the enlargement ratio for the dynamic range enlarged by the predictive interpolation process in the pixel output correction processing step based on the enlargement ratio set by the dynamic range enlargement ratio setting means for setting the enlargement ratio of the dynamic range And a change control step.

  According to the fifteenth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a three primary color filter or a complementary color system is provided on the front side of each pixel. An image sensor with a filter color separation filter is arranged, and a pixel with a filter of a specific color for light having a wide wavelength band has higher luminance sensitivity than a pixel with a filter of another color. In the imaging method of the imaging apparatus, the output from each pixel in which the color separation filter is arranged is detected based on the electrical signal output from the imaging element, and the output from each pixel is A pixel output detecting step for determining whether or not a predetermined saturation level or more is reached, and an output from the pixel in which the filter of the specific color is arranged by the pixel output detecting step is the predetermined output When it is determined that the sum level or higher is reached, the pixel output in the region above the saturation level is predicted based on the output from the pixel in which the filter of the same color as the specific color that is not saturated is arranged. A pixel output correction process for interpolating and expanding the dynamic range, and a dynamic range expansion ratio setting for setting the expansion ratio for the dynamic range expanded by the predictive interpolation process by the pixel output correction process, and setting the expansion ratio of the dynamic range And an enlargement rate change control step for performing change control based on the enlargement rate set by the means.

  According to the sixteenth aspect of the present invention, a subject image incident through an optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a three primary color filter or a complementary color system is provided on the front side of each pixel. An image sensor with a filter color separation filter is arranged, and a pixel with a filter of a specific color for light having a wide wavelength band has higher luminance sensitivity than a pixel with a filter of another color. In the imaging method of the imaging apparatus, the output from each pixel in which the color separation filter is arranged is detected from the electrical signal output from the imaging element, and the output from each pixel is a predetermined value. A pixel output detecting step for determining whether or not the saturation level is reached, and an output from the pixel in which the filter of the specific color is arranged by the pixel output detecting step is the predetermined saturation level. The pixel output in the region above the saturation level is predicted and interpolated based on the output from the pixel in which the other color filters that are not saturated are arranged. When the pixel output correction processing step and the pixel output detection step determine that the output from the pixel in which the filter of the specific color is disposed has reached the predetermined saturation level or higher, the surrounding saturation is performed. A second pixel output correction processing step for predictively interpolating a pixel output in the region above the saturation level based on an output from a pixel in which a filter of the same color as the specific color is not disposed; and the first pixel output Based on prediction interpolation information output from the correction processing step and the second pixel output correction processing step, prediction appropriate for pixel output in the region above the saturation level Predictive interpolation processing step for expanding the dynamic range by performing inter-process, and the dynamic range expansion rate setting for setting the expansion rate of the dynamic range for the dynamic range expanded by the predictive interpolation processing by the predictive interpolation processing step And an enlargement rate change control step for performing change control based on the enlargement rate set by the means.

  The invention according to claim 18 further includes a histogram generation step of calculating an occurrence frequency of pixels between low luminance and maximum high luminance corresponding to the dynamic range expanded by the predictive interpolation processing in the second step. The enlargement ratio change control step generates the histogram when it is determined by the pixel output detection step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high luminance side to the low luminance side with respect to the histogram generated in the process, the pixel occurrence frequency is equal to or higher than a predetermined value. To the dynamic range expanded by the pixel output correction processing step so that the position becomes the new maximum brightness position of the histogram. It is characterized by changing control that magnification.

  According to a nineteenth aspect of the present invention, there is provided a histogram generation step of calculating a pixel occurrence frequency between a low luminance and a maximum high luminance corresponding to a dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step. In addition, the enlargement ratio change control step includes the step of determining when the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more by the pixel output detection step. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high luminance side to the low luminance side with respect to the histogram generated by the histogram generation means, a prescribed value in which the pixel occurrence frequency is prescribed in advance The dynamic range expanded by the predictive interpolation processing step so that the above position becomes the new maximum high brightness position of the histogram. It is characterized by changing control the magnification against.

  The invention according to claim 20 includes a histogram generation step of calculating an occurrence frequency of pixels between low luminance and maximum high luminance corresponding to the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step. In addition, the enlargement ratio change control step includes the step of determining when the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more by the pixel output detection step. The histogram shows the position where the value of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated in the histogram generation step is a predetermined ratio with respect to the value of the total luminance data number The enlargement ratio with respect to the dynamic range expanded by the pixel output correction processing step is set to be a new maximum high luminance position. It is characterized in that a further control.

  According to a twenty-first aspect of the present invention, there is provided a histogram generation step of calculating a pixel occurrence frequency between low luminance and maximum high luminance corresponding to the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step. In addition, the enlargement ratio change control step includes the step of determining when the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more by the pixel output detection step. The histogram shows the position where the value of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated in the histogram generation step is a predetermined ratio with respect to the value of the total luminance data number The enlargement ratio for the dynamic range expanded by the predictive interpolation process is changed to be the new maximum high-intensity position. It is characterized in that Gosuru.

  According to a twenty-second aspect of the present invention, the predetermined saturation level output from the pixel output correction processing step when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. A bit compression conversion step of compressing pixel output data once expanded from the first bit number to the second bit number to the first bit number again based on a preset bit compression conversion characteristic table. In addition, the enlargement ratio change control step controls to change the enlargement ratio for the dynamic range expanded by the pixel output correction processing step based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed. It is said.

  The invention according to claim 23 is the predetermined saturation level output from the pixel output correction processing step when the output of the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. A bit compression conversion step of compressing pixel output data once expanded from the first bit number to the second bit number to the first bit number again based on a preset bit compression conversion characteristic table. In addition, the enlargement ratio change control step is characterized in that, based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed, the enlargement ratio for the dynamic range expanded by the predictive interpolation process step is controlled to change. Yes.

  According to a twenty-fourth aspect of the present invention, the enlargement ratio changing control step outputs from the pixel output correction processing step when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. The pixel output data is expanded by multiplying the data of the pixel output once expanded from the first bit number below the predetermined saturation level to the second bit number by a predetermined coefficient. It is characterized by changing and controlling the enlargement ratio with respect to the dynamic range.

  According to a twenty-fifth aspect of the present invention, the enlargement ratio change control step outputs the pixel output correction processing step when the output of the pixel in which the filter of the specific color is arranged reaches the predetermined saturation level or more. The pixel output data expanded once from the first bit number below the predetermined saturation level to the second bit number is multiplied by a predetermined coefficient to be expanded by the predictive interpolation processing step. It is characterized by changing and controlling the enlargement ratio for the dynamic range.

  According to a twenty-sixth aspect of the present invention, the operation of performing the predictive interpolation when the output of the pixel in which the filter of the specific color is arranged exceeds the predetermined saturation level is selected by an operation selection unit. It is characterized by being executed.

  According to the invention described in claims 1 and 14, when it is determined that the output from the pixel in which the filter of the specific color is arranged has reached a predetermined saturation level or more, other surroundings that are not saturated Based on the output from the pixel where the color filter is placed, the pixel output in the region above the saturation level is predicted and interpolated to expand the dynamic range, thereby changing the exposure amount and shooting multiple times to synthesize the image. The dynamic range can be expanded by one shooting without any problem. Furthermore, since the enlargement ratio of the dynamic range to be enlarged can be changed and controlled, the dynamic range can be appropriately enlarged according to the captured image.

  According to the second and fifteenth aspects of the present invention, when it is determined that the output from the pixel in which the filter of the specific color is arranged has reached a predetermined saturation level or higher, the specific color that is not saturated around the output. Based on the output from the pixel where the same color filter is placed, the pixel output in the region above the saturation level is predicted and interpolated to expand the dynamic range, thereby changing the exposure amount and shooting multiple times to capture the image. Without combining, the dynamic range can be expanded by one shooting. Furthermore, since the enlargement ratio of the dynamic range to be enlarged can be changed and controlled, the dynamic range can be appropriately enlarged according to the captured image.

  According to the invention described in claims 3 and 16, when it is determined that the output from the pixel in which the filter of the specific color is arranged has reached a predetermined saturation level or more, other surroundings that are not saturated Based on the output from the pixel where the color filter is arranged, the output of the pixel that has reached the saturation level is predicted and interpolated, and the filter of the same color as the surrounding non-saturated specific color is arranged. Based on the output from the pixel, the output of the pixel that has reached the saturation level or higher is predicted interpolated, and appropriate dynamic interpolation processing is performed based on both of these predicted interpolation information to expand the dynamic range. The dynamic range can be expanded by a single shooting without changing the amount and shooting multiple times to compose an image. Furthermore, since the enlargement ratio of the dynamic range to be enlarged can be changed and controlled, the dynamic range can be appropriately enlarged according to the captured image.

  According to the invention described in claims 5, 6, 18 and 19, based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high luminance side to the low luminance side in the generated histogram. By changing and controlling the enlargement ratio for the enlarged dynamic range so that the position where the pixel occurrence frequency is greater than or equal to the pre-specified value is the new maximum high-brightness position of the histogram, The dynamic range can be expanded appropriately.

  According to the invention described in claims 7, 8, 20, and 21, the value of the number of luminance data when the generated histogram is counted from the low luminance side to the maximum high luminance side is the total luminance data number. Appropriately expand the dynamic range according to the captured image by changing and controlling the enlargement ratio with respect to the dynamic range so that the position with a predetermined ratio to the value becomes the new maximum high brightness position of the histogram Can do.

  According to the invention described in claims 9, 10, 22, and 23, the captured image is controlled by changing and controlling the enlargement ratio for the dynamic range to be enlarged based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed. The dynamic range can be expanded appropriately according to the situation.

  According to the invention described in claims 11, 12, 24 and 25, the predetermined coefficient is applied to the pixel output data once expanded from the first bit number to the second bit number below the predetermined saturation level. By multiplying and controlling the change of the enlargement ratio for the dynamic range to be enlarged, the dynamic range can be appropriately enlarged according to the captured image.

  According to the invention described in claims 13 and 26, the photographer selects an operation for predictive interpolation when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds a predetermined saturation level. This can be done by selecting the means.

Hereinafter, the present invention will be described based on the illustrated embodiments.
<Embodiment 1>

  FIG. 1A is a front view showing a digital still camera (hereinafter referred to as “digital camera”) as an example of an imaging apparatus according to Embodiment 1 of the present invention, and FIG. 1 (c) is a rear view thereof, and FIG. 2 is a block diagram showing an outline of a system configuration in the digital camera shown in FIGS. 1 (a), (b), and (c).

(Appearance structure of digital camera)
As shown in FIGS. 1A, 1B, and 1C, a release button (shutter button) 2, a power button 3, a shooting / playback switching dial 4 are provided on the upper surface side of the digital camera 1 according to the present embodiment. In the front (front) side of the digital camera 1, a lens barrel unit 6 having a photographing lens system 5, a strobe light emitting unit 7, and an optical finder 8 are provided.

  On the back side of the digital camera 1, there are a liquid crystal monitor (LCD) 9, an eyepiece 8 a of the optical viewfinder 8, a wide-angle zoom (W) switch 10, a telephoto zoom (T) switch 11, and a menu (MENU) button 12. , A confirmation button (OK button) 13 and the like are provided. Further, a memory card storage unit 15 for storing a memory card 14 (see FIG. 2) for storing captured image data is provided inside the side surface of the digital camera 1.

(Digital camera system configuration)
As shown in FIG. 2, in the digital camera 1, a subject image incident through a photographic lens system 5 installed in the lens barrel unit 6 is output from the CCD 20 and the CCD 20 as a solid-state imaging device that forms an image on the light receiving surface. The analog front end unit (hereinafter referred to as “AFE unit”) 21 that processes the electrical signal (analog RGB image signal) to be processed into a digital signal, the signal processing unit 22 that processes the digital signal output from the AFE unit 21, and the data There are provided an SDRAM 23 for temporary storage, a ROM 24 for storing a control program and the like, a motor driver 25 and the like.

  The lens barrel unit 6 includes a photographic lens system 5 having a zoom lens, a focus lens, and the like, an aperture unit 26, and a mechanical shutter unit 27, and each drive unit of the photographic lens system 5, the aperture unit 26, and the mechanical shutter unit 27 ( (Not shown) is driven by a motor driver 25. The motor driver 25 is driven and controlled by a drive signal from a control unit (CPU) 28 of the signal processing unit 22.

  In the CCD 20, RGB primary color filters (see FIG. 7: hereinafter referred to as “RGB filters”) as color separation filters are arranged on a plurality of pixels constituting the CCD 20, and electrical signals (analog RGB images) corresponding to the RGB three primary colors are arranged. Signal) is output.

  The AFE unit 21 is sampled by a TG (timing signal generating unit) 30 that drives the CCD 20, a CDS (correlated double sampling unit) 31 that samples an electrical signal (analog RGB image signal) output from the CCD 20, and the CDS 31. An AGC (analog gain controller) 32 that adjusts the gain of the image signal, and an A / D converter 33 that converts the image signal gain-adjusted by the AGC 32 into a digital signal (hereinafter referred to as “RAW-RGB data”) are provided. Yes.

  The signal processing unit 22 outputs a screen horizontal synchronization signal (HD) and a screen vertical synchronization signal (VD) to the TG 30 of the AFE unit 21, and an A / D conversion unit 33 of the AFE unit 21 in accordance with these synchronization signals. CCD interface (hereinafter referred to as “CCD I / F”) 34 that captures RAW-RGB data output from the CPU, a memory controller 35 that controls the SDRAM 23, and a YUV format that can display and record the captured RAW-RGB data. A YUV conversion unit 36 for converting to image data, a resizing processing unit 37 for changing the image size in accordance with the size of image data to be displayed or recorded, a display output control unit 38 for controlling display output of the image data, and an image A data compression unit 39 for recording data by JPEG formation and the like, and writing image data to the memory card 14; A digital camera based on a control program stored in the ROM 24 based on a media interface (hereinafter referred to as “media I / F”) 40 for reading image data written in the memory card 14 and operation input information from the operation unit 41. 1 is provided with a control unit (CPU) 28 for performing overall system control and the like.

  The operation unit 41 includes a release button 2, a power button 3, a photographing / playback switching dial 4, and a wide-angle zoom provided on the external surface of the digital camera 1 (see FIGS. 1A, 1 </ b> B, and 1 </ b> C). The switch 10, the telephoto zoom switch 11, the menu button 12, the confirm button 13, and the like, and a predetermined operation instruction signal is input to the control unit 28 by the photographer's operation.

  The SDRAM 23 stores the RAW-RGB data captured by the CCD I / F 34, and stores the YUV data (YUV format image data) converted by the YUV conversion unit 36. Further, the SDRAM 23 stores the RAW-RGB data. The compressed image data such as JPEG formation is stored.

  Note that YUV of the YUV data is luminance data (Y), color difference (difference (U) between luminance data and blue (B) component data, and difference (V) between luminance data and red (R) component data). It is a format that expresses color with information.

(Monitoring operation of digital camera 1, still image shooting operation)
Next, the monitoring operation and still image shooting operation of the digital camera 1 will be described. In the still image shooting mode, the digital camera 1 performs a still image shooting operation while executing a monitoring operation as described below.

  First, when the photographer turns on the power button 3 and sets the photographing / playback switching dial 4 to the photographing mode (still image photographing mode), the digital camera 1 is activated in the recording mode. When the control unit 28 detects that the power button 3 is turned on and the shooting / playback switching dial 4 is set to the shooting mode, the control unit 28 outputs a control signal to the motor driver 25 to switch the lens barrel unit 6. The camera 20 is moved to a photographing position and the CCD 20, the AFE unit 21, the signal processing unit 22, the SDRAM 23, the ROM 24, the liquid crystal monitor 9 and the like are activated.

  Then, by directing the photographic lens system 5 of the lens barrel unit 6 toward the subject, a subject image incident through the photographic lens system 5 is formed on the light receiving surface of each pixel of the CCD 20. Then, an electrical signal (analog RGB image signal) corresponding to the subject image output from the CCD 20 is input to the A / D conversion unit 33 via the CDS 31 and the AGC 32, and 12 bits (bit) by the A / D conversion unit 33. To RAW-RGB data.

  This RAW-RGB data is taken into the CCD I / F 34 of the signal processing unit 22 and stored in the SDRAM 23 via the memory controller 35. The RAW-RGB data read from the SDRAM 23 is input to the YUV conversion unit 36 and converted into YUV data (YUV signal) in a displayable format, and then the YUV data is transferred to the SDRAM 23 via the memory controller 35. Is saved.

  Then, the YUV data read from the SDRAM 23 via the memory controller 35 is sent to the liquid crystal monitor (LCD) 9 via the display output control unit 38, and a photographed image (moving image) is displayed. At the time of monitoring in which a photographed image is displayed on the liquid crystal monitor (LCD) 9 described above, one frame is read out in a time of 1/30 second by thinning out the number of pixels by the CCD I / F 34.

  During this monitoring operation, the photographed image (moving image) is only displayed on the liquid crystal monitor (LCD) 9 functioning as an electronic viewfinder, and the release button 2 has not yet been pressed (including half-pressed). It is.

  By displaying the photographed image on the liquid crystal monitor (LCD) 9, the composition for photographing a still image can be confirmed. In addition, it can output as a TV video signal from the display output control part 38, and a picked-up image (moving image) can also be displayed on external TV (television) via a video cable.

  The CCD I / F 34 of the signal processing unit 22 calculates an AF (automatic focus) evaluation value, an AE (automatic exposure) evaluation value, and an AWB (auto white balance) evaluation value from the captured RAW-RGB data.

  The AF evaluation value is calculated by, for example, the output integrated value of the high frequency component extraction filter or the integrated value of the luminance difference between adjacent pixels. When in the in-focus state, the edge portion of the subject is clear, so the high frequency component is the highest. By utilizing this, at the time of AF operation (at the time of focus detection operation), an AF evaluation value at each focus lens position in the photographing lens system 5 is acquired, and the point where the maximum is obtained is used as the focus detection position for AF. The action is executed.

  The AE evaluation value and the AWB evaluation value are calculated from the integrated values of the RGB values in the RAW-RGB data. For example, the screen corresponding to the light receiving surfaces of all the pixels of the CCD 20 is equally divided into 256 areas (16 horizontal divisions and 16 vertical divisions), and the RGB integration of each area is calculated.

  Then, the control unit 28 reads the calculated RGB integrated value, and in the AE process, calculates the luminance of each area of the screen and determines an appropriate exposure amount from the luminance distribution. Based on the determined exposure amount, exposure conditions (the number of electronic shutters of the CCD 20, the aperture value of the aperture unit 26, etc.) are set. In the AWB process, an AWB control value that matches the color of the light source of the subject is determined from the RGB distribution. By this AWB process, white balance is adjusted when the YUV conversion unit 36 performs conversion to YUV data. The AE process and AWB process described above are continuously performed during the monitoring.

  When the still image shooting operation in which the release button 2 is pressed (half-pressed to full-press) is started during the monitoring operation described above, the AF operation and the still image recording process that are the focus position detection operations are performed. .

  That is, when the release button 2 is pressed (half-pressed to full-pressed), the focus lens of the photographing lens system 5 is moved by a drive command from the control unit 28 to the motor driver 25, and is referred to as so-called hill-climbing AF, for example. The contrast evaluation AF operation is executed.

  When the AF (focusing) target range is the entire region from infinity to close, the focus lens of the photographing lens system 5 moves to each focus position from close to infinity, or from infinity to close, and CCDI / The control unit 28 reads out the AF evaluation value at each focus position calculated in F34. Then, the focus lens is moved to the in-focus position with the point where the AF evaluation value at each focus position is maximized as the in-focus position, and in-focus.

  Then, the AE process described above is performed, and when the exposure is completed, the mechanical shutter unit 27 is closed by a drive command from the control unit 28 to the motor driver 25, and an analog RGB image signal for a still image is output from the CCD 20. As in the monitoring, the A / D conversion unit 33 of the AFE unit 21 converts the data into RAW-RGB data.

  The RAW-RGB data is taken into the CCD I / F 34 of the signal processing unit 22, converted into YUV data by the YUV conversion unit 36, and stored in the SDRAM 23 via the memory controller 35. The YUV data is read from the SDRAM 23, converted into a size corresponding to the number of recorded pixels by the resizing processing unit 37, and compressed into image data such as JPEG format by the data compression unit 39. The compressed image data such as JPEG format is written back to the SDRAM 23, read out from the SDRAM 23 through the memory controller 35, and stored in the memory card 14 through the media I / F 40.

(Principle of dynamic range expansion in the present invention)
An RGB filter is arranged on each pixel constituting the CCD 20 of the digital camera 1, but each pixel on which a normal RGB filter is arranged for light having a wide wavelength band such as sunlight. The sensitivity to brightness differs for each color.

  For example, as shown in FIG. 3, the luminance sensitivity of a pixel in which a G (green) filter is arranged has RGB luminance sensitivity about twice that of a pixel in which an R (red) filter and a B (blue) filter are arranged. In the case of the CCD 20 in which the filters (a, b, and c in FIG. 3) are arranged on each pixel, when light having a wide wavelength band such as sunlight is incident on the pixels having the same RGB filter, The output of the pixel in which the G filter (shaded portion in FIG. 3c) is arranged for the output of the pixel in which the R and B filters are arranged reaches the saturation level A first. In FIG. 3, f is the luminance sensitivity characteristic of the pixel in which the G filter is arranged, g is each luminance sensitivity characteristic of the pixel in which the R and B filters are arranged, and the luminance sensitivity characteristic of the pixel in which the G filter is arranged. Has a sensitivity about twice as high as each luminance sensitivity characteristic of the pixel in which the R and B filters are arranged.

  By the way, in a digital camera equipped with a solid-state imaging device such as a CCD having such a conventional RGB filter, a highly sensitive G filter such as a CCD in which the RGB filter c in FIG. The range of the dynamic range is set according to the saturation level A corresponding to the output (luminance sensitivity) of the arranged pixels. That is, the R and B filters are arranged even when the output of the pixel in which the G filter is arranged reaches the saturation level A as in the CCD in which the RGB filter in FIG. 3c is arranged on each pixel. The output (luminance sensitivity) of the pixel is about ½ of the saturation level A.

  On the other hand, in the present invention, even if the output of the pixel in which the G filter is disposed exceeds the saturation level A, such as a CCD in which the RGB filters of d and e in FIG. When each output of the pixel in which the R and B filters are arranged is in a range not exceeding the saturation level A, the R and B filters are arranged from each output level of the pixel in which the R and B filters are arranged. Predicting the output level of the pixel with the G filter based on each luminance sensitivity characteristic of the pixel (g in FIG. 3) and the luminance sensitivity characteristic of the pixel with the G filter (f in FIG. 3) (one-dot chain line) And the dynamic range is expanded by the amount of the predicted interpolation.

  As described above, in the present embodiment, the luminance sensitivity characteristics of the pixel in which the G filter is arranged with respect to light having a wide wavelength band such as sunlight are the luminance sensitivities of the pixels in which the R and B filters are arranged. The sensitivity is about twice as high as the characteristic. Therefore, the maximum value of the degree of expansion of the dynamic range in the present embodiment is about twice that during normal shooting without performing the dynamic range expansion processing operation.

  In the present embodiment, the luminance sensitivity characteristic of the pixel in which the G filter is arranged has a sensitivity that is about twice that of each luminance sensitivity characteristic of the pixel in which the B filter is arranged. Although the maximum value of the degree of enlargement is doubled, the maximum value of the degree of enlargement of the dynamic range is set to a predetermined value that is twice or more by changing each luminance sensitivity characteristic of the pixel in which the R, G, and B filters are arranged. Or it can set to the predetermined value of 2 times or less.

(Dynamic range expansion processing by the YUV converter 36)
The YUV conversion unit 36 of the digital camera 1 according to the present embodiment has a dynamic range expansion processing function for expanding the dynamic range described above.

  As shown in FIG. 4, the YUV conversion unit 36 includes a dynamic range expansion prediction interpolation unit (hereinafter referred to as “D range expansion prediction interpolation unit”) 50, a bit compression conversion unit 51, a white balance control unit 52, and synchronization described later. Unit 53, tone curve conversion unit 54, RGB-YUV conversion unit 55, image size converter unit 56, luminance histogram generation unit 57, and edge enhancement unit 58.

  As illustrated in FIG. 5, the D range expansion prediction interpolation unit 50 includes a luminance level determination unit 60, a pixel output correction processing unit 61, and a bit expansion processing unit 62.

  The luminance level determination unit 60 detects the pixel output of each pixel provided with the RGB filter from the input RAW-RGB data, and also outputs the pixel output of the pixel provided with the highest sensitivity G filter (hereinafter referred to as “G filter”). It is determined whether or not the “pixel output of” has reached a saturation level or higher.

  When the luminance level determination unit 60 determines that the pixel output of the G filter within the processing unit described later has reached the saturation level or higher, the pixel output correction processing unit 61 has a surrounding area (the pixel output has reached the saturation level or higher). The pixel output of the G filter that has reached the saturation level or higher is predicted based on the pixel output of the pixel provided with the R and B filters (hereinafter referred to as “R and B filter pixel outputs”). The dynamic range is expanded by interpolation (details will be described later).

  When the luminance level determination unit 60 determines that the pixel output of the G filter has not reached the saturation level, the bit extension processing unit 62 performs the pixel output of the G filter and the pixel output of the R and B filters. Only bit expansion is performed from 12 bits to 14 bits without converting the output level.

  The dynamic range expansion processing operation in this embodiment will be described below.

  For example, when there is an extremely bright part of the background of the subject to be photographed and the photographer wants to expand the dynamic range, the photographer presses the menu button 12 (see FIG. 1C), for example, A shooting setting screen as shown in FIG. 6A is displayed on the liquid crystal monitor (LCD) 9.

  Then, as shown in FIG. 6A, the “dynamic range 2” item is pressed by pressing the confirm button 13 (see FIG. 1C) for the item “double dynamic range” selected by pressing the menu button 12. Double "is determined. As a result, the dynamic range expansion processing operation is turned ON, and a control signal for doubling the dynamic range expansion rate is output from the control unit 28 to the luminance level determination unit 60. Thus, in the present embodiment, the dynamic range expansion rate setting means and the operation selection means in “Claims” correspond to the menu button 12, and the dynamic range expansion rate change control means in “Claims”. Corresponds to the control unit 28.

  Note that the dynamic range expansion ratio in the present embodiment can be arbitrarily set in a range of 1.1 to 2 times, with the above-mentioned double being the maximum value. In the present embodiment, the expansion rate of the dynamic range is set to be changeable to, for example, 1.3 times, 1.6 times, and 2 times by pressing the menu button 12. When setting the enlargement ratio of the dynamic range to 1.3 times, as shown in FIG. 6B, the “dynamic range” is displayed from the shooting setting screen displayed on the liquid crystal monitor (LCD) 9 by pressing the menu button 12. The item “1.3 times” is selected, and the confirmation button 13 is pressed to determine “dynamic range 1.3 times”.

  Similarly, when the enlargement ratio of the dynamic range is set to 1.6 times, as shown in FIG. 6C, from the shooting setting screen displayed on the liquid crystal monitor (LCD) 9 by pressing the menu button 12. Select the item “dynamic range 1.6 times” and press the confirm button 13 to determine “dynamic range 1.6 times”.

  When a control signal for doubling the dynamic range expansion rate is output from the control unit 28 to the luminance level determination unit 60 and the dynamic range expansion processing operation is executed, the luminance level of the D range expansion prediction interpolation unit 50 is The determination unit 60 determines whether the pixel output of the G filter has reached a saturation level or higher based on the input RAW-RGB data. In the case where this determination process is performed, in the present embodiment, as shown in FIG. 7, 2 × 2 pixels (two G filter pixels, 1 G pixels) in a thick frame A are provided for each pixel of the CCD 20 having an RGB filter. Each R and B filter pixel) is defined as a processing unit (minimum unit).

  When at least one pixel output of two G filter pixels in the processing unit (thick frame A) has reached the saturation level or higher, the sensitivity of the G filter is the R and B filters as described above. Therefore, the pixel output value (G) of the G filter is calculated from the following formula (1).

  G = {(R + B) / 2} × 2 Formula (1)

  The pixel output correction processing unit 61 calculates the average value of the pixel outputs of the R and B filters and doubles the average value of the pixel outputs of the R and B filters as shown in the equation (1) to calculate the pixel output value of the G filter. The calculated pixel output values of the G filter are replaced with the pixel output values of the two G filters in the processing unit (2 × 2 pixels). Since the pixel output value of the G filter exceeds 12 bits, it is replaced here with 14-bit data. Therefore, since the maximum value of the pixel output of the R and B filters is 4095 (12 bits), the maximum value of the pixel output of the G filter is 8190 (13 bits), and can be handled as 14-bit data.

  Note that correction of defective pixels needs to be completed before the luminance level determination unit 60 determines whether or not the pixel output of the G filter has reached the saturation level or higher. That is, if there is a defective pixel in the pixel provided with the G filter and there is a pixel that always outputs a saturated value, the pixel provided with the G filter in the same processing unit is replaced with a large value. A defective pixel will be generated.

  Further, when a pixel provided with the R and B filters has a defective pixel, the conversion of the pixel provided with the G filter according to the equation (1) becomes an incorrect value. Therefore, in the present embodiment, the CCD I / F 34 includes a defective pixel removal processing unit (not shown) that removes defective pixels.

  Then, the pixel output data of the R and B filters from the pixel output correction processing unit 61 of the D range expansion prediction interpolation unit 50 and the pixel output data of the G filter that has undergone the prediction interpolation processing after reaching the saturation level or higher are converted into the bit compression conversion unit. 51 is output. The bit compression conversion unit 51 is expanded to 14 bits by, for example, a bit compression conversion characteristic (four segment broken line approximation characteristic that designates three nodes and approximates them with a straight line) as shown in FIG. Among the pixel outputs of the R, G, and B filters, the G filter pixel output is compressed to 12 bits. In FIG. 8, a is a 12-bit range (dotted line portion).

  In the bit compression conversion characteristics shown in FIG. 8, since the maximum value of the pixel output of the G filter is 8190, compression is performed so that 8190 becomes 4095. Then, the pixel output values of the R and B filters are also compressed in accordance with the bit compression conversion characteristics of the pixel output of the G filter.

  The pixel output data of the R, G, and B filters that have been compression-converted from 14 bits to 12 bits by the bit compression conversion unit 51 are input to the white balance control unit 52. The white balance control unit 52 amplifies input pixel output data of the R, G, and B filters. At this time, the control unit 28 calculates a correction value for adjusting the white balance based on the AWB evaluation value calculated by the CCD I / F 34 and outputs the calculated correction value to the white balance control unit 52. The white balance control unit 52 adjusts the white balance based on the input correction value.

  Then, the pixel output data (12 bits) of the R, G, and B filters with the white balance adjusted from the white balance control unit 52 is input to the synchronization unit 53. The synchronizer 53 performs interpolation calculation processing on RAW data having only one color data per pixel, such as a Bayer array, and generates all RGB data for one pixel.

  All the RGB data (12 bits) generated by the synchronization unit 53 is input to the tone curve conversion unit 54. The tone curve conversion unit 54 performs γ conversion for converting 12-bit RGB data into 8-bit RGB data using a conversion table as shown in FIG. 9 to generate an 8-bit RGB value, and RGB-YUV The data is output to the conversion unit 55.

  The RGB-YUV conversion unit 55 converts input RGB data (8 bits) into YUV data by matrix calculation and outputs the YUV data to the image size converter unit 56. The image size converter unit 56 reduces or enlarges the input YUV data (8 bits) to a desired image size, and outputs it to the luminance histogram generation unit 57 and the edge enhancement unit 58.

  The luminance histogram generation unit 57 generates a luminance histogram based on the input YUV data. The edge enhancement unit 58 performs processing such as edge enhancement according to the image on the input YUV data, and stores the processed data in the SDRAM 23 via the memory controller 35.

  As described above, the digital camera 1 according to the present embodiment has a high-luminance portion in the background where the pixel output of the G filter having a high sensitivity within the processing unit exceeds the saturation level (the background in the captured image has a high luminance portion). ) By pressing the menu button 12 and selecting and determining, for example, the item “double dynamic range” from the shooting setting screen, so that the pixel output of the R and B filters having lower sensitivity than the G filter is obtained. Based on this, the pixel output of the saturated G filter is subjected to predictive interpolation processing. As a result, as shown in FIG. 3, the pixel output of the G filter (d and e in FIG. 3) is expanded based on the predicted interpolation (the dashed line portion of the pixel output of the G filter in d and e in FIG. 3). The dynamic range can be doubled by one shooting.

  Therefore, even when there is a high-luminance portion in the background or the like in the captured image, it is possible to prevent the occurrence of overexposure and obtain good gradation.

  Also, FIG. 10A is generated by the luminance histogram generation unit 57 when the dynamic range is doubled as described above for a captured image in which the pixel output of the G filter exceeds the saturation level. It is an example of the done histogram. FIG. 10B illustrates a case where the luminance histogram generation unit 57 generates the captured image in which the pixel output of the G filter exceeds the saturation level when the dynamic range expansion process in the present embodiment is not performed. It is an example of the done histogram. In FIGS. 10A and 10B, the horizontal axis represents luminance (256 gradations (8 bits) from 0 to 255), and the vertical axis represents pixel occurrence frequency (0 to 1 (= 100%)). .

  As is apparent from the histogram shown in FIG. 10A, when the dynamic range expansion process is performed in this embodiment, almost no whiteout occurs near the maximum luminance portion (255). On the other hand, as is apparent from the histogram shown in FIG. 10B, when the dynamic range expansion process in this embodiment is not performed, pixels are generated near the maximum luminance portion (255), and white You can see that the jump has occurred.

  By the way, in the histogram shown in FIG. 10A when the above-described dynamic range is doubled, there is a region where there are almost no pixels in the vicinity of the maximum luminance portion (255). This is due to the fact that the dynamic range has been expanded more than necessary even though no image data originally exists in the vicinity of the maximum luminance portion, and all gradations (0 which can be expressed by 8 bits of RGB). (256 gradations of ~ 255) cannot be used effectively.

  Therefore, in the present invention, the dynamic range is used so that the range of all gradations that can be expressed by 8 bits of RGB can be effectively used for a captured image in which the pixel output of the G filter exceeds the saturation level. For example, the enlargement ratio can be changed to 1.3 times and 1.6 times in addition to 2 times as described above.

  That is, when there is an extremely bright part of the background of a subject to be photographed, and when it is desired to expand the dynamic range by effectively using the range of all gradations that can be expressed by 8 bits of RGB, for example, FIG. As shown in FIG. 6B, the item “dynamic range 1.3 times” is selected from the shooting setting screen displayed on the liquid crystal monitor 9 by pressing the menu button 12, and the confirmation button 13 is pressed to select “dynamic range 1”. .3 times ".

  As a result, the dynamic range expansion processing operation is turned ON, and a control signal for increasing the dynamic range expansion ratio to 1.3 times is output from the control unit 28 to the pixel output correction processing unit 61. Then, the pixel output correction processing unit 61 executes a processing operation for expanding the dynamic range by 1.3 times in the same manner as described above, in accordance with the dynamic range expansion rate set based on the input control signal.

  When the dynamic range is expanded to 1.3 times, the pixel output value of the G filter, which has been replaced by the pixel output correction processing unit 61 from 12 bits to 14 bits, is compressed by the bit compression conversion unit 51 to 12 bits. For example, a bit compression conversion characteristic (a three-line broken line approximation characteristic in which two nodes are designated and approximated by a straight line) as shown in FIG. 11 is used. In FIG. 11, a is a 12-bit range (dotted line portion).

  In the bit compression conversion characteristic shown in FIG. 11, when the input 14-bit data (pixel output value of the G filter) reaches 5461, conversion is performed so that the maximum value of output 12-bit data is 4095. The pixel output values of the R and B filters are also compressed in accordance with the compression characteristics of the pixel output of the G filter.

  FIG. 12 is an example of a histogram generated by the luminance histogram generation unit 57 when the dynamic range is expanded by 1.3 times for a certain captured image in which the pixel output of the G filter exceeds the saturation level. . In FIG. 12, the horizontal axis represents luminance (256 gradations (8 bits) from 0 to 255), and the vertical axis represents pixel occurrence frequency (0 to 1 (= 100%)).

  During the above-described double processing of the dynamic range, there was a region with almost no pixels in the vicinity of the maximum luminance portion (255). As is clear from the histogram shown in FIG. 12, the dynamic range was expanded to 1.3 times. In the case of processing, it was possible to effectively use the range of all gradations that can be expressed by 8 bits of RGB, and to suppress the occurrence of whiteout in the maximum luminance part (255).

  Note that when the dynamic range is expanded to 1.3 times, the pixel output value of the G filter replaced with 14 bits is compressed and converted to 12 bits based on the bit compression conversion characteristics shown in FIG. For example, the G filter pixel output value (G ′) with the upper limit set is calculated from the following equation (2) with respect to the G filter pixel output value (G) calculated by the above equation (1). The same compression conversion can be performed in the process.

  G ′ = G × (2 / 1.3) Equation (2)

  When the upper limit is set based on Expression (2), if the input 14-bit data (G filter pixel output value) is 5461 or more, the maximum value of the G filter pixel output exceeds 8190. Therefore, the bits shown in FIG. When the compression conversion characteristic is used, the output 12-bit data is 4091.

  In other words, this is equivalent to the upper limit process being performed at 5460 for input 14-bit data. The upper limit processing result is compressed to 12 bits by the bit compression conversion characteristic shown in FIG. Thus, by performing the process of calculating the pixel output value (G ′) of the G filter for which the upper limit is set by the above equation (2), the bit compression conversion table is shown in FIG. 8 regardless of the upper limit value. Bit compression conversion characteristics can be used.

  Similarly, when the dynamic range is enlarged by 1.6 times, as shown in FIG. 6C, the “dynamic range” is displayed from the shooting setting screen displayed on the liquid crystal monitor 9 by pressing the menu button 12. The item “1.6 times” is selected, and the confirmation button 13 is pressed to determine “dynamic range 1.6 times”.

  As a result, the dynamic range expansion processing operation is turned ON, and a control signal for increasing the dynamic range expansion ratio by 1.6 is output from the control unit 28 to the pixel output correction processing unit 61. Then, the pixel output correction processing unit 61 executes a processing operation for expanding the dynamic range by 1.6 times as described above based on the input control signal.

  When the dynamic range is enlarged by 1.6 times, the pixel output value of the G filter, which has been replaced by the pixel output correction processing unit 61 from 12 bits to 14 bits, is compressed by the bit compression conversion unit 51 to 12 bits. For example, a bit compression conversion characteristic (a three-line broken line approximation characteristic that designates two nodes and approximates them with a straight line) as shown in FIG. 13 is used. In FIG. 13, a is a 12-bit range (dotted line portion).

  In the bit compression conversion characteristic shown in FIG. 13, when the input 14-bit data (pixel output value of the G filter) reaches 6826, conversion is performed so that the maximum value of output 12-bit data is 4095. The pixel output values of the R and B filters are also compressed in accordance with the compression characteristics of the pixel output of the G filter.

  FIG. 14 is an example of a histogram generated by the luminance histogram generation unit 57 when the dynamic range is enlarged by 1.6 times for a certain captured image in which the pixel output of the G filter exceeds the saturation level. . In FIG. 14, the horizontal axis represents luminance (256 gradations (8 bits) from 0 to 255), and the vertical axis represents pixel occurrence frequency (0 to 1 (= 100%)).

  Although there was a region with almost no pixels in the vicinity of the maximum luminance portion (255) at the time of the above double processing of the dynamic range, the dynamic range was expanded to 1.6 times as apparent from the histogram shown in FIG. In the case of processing, the range of all gradations that can be expressed by 8 bits of RGB can be used effectively, and the occurrence of whiteout in the vicinity of the maximum luminance portion (255) can be suppressed.

  Note that when the dynamic range is enlarged 1.6 times, the pixel output value of the G filter replaced with 14 bits is compressed and converted to 12 bits based on the bit compression conversion characteristics shown in FIG. For example, the G filter pixel output value (G ′) with an upper limit set is calculated from the following equation (3) with respect to the G filter pixel output value (G) calculated by the equation (1). The same compression conversion can be performed in the process.

  G ′ = G × (2 / 1.6) Equation (3)

  When the upper limit is set based on Expression (3), when the input 14-bit data (pixel output value of the G filter) is 6826 or more, the maximum value of the pixel output of the G filter exceeds 8192. Therefore, the bits shown in FIG. When the compression conversion characteristic is used, the output 12-bit data is 4091.

  In other words, this is equivalent to performing upper limit processing on input 6-bit data at 6826. The upper limit processing result is compressed to 12 bits by the bit compression conversion characteristic shown in FIG. Thus, by performing the process of calculating the pixel output value (G ′) of the G filter for which the upper limit is set by the above equation (3), the bit compression conversion table is shown in FIG. 8 regardless of the upper limit value. Bit compression conversion characteristics can be used.

  In the present embodiment, for example, when the values of the pixel output of the G filter and the pixel output of the R filter reach a saturation level or higher, the value of the pixel output of the G filter subjected to the prediction interpolation calculated from the equation (1) is used. May become inaccurate, and since the bit compression is performed at the compression rate used in the pixel output of the G filter without predictive interpolation of the pixel output value of the R filter, the hue may change.

Therefore, when the values of the pixel output of the G filter and the pixel output of the R filter (or B filter) have reached at least the saturation level, the dynamic range expansion processing by the predictive interpolation is not performed. Is preferred. Alternatively, the fact that the pixel output values of a plurality of pixels (G filter pixel output and R filter (or B filter)) have reached a saturation level or higher assumes that the brightness of the processing unit area is extremely bright, A predetermined value for the pixel output value of the G filter, for example,
G filter pixel output = 4096 x 1.8 = 7372 (14 bits)
You may set it.

  In this embodiment, as shown in FIG. 4, 14-bit RAW-RGB data (pixel output data of R, G, and B filters) output from the D-range expansion prediction interpolation unit 50 is converted into a bit compression conversion unit. 51, the white balance control unit 52 and the synchronization unit 53 perform 12-bit data processing. However, in addition to this, the bit compression conversion unit 51 is provided after the synchronization unit 53. The 14-bit data output from the synchronization unit 53 may be compressed into 12-bit data.

<Embodiment 2>
In the first embodiment, as shown in FIG. 7, 2 × 2 pixels are used as a processing unit (minimum unit) for the CCD 20 having the RGB primary color filter. In the present embodiment, as shown in FIG. 15, 5 pixels in the thick frame A (one G filter pixel, two R (R1, R2) and B (B1, B2) filter pixels) are set as processing units (minimum units), and the processing unit is the above-described implementation. This is an example in which the range is wider than that of the first embodiment. The configuration of the digital camera, the monitoring operation, the still image shooting operation, and the dynamic range expansion processing operation are the same as those in the first embodiment.

  When the pixel output of the G filter in the processing unit of the thick frame A shown in FIG. 15 has reached the saturation level or higher, the sensitivity of the G filter is about twice the sensitivity of the R and B filters as described above. Therefore, the pixel output value of the G filter is calculated from the following equation (4).

  G = [{(R1 + R2) / 2 + (B1 + B2) / 2} / 2] × 2 Formula (4)

  Then, the pixel output correction processing unit 61 of the D range expansion prediction interpolation unit 50 shown in FIG. 5 sets the pixel output value of the G filter calculated by the equation (4) in the processing unit (see FIG. 15). The pixel output value of a certain G filter is replaced, and the same processing as in the first embodiment is performed.

  In this way, by widening the processing unit, it is possible to reduce the influence due to the sensitivity difference of the other R1, R2 filter pixels and B1, B2 filter pixels in the processing unit. More accurate dynamic range expansion prediction interpolation can be performed on the output.

<Embodiment 3>
In the present embodiment, as shown in FIG. 16, the processing unit (thick frame A) is wider than that in the second embodiment with respect to the CCD 20 having the RGB filter. The configuration of the digital camera, the monitoring operation, the still image shooting operation, and the dynamic range expansion processing operation are the same as those in the first embodiment.

  If the processing unit is widened, processing is performed based on a wide range of luminance information, which is equivalent to applying a low-pass filter. Therefore, the edge portion of the luminance change is lost. Therefore, in the present embodiment, the size of the widened processing unit is partially changed using, for example, the AF evaluation value.

  That is, the CCD I / F 34 of the signal processing unit 22 shown in FIG. 2 calculates an AF evaluation value for performing AF as described above. This is an output of a high-pass filter (HPF), and a large value is output in a portion where there is a change in luminance in the screen of the captured image. Then, the control unit 28 reads out an AF evaluation value at the time of still image shooting, and discriminates between a portion having a luminance change and a portion having no luminance change in the screen. Then, the control unit 28 sets the D range expansion prediction interpolation unit 50 based on this discrimination data so that the processing unit is narrowed in the portion where the luminance change occurs, and the portion where the luminance change does not exist. As shown in FIG. 16, setting is performed so that the processing unit is in a wide range.

  Thus, even when the processing unit is further widened, accurate dynamic range expansion prediction interpolation can be performed without reducing the resolution by setting the processing unit to be narrower in a portion where there is a luminance change. .

<Embodiment 4>
In the present embodiment, as shown in FIG. 17, for the CCD 20 having an RGB filter, 5 × 5 pixels of 5 pixels in the horizontal direction and 5 pixels in the vertical direction in the thick frame A are used as processing units, and the center of this processing unit is set. A plurality of G filter pixels are also arranged in the diagonal direction (a direction and b direction in FIG. 17) with respect to the G filter in a part. It is determined whether or not the pixel output of the G filter at the center of the processing unit (inside the thick frame A) has reached the saturation level or higher.

  In addition, the D range expansion prediction interpolation unit 50 of the YUV conversion unit 36 in this embodiment includes a luminance level determination unit 60, a pixel output correction processing unit 61a, and a bit expansion processing unit 62, as shown in FIG. . The other configuration of the digital camera, the monitoring operation, and the still image shooting operation are the same as those in the first embodiment.

  The luminance level determination unit 60 determines whether or not the pixel output of the G filter at the center of the input processing unit (see FIG. 17: thick frame A) has reached or exceeded the saturation level. When the luminance level determination unit 60 determines that the pixel output of the central G filter has reached the saturation level or higher, the pixel output correction processing unit 61a outputs the central G filter output from the luminance level determination unit 60. A central portion that reaches a saturation level or higher due to pixel outputs of a plurality of other G filters arranged in a diagonal direction (a direction and b direction in FIG. 17) of the processing unit (thick frame A) with respect to the pixel output. Predictive interpolation processing of the pixel output of the G filter is performed.

  When the luminance level determination unit 60 determines that the pixel output of the G filter has not reached the saturation level, the bit extension processing unit 62 performs the pixel output of the G filter and the pixel output of the R and B filters. Only bit expansion is performed from 12 bits to 14 bits without converting the output level.

  In the present embodiment, during the dynamic range expansion process, the luminance level determination unit 60 of the D range expansion prediction interpolation unit 50 outputs the pixel output of the G filter at the center of the processing unit (thick frame A) shown in FIG. When it is determined that the saturation level has been reached, the pixel output correction processing unit 61a performs the pixel output of the central G filter output from the luminance level determination unit 60 with reference to FIGS. 19 (a) and 19 (b). As shown, the pixel output of the saturated G filter is predicted from the distribution of the pixel output values of the plurality of G filters arranged in the respective diagonal directions (direction a and b in FIG. 17). I tried to do it.

  FIG. 19A shows the pixel output value of each G filter on the diagonal line in the a direction. From these output values, the predicted value a1 of the pixel output of the central G filter is about 6000 (14 bits). Can be predicted. FIG. 19B shows the pixel output value of each G filter on the diagonal line in the b direction. From these output values, the predicted value b1 of the pixel output of the central G filter is about 5000 (14 bits). Can be predicted. In FIGS. 19A and 19B, 4095 (12 bits) is the saturation level of the pixel output of the central G filter.

  From these prediction results, it can be predicted that the pixel output value of the G filter that has reached the saturation level or more in the center of the processing unit (see FIG. 17: thick frame A) is about 5000 to 6000. When determining the value of the pixel output of the G filter, for example, the average of the two predicted values is 5500.

  As described above, when the pixel output of the central G filter within the processing unit reaches the saturation level or higher, the pixel output of the G filter saturated from the distribution of the pixel output values of the other G filters in the processing unit. Predicting the value enables accurate dynamic range expansion prediction interpolation.

  In the present embodiment, when predicting the pixel output value of the G filter that has reached the saturation level or higher, prediction is performed from the pixel output values of a plurality of G filters arranged in the diagonal direction within the processing unit. In addition, the pixel output of the G filter that has reached the saturation level or more is considered in consideration of the values of the pixel outputs of the plurality of G filters arranged in the horizontal direction and the vertical direction in the processing unit. May be predicted. This enables more accurate dynamic range expansion prediction interpolation.

  In the present embodiment, only the G filter pixels in the processing unit have been described, but the R and B filter pixels can be processed in the same manner.

<Embodiment 5>
As shown in FIG. 20, the D range expansion prediction interpolation unit 50 of the YUV conversion unit 36 in this embodiment includes a luminance level determination unit 60, a bit expansion processing unit 62, a first pixel output correction processing unit 63, and a second pixel output. A correction processing unit 64 and a correction data combining unit 65 are provided. The brightness level determination unit 60 and the bit expansion processing unit 62 are the same as those in the first embodiment shown in FIG. The other configuration of the digital camera, the monitoring operation, and the still image shooting operation are the same as those in the first embodiment.

  The first pixel output correction processing unit 63 (corresponding to the “pixel output correction processing unit 61” in the first embodiment shown in FIG. 5) is the luminance level determination unit 60, and the pixel output of the G filter in the processing unit is equal to or higher than the saturation level. In the same way as in the first embodiment (including the second and third embodiments), the G filter that has reached the saturation level or higher from the pixel output of the R and B filters in the processing unit is determined. Predictive interpolation of pixel output.

  The second pixel output correction processing unit 64 (corresponding to the “pixel output correction processing unit 61a” in the fourth embodiment shown in FIG. 18) has a luminance level determination unit 60 in which the pixel output of the G filter in the processing unit is equal to or higher than the saturation level. In the same way as in the fourth embodiment, the pixel output of the G filter that has reached the saturation level or higher is determined from the distribution of the pixel output values of the other G filters in the processing unit. Predictive interpolation.

  The correction data synthesizing unit 65 performs appropriate predictive interpolation processing based on the following processing conditions (first to fourth processing conditions).

(First processing condition)
Under the first processing condition, the prediction interpolation information output from the first and second pixel output correction processing units 63 and 64 is averaged.

  That is, when each prediction interpolation information for the G filter pixel output is input from the first and second pixel output correction processing units 63 and 64, the correction data synthesis unit 65 averages both prediction interpolation information and calculates the average. Based on the predicted interpolation information, the pixel output of the G filter that has reached the saturation level or higher is predicted interpolated.

(Second processing condition)
In the second processing condition, the predicted interpolation value information on the first pixel output correction processing unit 63 side is given priority over the predicted interpolation value information output from the first and second pixel output correction processing units 63 and 64, respectively. .

  For example, when the main subject is in the green background area, the pixel outputs of the R and B filters are very low values. In the green background region, the pixel output of the G filter has reached the saturation level or higher, and the pixel output of the pixel output when the first pixel output correction processing unit 63 performs the predictive interpolation on the pixel output of the G filter. The predicted value may be lower than the pixel output value of the G filter that has reached the original saturation level or higher.

  Then, the correction data combining unit 65 predicts the interpolation value for the pixel output of the G filter that has reached the saturation level from the first pixel output correction processing unit 63 and the value used for the determination of the saturation level by the luminance level determination unit 60. When the predicted interpolation value from the first pixel output correction processing unit 63 is higher than the saturation level, the predicted interpolation value from the first pixel output correction processing unit 63 is used as the final prediction. When the predicted interpolation value from the first pixel output correction processing unit 63 is lower than the saturation level, the predicted interpolation value from the second pixel output correction processing unit 64 is used as the final prediction value. Adopted as an interpolation value.

(Third processing condition)
In the third processing condition, the prediction interpolation value information on the second pixel output correction processing unit 64 side is given priority over the prediction interpolation value information output from the first and second pixel output correction processing units 63 and 64, respectively. .

  When the pixel output of the G filter in the processing unit has reached the saturation level, the pixel output of the surrounding G filter may have reached the saturation level. Therefore, as described above, when the luminance level determination unit 60 determines that the pixel output of the G filter in the processing unit has reached the saturation level or more, the second pixel output correction processing unit 64 performs other processing in the processing unit. From the distribution of the pixel output values of the plurality of G filters, the pixel output of the G filter that has reached the saturation level or higher is predicted interpolated.

  At this time, the second pixel output correction processing unit 64 can correctly predict and interpolate the pixel output of the G filter that has reached the saturation level or higher from the distribution of the pixel output values of the plurality of other G filters in the processing unit. If not, the output value from the second pixel output correction processing unit 64 is set to “0”.

  Then, the correction data synthesis unit 65 uses the predicted interpolation value for the pixel output of the G filter that has reached the saturation level from the second pixel output correction processing unit 64 and the value used for the determination of the saturation level by the luminance level determination unit 60. When the predicted interpolation value from the second pixel output correction processing unit 64 is smaller than the saturation level, the predicted interpolation value from the first pixel output correction processing unit 63 is used as the final prediction. When the predicted interpolation value from the second pixel output correction processing unit 64 is adopted as the interpolation value and is higher than the saturation level, the predicted interpolation value from the second pixel output correction processing unit 64 is used as the final prediction. Adopted as an interpolation value.

(Fourth processing condition)
In the fourth processing condition, the first to third processing conditions are appropriately selected according to the predicted interpolation value information output from the first and second pixel output correction processing units 63 and 64, respectively.

  That is, the correction data combining unit 65 receives the first interpolation output value information of the pixel output of the G filter that has reached the saturation level from the first and second pixel output correction processing units 63 and 64, and One processing condition is selected, both prediction interpolation values are weighted and averaged, and the averaged prediction interpolation value is adopted as the final prediction interpolation value.

  When the output from the first pixel output correction processing unit 63 is higher than the saturation level, the correction data combining unit 65 selects the second processing condition and selects the first pixel output correction processing unit. The predicted interpolation value from 63 is adopted as the final predicted interpolation value. When the output from the second pixel output correction processing unit 64 is higher than the saturation level, the correction data combining unit 65 selects the third processing condition and selects the second pixel output correction processing unit. The predicted interpolation value from 64 is adopted as the final predicted interpolation value.

  As described above, the correction data combining unit 65 outputs the pixel output of the G filter that has reached the saturation level or higher according to the prediction interpolation value information output from the first and second pixel output correction processing units 63 and 64, respectively. Since the value can be appropriately predicted and corrected, accurate dynamic range expansion prediction interpolation can be performed.

<Embodiment 6>
In the digital camera according to this embodiment, an appropriate dynamic range enlargement ratio can be automatically set by the control unit 28 (see FIG. 2). The dynamic range expansion rate setting process according to this embodiment will be described below.

  In the dynamic range enlargement ratio setting process in the present embodiment, the dynamic range enlargement ratio is first set in the same manner as in the first embodiment (or any one of the second, third, fourth, and fifth embodiments). Set to 2 times and perform dynamic range expansion. FIG. 21A is generated by the luminance histogram generation unit 57 (see FIG. 4) when the dynamic range is doubled for a certain captured image in which the pixel output of the G filter exceeds the saturation level. It is an example of a histogram. In FIG. 21A, the horizontal axis represents luminance (256 gradations (8 bits) from 0 to 255), and the vertical axis represents pixel occurrence frequency (0 to 1 (= 100%)).

  And the control part 28 (refer FIG. 2) of the signal processing part 22 is the maximum high brightness | luminance with respect to the histogram (refer FIG. 21 (a)) produced | generated based on the histogram data input from the brightness histogram production | generation part 57. FIG. (255) Count the frequency of occurrence of pixels from the low luminance side to the low luminance side. Then, the control unit 28 sets the position (point) where the occurrence frequency of the pixel in the vicinity of the counted maximum high luminance (255) is equal to or higher than a predetermined value, the new maximum luminance value after the dynamic range is expanded. The enlargement ratio of the dynamic range is determined so that (maximum high brightness) is obtained.

  That is, when the pixel occurrence frequency is counted from the maximum high luminance (255) side to the low luminance side with respect to the generated histogram of FIG. A region up to (for example, 0.01) or more is set as the expansion range of the histogram. In the histogram of FIG. 21A, the position indicated by the arrow a near the maximum high luminance (255) is a position (point) where the value is equal to or greater than the specified value.

  Then, like the bit compression conversion characteristics shown in FIG. 11 and FIG. 13, the specified value becomes a point for converting the input 14-bit data to the maximum value (4095) when compression-converting to the output 12-bit data. A bit compression conversion characteristic is generated. Alternatively, as shown in the equations (2) and (3), the upper limit value is obtained when the upper limit processing is performed on the input 14-bit data.

  By performing such processing, the control unit 28 can automatically set the expansion rate of the dynamic range in which an appropriate histogram as shown in FIG. Thereby, the range of all gradations that can be expressed by 8 bits of RGB can be used effectively, and occurrence of whiteout in the vicinity of the maximum high-luminance portion (255) can be suppressed.

  In addition, although 0.01 was mentioned as an example in this embodiment as said regulation value, it is good also as 0 (zero) or more, without being limited to this numerical value. A value of 0 (zero) or more means that the maximum value of luminance (maximum luminance part (255)) is detected when the expansion rate of the dynamic range is doubled, and the detected value is the maximum value of the final dynamic range. Is meant to be.

<Embodiment 7>
Also in the digital camera according to the present embodiment, an appropriate dynamic range enlargement ratio can be automatically set by the control unit 28 (see FIG. 2) as in the seventh embodiment. The dynamic range expansion rate setting process according to this embodiment will be described below.

  Also in the dynamic range expansion rate setting process according to the present embodiment, first, similarly to the first embodiment (or any one of the second, third, fourth, and fifth embodiments), the dynamic range expansion rate is set. Is set to double and dynamic range expansion processing is performed. In FIG. 22A, the dynamic range is doubled for a captured image in which the pixel output of the G filter exceeds the saturation level (for example, a high-luminance illuminator is in a very narrow area in the captured image). It is an example of the histogram produced | generated by the luminance histogram production | generation part 57 (refer FIG. 4) at the time of an expansion process.

  In this histogram, there is a pixel corresponding to the high-luminance luminous body in the vicinity of the maximum high luminance (255) (the location indicated by the arrow a in FIG. 22A). In FIG. 22A, the horizontal axis represents luminance (256 gradations (8 bits) from 0 to 255), and the vertical axis represents pixel occurrence frequency (0 to 1 (= 100%)).

  And the control part 28 (refer FIG. 2) of the signal processing part 22 is a low-intensity side with respect to the histogram (refer FIG. 22A) produced | generated based on the histogram data input from the brightness histogram production | generation part 57. FIG. To the maximum high luminance (255) side is counted. Then, the control unit 28, for example, a point that reaches about 98% before the maximum high luminance (255) with respect to all the count values from the low luminance to the maximum high luminance (255) (arrow b in FIG. 22A). The expansion rate of the dynamic range is determined so that the new luminance maximum value (maximum high luminance) after the dynamic range expansion becomes a portion indicated by.

  The range from the low luminance to the maximum high luminance (255) side of about 98% is the same as the bit compression conversion characteristics shown in FIGS. 11 and 13, and the input 14-bit data is the output 12-bit data. A bit compression conversion characteristic is generated so that it becomes a point to be converted to the maximum value (4095) at the time of compression conversion. Alternatively, as shown in the equations (2) and (3), the upper limit value is obtained when the upper limit processing is performed on the input 14-bit data.

  By performing such processing, the control unit 28 can automatically set the expansion rate of the dynamic range in which an appropriate histogram as shown in FIG. As a result, even when there is a high-luminance illuminant in a very narrow area in the photographed image, the influence is suppressed, and a high-luminance portion such as a light source that does not become unnatural even when saturated is saturated (FIG. 22B). ) In the range other than the high-intensity light source of the photographed image, the dynamic range can be expanded.

  In each of the embodiments described above, the RGB three primary color filters are arranged as the color separation filters. However, the present invention is similarly applied to a configuration in which the complementary color filters are arranged as the color separation filters. Can do.

(A) is a front view which shows the digital camera as an example of the imaging device concerning Embodiment 1-7 of this invention, (b) is the top view, (c) is the back view. The block diagram which shows the outline | summary of the system configuration | structure in the digital camera as an example of the imaging device which concerns on Embodiments 1-7 of this invention. The figure for demonstrating the principle of the dynamic range expansion in Embodiment 1 of this invention. The block diagram which shows the structure of the YUV conversion part in Embodiment 1 of this invention. The block diagram which shows the structure of the D range expansion prediction interpolation part in Embodiment 1 of this invention. (A), (b), (c) is a figure which shows an example of the imaging | photography setting screen displayed on the liquid crystal monitor. The figure which shows the pixel arrangement position and processing unit of CCD which have the RGB filter in Embodiment 1 of this invention. The figure which shows the bit compression conversion characteristic which compresses 14-bit data of the pixel output of the extended G filter in the case of carrying out the expansion process of a dynamic range twice in Embodiment 1 of this invention to 12 bits. The figure which shows the conversion table which converts 12-bit RGB data into 8-bit RGB data (gamma conversion). (A) is a figure which shows the histogram at the time of performing the expansion process of the dynamic range in Embodiment 1 of this invention, (b) shows the histogram at the time of not performing the expansion process of the dynamic range in this embodiment. Figure. The figure which shows the bit compression conversion characteristic which compresses 14-bit data of the pixel output of the extended G filter in the case of enlarging a dynamic range 1.3 times in Embodiment 1 of this invention to 12 bits. The figure which shows the histogram at the time of enlarging a dynamic range 1.3 times. The figure which shows the bit compression conversion characteristic which compresses 14-bit data of the pixel output of the extended G filter in the case of enlarging a dynamic range 1.6 times in Embodiment 1 of this invention to 12 bits. The figure which shows the histogram at the time of enlarging a dynamic range 1.6 times. The figure which shows the pixel arrangement position and processing unit of CCD which have the RGB filter in Embodiment 2 of this invention. The figure which shows the pixel arrangement position and processing unit of CCD which have an RGB filter in Embodiment 3 of this invention. The figure which shows the pixel arrangement position and processing unit of CCD which have an RGB filter in Embodiment 4 of this invention. The block diagram which shows the structure of the D range expansion prediction interpolation part in Embodiment 4 of this invention. (A) is a figure which shows the pixel output value of each G filter in the diagonal line of a direction, and the predicted value of the pixel output of G filter of a center part, (b) is the pixel output of each G filter in the diagonal line of b direction. The figure which shows the predicted value of the pixel output of the value of G, and G filter of the center part. The block diagram which shows the structure of the D range expansion prediction interpolation part in Embodiment 5 of this invention. (A) is a figure which shows the histogram at the time of enlarging a dynamic range to 2 times, (b) is a figure which shows the histogram at the time of automatically setting the expansion ratio of the dynamic range in Embodiment 6. (A) is a figure which shows the histogram at the time of enlarging a dynamic range twice, (b) is a figure which shows the histogram at the time of automatically setting the expansion rate of the dynamic range in Embodiment 7.

Explanation of symbols

1 Digital camera (imaging device)
5 Shooting lens system (optical system)
6 Lens unit 9 LCD monitor 12 Menu button (dynamic range expansion ratio setting means, operation selection means)
20 CCD (imaging device)
21 Analog Front End 22 Signal Processor 23 SDRAM
28 Control Unit (Magnification Ratio Change Control Unit)
34 CCD interface 35 Memory controller 36 YUV conversion unit 50 D range expansion prediction interpolation unit 51 Bit compression conversion unit (bit compression conversion means)
57 Luminance histogram generator (histogram generator)
60 Luminance level determination unit (pixel output detection means)
61, 61a Pixel output correction processing unit (pixel output correction processing means)
62-bit extension processing unit 63 first pixel output correction processing unit (first pixel output correction processing means)
64 Second pixel output correction processing section (second pixel output correction processing means)
65 Correction data synthesis unit (predictive interpolation processing means)

Claims (26)

A subject image incident through the optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a color separation filter of a three primary color filter or a complementary color filter is disposed on the front side of each pixel. In an imaging device that includes an imaging element and in which a pixel in which a filter of a specific color is arranged for light having a wide wavelength band has a higher luminance sensitivity than a pixel in which a filter of another color is arranged,
Based on the electrical signal output from the image sensor, the output from each pixel in which the color separation filter is arranged is detected, and whether the output from each pixel has reached a predetermined saturation level or more. Pixel output detection means for determining
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the other color filters that are not saturated are Pixel output correction processing means for predictively interpolating the pixel output in the region above the saturation level and expanding the dynamic range based on the output from the arranged pixels;
A dynamic range expansion rate setting means for setting a dynamic range expansion rate;
An enlargement ratio change control means for changing and controlling an enlargement ratio for the dynamic range enlarged by the predictive interpolation process by the pixel output correction processing means based on the enlargement ratio set by the dynamic range enlargement ratio setting means; ,
An imaging apparatus characterized by that. A subject image incident through the optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a color separation filter of a three primary color filter or a complementary color filter is disposed on the front side of each pixel. In an imaging device that includes an imaging element and in which a pixel in which a filter of a specific color is arranged for light having a wide wavelength band has a higher luminance sensitivity than a pixel in which a filter of another color is arranged,
Based on the electrical signal output from the image sensor, the output from each pixel in which the color separation filter is arranged is detected, and whether the output from each pixel has reached a predetermined saturation level or more. Pixel output detection means for determining
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the surrounding color of the specific color not saturated is the same Pixel output correction processing means for predictively interpolating the pixel output in the region above the saturation level and expanding the dynamic range based on the output from the pixel in which the filter is disposed;
A dynamic range expansion rate setting means for setting a dynamic range expansion rate;
An enlargement ratio change control means for changing and controlling an enlargement ratio for the dynamic range enlarged by the predictive interpolation process by the pixel output correction processing means based on the enlargement ratio set by the dynamic range enlargement ratio setting means; ,
An imaging apparatus characterized by that. A subject image incident through the optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a color separation filter of a three primary color filter or a complementary color filter is disposed on the front side of each pixel. In an imaging device that includes an imaging element and in which a pixel in which a filter of a specific color is arranged for light having a wide wavelength band has a higher luminance sensitivity than a pixel in which a filter of another color is arranged,
From the electrical signal output from the image sensor, the output from each pixel in which the color separation filter is arranged is detected, and whether or not the output from each pixel has reached a predetermined saturation level or more is determined. A pixel output detecting means,
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the other color filters that are not saturated are First pixel output correction processing means for predictively interpolating the pixel output in the region above the saturation level based on the output from the arranged pixels;
When it is determined by the pixel output detection means that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the surrounding color of the specific color not saturated is the same Second pixel output correction processing means for predictively interpolating the pixel output in the region above the saturation level based on the output from the pixel in which the filter is disposed;
Based on the prediction interpolation information respectively output from the first pixel output correction processing means and the second pixel output correction processing means, an appropriate prediction interpolation process is performed on the pixel output in the region above the saturation level. Predictive interpolation processing means for expanding the dynamic range;
A dynamic range expansion rate setting means for setting a dynamic range expansion rate;
An enlargement ratio change control means for changing and controlling the enlargement ratio for the dynamic range expanded by the predictive interpolation process by the predictive interpolation processing means based on the enlargement ratio set by the dynamic range enlargement ratio setting means,
An imaging apparatus characterized by that. The processing unit when the pixel output detection means detects the output of each pixel is a size of 2 × 2 pixels in the horizontal and vertical directions.
The imaging apparatus according to any one of claims 1 to 3, wherein Corresponding to the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means, comprising histogram generation means for calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
The enlargement ratio change control unit is configured to generate the histogram generation unit when it is determined by the pixel output detection unit that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high-luminance side to the low-luminance side with respect to the histogram generated in step 1, the pixel occurrence frequency becomes equal to or greater than a predetermined value. Changing and controlling the enlargement ratio with respect to the dynamic range enlarged by the pixel output correction processing means so that the position becomes a new maximum high brightness position of the histogram,
The imaging apparatus according to claim 1 or 2, wherein Corresponding to the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means, comprising histogram generation means for calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
The enlargement ratio change control unit is configured to generate the histogram generation unit when it is determined by the pixel output detection unit that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high-luminance side to the low-luminance side with respect to the histogram generated in step 1, the pixel occurrence frequency becomes equal to or greater than a predetermined value. Changing and controlling the enlargement ratio for the dynamic range expanded by the predictive interpolation processing means so that the position becomes the new maximum high-intensity position of the histogram,
The imaging apparatus according to claim 3. Corresponding to the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means, comprising histogram generation means for calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
The enlargement ratio change control unit is configured to generate the histogram generation unit when it is determined by the pixel output detection unit that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. The position of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated in step 4 Change and control the enlargement ratio for the dynamic range enlarged by the pixel output correction processing means so as to be the maximum high luminance position,
The imaging apparatus according to claim 1 or 2, wherein Corresponding to the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing means, comprising histogram generation means for calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
The enlargement ratio change control unit is configured to generate the histogram generation unit when it is determined by the pixel output detection unit that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. The position of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated in step 4 To change and control the enlargement ratio for the dynamic range enlarged by the predictive interpolation processing means so as to be the maximum high luminance position,
The imaging apparatus according to claim 3. When the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level, the pixel output correction processing means outputs the first bit number below the predetermined saturation level. Bit compression conversion means for compressing the pixel output data once expanded to a bit number of 2 to the first bit number again based on a preset bit compression conversion characteristic table;
The enlargement ratio change control means changes and controls an enlargement ratio for the dynamic range expanded by the pixel output correction processing means based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed.
The imaging apparatus according to claim 1 or 2, wherein When the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level, the pixel output correction processing means outputs the first bit number below the predetermined saturation level. Bit compression conversion means for compressing the pixel output data once expanded to a bit number of 2 to the first bit number again based on a preset bit compression conversion characteristic table;
The enlargement ratio change control means changes and controls the enlargement ratio for the dynamic range expanded by the predictive interpolation processing means based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed.
The imaging apparatus according to claim 3. The enlargement ratio change control means outputs the pixel output correction processing means below the predetermined saturation level when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. The pixel output data once expanded from the first number of bits to the second number of bits is multiplied by a predetermined coefficient to change the expansion ratio for the dynamic range expanded by the pixel output correction processing means Control,
The imaging apparatus according to claim 1 or 2, wherein The enlargement ratio change control means outputs the pixel output correction processing means below the predetermined saturation level when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level. The pixel output data once expanded from the first number of bits to the second number of bits is multiplied by a predetermined coefficient, thereby changing the expansion rate for the dynamic range expanded by the predictive interpolation processing means To
The imaging apparatus according to claim 3. An operation selecting means for selecting and executing the operation of performing the predictive interpolation when the predetermined saturation level is reached or higher with respect to the output of the pixel in which the filter of the specific color is disposed;
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. A subject image incident through the optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a color separation filter of a three primary color filter or a complementary color filter is disposed on the front side of each pixel. An imaging method for an imaging apparatus having an imaging element, in which a pixel in which a filter of a specific color is arranged for light having a wide wavelength band has a higher luminance sensitivity than a pixel in which a filter of another color is arranged In
Based on the electrical signal output from the image sensor, the output from each pixel in which the color separation filter is arranged is detected, and whether the output from each pixel has reached a predetermined saturation level or more. A pixel output detection step of determining
When it is determined in the pixel output detection step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the other color filters that are not saturated around the filter are Based on the output from the arranged pixels, a pixel output correction processing step that predictively interpolates the pixel output in the region above the saturation level and expands the dynamic range; and
An enlargement ratio for changing and controlling the enlargement ratio for the dynamic range enlarged by the predictive interpolation process in the pixel output correction processing step based on the enlargement ratio set by the dynamic range enlargement ratio setting means for setting the enlargement ratio of the dynamic range A change control process,
An image pickup method for an image pickup apparatus. A subject image incident through the optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a color separation filter of a three primary color filter or a complementary color filter is disposed on the front side of each pixel. An imaging method for an imaging apparatus having an imaging element, in which a pixel in which a filter of a specific color is arranged for light having a wide wavelength band has a higher luminance sensitivity than a pixel in which a filter of another color is arranged In
Based on the electrical signal output from the image sensor, the output from each pixel in which the color separation filter is arranged is detected, and whether the output from each pixel has reached a predetermined saturation level or more. A pixel output detection step of determining
When it is determined in the pixel output detection step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the surrounding color of the specific color that is not saturated Based on the output from the pixel in which the filter is arranged, a pixel output correction processing step that predictively interpolates the pixel output in the region above the saturation level and expands the dynamic range;
An enlargement ratio for changing and controlling the enlargement ratio for the dynamic range enlarged by the predictive interpolation process in the pixel output correction processing step based on the enlargement ratio set by the dynamic range enlargement ratio setting means for setting the enlargement ratio of the dynamic range A change control process,
An image pickup method for an image pickup apparatus. A subject image incident through the optical system is received by a light receiving surface having a plurality of pixels and converted into an electrical signal, and a color separation filter of a three primary color filter or a complementary color filter is disposed on the front side of each pixel. An imaging method for an imaging apparatus having an imaging element, in which a pixel in which a filter of a specific color is arranged for light having a wide wavelength band has a higher luminance sensitivity than a pixel in which a filter of another color is arranged In
From the electrical signal output from the image sensor, the output from each pixel in which the color separation filter is arranged is detected, and whether or not the output from each pixel has reached a predetermined saturation level or more is determined. A pixel output detecting step,
When it is determined in the pixel output detection step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the other color filters that are not saturated around the filter are A first pixel output correction processing step of predictively interpolating a pixel output in the region above the saturation level based on an output from the arranged pixel;
When it is determined in the pixel output detection step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the surrounding color of the specific color that is not saturated A second pixel output correction processing step of predictively interpolating the pixel output in the region above the saturation level based on the output from the pixel in which the filter is disposed;
Based on the prediction interpolation information output from each of the first pixel output correction processing step and the second pixel output correction processing step, appropriate prediction interpolation processing is performed on the pixel output in the region above the saturation level. A predictive interpolation process to expand the dynamic range;
An enlargement ratio change for controlling the enlargement ratio for the dynamic range enlarged by the predictive interpolation process in the predictive interpolation process based on the enlargement ratio set by the dynamic range enlargement ratio setting means for setting the enlargement ratio of the dynamic range A control process,
An image pickup method for an image pickup apparatus. The processing unit when detecting the output of each pixel is 2 × 2 pixels in the horizontal and vertical directions.
The imaging method of the imaging device according to claim 14, wherein the imaging method is an imaging method. In correspondence with the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step, further includes a histogram generation step of calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
In the enlargement ratio change control step, the histogram generation step is performed when it is determined in the pixel output detection step that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high-luminance side to the low-luminance side with respect to the histogram generated in step 1, the pixel occurrence frequency becomes equal to or greater than a predetermined value. Changing and controlling the enlargement ratio for the dynamic range expanded by the pixel output correction processing step so that the position becomes a new maximum high-intensity position of the histogram,
The imaging method of the imaging device according to claim 14, wherein the imaging method is an imaging method. In correspondence with the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step, further includes a histogram generation step of calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
In the enlargement ratio change control step, when it is determined by the pixel output detection step that the output from the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more, the histogram generation means Based on the pixel occurrence frequency information when the pixel occurrence frequency is detected from the maximum high-luminance side to the low-luminance side with respect to the histogram generated in step 1, the pixel occurrence frequency becomes equal to or greater than a predetermined value. Change and control the enlargement ratio for the dynamic range expanded by the predictive interpolation processing step so that the position becomes the new maximum high-intensity position of the histogram.
The imaging method of the imaging device according to claim 16. In correspondence with the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step, further includes a histogram generation step of calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
In the enlargement ratio change control step, the histogram generation step is performed when it is determined in the pixel output detection step that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. The position of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated in step 4 To change and control the enlargement ratio for the dynamic range enlarged by the pixel output correction processing step so as to be the maximum high luminance position,
The imaging method of the imaging device according to claim 14, wherein the imaging method is an imaging method. In correspondence with the dynamic range expanded by the predictive interpolation processing by the pixel output correction processing step, further includes a histogram generation step of calculating the occurrence frequency of pixels between low luminance and maximum high luminance,
In the enlargement ratio change control step, the histogram generation step is performed when it is determined in the pixel output detection step that an output from a pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or more. The position of the number of luminance data when counting from the low luminance side to the maximum high luminance side with respect to the histogram generated in step 4 Change and control the enlargement ratio for the dynamic range expanded by the predictive interpolation processing step so as to be the maximum high luminance position.
The imaging method of the imaging device according to claim 16. When the output of the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the pixel output correction processing step outputs the first bit number below the predetermined saturation level. A bit compression conversion step of compressing the pixel output data once expanded to the number of bits of 2 to the first number of bits again based on a preset bit compression conversion characteristic table;
The enlargement ratio change control step changes and controls the enlargement ratio for the dynamic range expanded by the pixel output correction processing step based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed.
The imaging method of the imaging device according to claim 14, wherein the imaging method is an imaging method. When the output of the pixel in which the filter of the specific color is arranged has reached the predetermined saturation level or higher, the pixel output correction processing step outputs the first bit number below the predetermined saturation level. A bit compression conversion step of compressing the pixel output data once expanded to the number of bits of 2 to the first number of bits again based on a preset bit compression conversion characteristic table;
The enlargement ratio change control step changes and controls the enlargement ratio for the dynamic range expanded by the predictive interpolation processing step based on the bit compression conversion characteristic table in which the bit compression conversion characteristic is changed.
The imaging method of the imaging device according to claim 16. In the enlargement ratio change control step, when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level, the pixel output correction processing step outputs the pixel level or less. The pixel output data once expanded from the first number of bits to the second number of bits is multiplied by a predetermined coefficient to change the expansion ratio for the dynamic range expanded by the pixel output correction processing step Control,
The imaging method of the imaging device according to claim 14, wherein the imaging method is an imaging method. In the enlargement ratio change control step, when the output of the pixel in which the filter of the specific color is arranged reaches or exceeds the predetermined saturation level, the pixel output correction processing step outputs the pixel level or less. The pixel output data once expanded from the first number of bits to the second number of bits is multiplied by a predetermined coefficient, thereby changing the expansion rate for the dynamic range expanded by the predictive interpolation processing step. To
The imaging method of the imaging device according to claim 16. The operation of performing the predictive interpolation when the predetermined saturation level is reached or higher with respect to the output of the pixel in which the filter of the specific color is arranged is executed by selecting an operation with an operation selection unit.
26. The imaging method for an imaging apparatus according to any one of claims 14 to 25.

Images