JP2003309857A - Image processing apparatus and method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To produce a wide dynamic color image with uniformized sensitivity of all pixels and excellent color reproducibility by correcting a problem of a color/sensitivity mosaic image due to a spectral transmissivity of an ND filter. <P>SOLUTION: The image processing method includes: a step S11 of allowing a local information calculation section to calculate local information corresponding to a target pixel being a low sensitivity pixel; a step S12 wherein a sensitivity correction value calculation section applies the local information calculated by the local information calculation section to a prepared function in advance to calculate a sensitivity correction value corresponding to the target pixel being the low sensitivity pixel; and a step S13 wherein a sensitivity correction section corrects the target pixel on the basis of the sensitivity correction value from the sensitivity correction calculation section so that the exposure sensitivity ratio in the target pixel being the low sensitivity pixel is a prescribed value. This invention is applicable to e.g. digital cameras. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, a recording medium, and a program, for example,
Used to perform sensitivity correction in consideration of the spectral transmittance characteristics of the transmission limiting filter for color / sensitivity mosaic images output from a solid-state image sensor that uses a transmission limiting filter that limits the transmission of incident light in pixel units. And an image processing apparatus and method, a recording medium, and a program which are suitable.

[0002]

2. Description of the Related Art Conventionally, in order to obtain a color image, a CCD (color filter having a Bayer array of red (R), green (G), and blue (B)) having different spectral sensitivity characteristics is used. A solid-state imaging device such as a charge coupled device) is covered in pixel units.

A phenomenon that occurs due to the wavelength dependence of the spectral sensitivity characteristic of such a color CCD will be described with reference to FIGS. 1 and 2.

FIG. 1 shows the spectral sensitivity characteristics of a color CCD. Curves R, G, and B show the spectral sensitivity characteristics of CCDs using red, green, or blue filters, respectively. The horizontal axis of the figure shows the wavelength and the vertical axis shows the relative sensitivity.

FIG. 2 shows the light reflected by the object, that is,
The spectral distribution of the incident light with respect to CCD is shown. Curve a,
b and c show the spectral distributions of sunlight, a fluorescent lamp, and an incandescent lamp, respectively. The horizontal axis of the figure shows the wavelength and the vertical axis shows the relative energy.

The light measured by the CCD is obtained as the area of the spectral characteristic obtained by multiplying the spectral distribution of the light energy reflected by the subject and the spectral sensitivity characteristic of the CCD. As shown in FIG. 1, since the CCD does not have uniform sensitivity over the entire wavelength band of the incident light, the CCD using the color filter cannot accurately measure the incident light and the output image Color misalignment may occur.

Conventionally, a method for correcting the problem caused by the spectral characteristic of such a color filter has been proposed.

For example, in Japanese Patent Laid-Open No. 7-307947, the color filter sensitivity characteristics for each color are based on the spectral distribution of the light source, the reflection and transmission characteristics of the object, the spectral transmission characteristics of the color filter, and the spectral transmission characteristics of the image sensor. A method (hereinafter, referred to as a first conventional method) for obtaining a value and correcting a problem caused by the spectral characteristic of the color filter using the value is disclosed. According to the first conventional method, color misregistration can be suppressed without using a birefringent optical element.

Further, for example, in Japanese Unexamined Patent Publication No. 10-336484, a single-plate color CCD provided with a solid-state image pickup device having a 2 × 2 color filter array composed of three complementary colors (yellow, cyan, magenta) and green. A method (hereinafter, referred to as a second conventional method) for correcting the spectral sensitivity characteristic of a magenta color filter by using an output signal corresponding to a green color filter in signal processing of a camera is disclosed. The second conventional method does not adversely affect the saturation and hue of the output signal.

By the way, ND (N
Eutral Density) filter covers single-chip CCD with normal dynamic range in pixel units, resulting in low-sensitivity pixels (using ND filter)
And high-sensitivity imaged pixels (ND filter not used)
And a predetermined image processing is performed on the image picked up by the single-plate CCD with the ND filter to generate an image having a wide dynamic range in which all pixels are uniform (hereinafter, referred to as a third conventional method). Is described) exists. This technique is described, for example, in the document SK Nayar and T.
Mitsunaga, “High Dynamic Range Imaging: Spatially
Varying Pixel Exposures ”, Proc. Of Computer Visi
on and Pattern Recognition 2000, Vol. 1, pp. 472-4
79, June, 2000 ".

Further, the present applicant has already specified the invention (hereinafter referred to as the fourth conventional method) for applying the third conventional method to a single-plate color CCD to generate a color image having a wide dynamic range. It has been proposed as Japanese Patent Application No. 2001-000979.

Now, an ND filter for limiting the transmission of incident light will be described with reference to FIGS. 3 to 5. Figure 3
5, the horizontal axis represents wavelength and the vertical axis represents transmittance of incident light.

FIG. 3 shows an ideal spectral transmittance characteristic of the ND filter. That is, the ideal spectral transmittance characteristic of the ND filter can uniformly limit the transmittance of incident light over the entire band. The third conventional method is premised on the use of an ideal ND filter.

However, it is difficult to manufacture such an ideal ND filter, and in a realistic ND filter, as shown in FIG. 4, for example, the transmittance changes linearly with respect to the wavelength of incident light. For example, as shown in FIG. 5, the transmittance changes non-linearly with respect to the wavelength of incident light.

Therefore, in the third conventional method, when an ND filter having an ideal spectral transmittance characteristic as shown in FIG. 3 is used, the exposure sensitivity ratio of low sensitivity and high sensitivity can be obtained regardless of the wavelength of incident light. Since it is a constant value, the exposure sensitivity of all pixels can be easily made uniform. However, when an ND filter having a non-ideal spectral transmittance characteristic as shown in FIG. 4 or FIG. 5 is used, the exposure sensitivity ratio changes depending on the wavelength of the incident light, so depending on the spectral distribution of the incident light, There is a problem that the sensitivity of all pixels cannot be made uniform appropriately.

[0016]

Therefore, in the fourth conventional method to which the third conventional method is applied, there are problems caused by the fact that the ND filter does not have an ideal spectral transmittance characteristic, and the color filter. You will have both problems due to the spectral characteristics of.

The problem caused by the spectral characteristic of the color filter, which the fourth conventional method has, overlaps with the above-mentioned problem caused by the fact that the ND filter does not have an ideal spectral transmittance characteristic. Therefore, there is a problem that the first and second conventional methods cannot be applied.

The present invention has been made in view of such a situation, and by covering a single-plate color CCD having a normal dynamic range with an ND filter on a pixel-by-pixel basis, it is possible to cause a difference in exposure sensitivity for each pixel. Based on the color / sensitivity mosaic image in which the color components of each pixel are different and the sensitivity of each pixel when shooting is different,
An object of the present invention is to enable generation of a wide dynamic color image in which all pixels have uniform sensitivity and excellent color reproducibility.

[0019]

An image processing apparatus according to the present invention includes a detecting means for detecting a low-sensitivity pixel of a pixel of a color / sensitivity mosaic image which is imaged with a low sensitivity by a transmission limiting filter, and a detecting means. A calculating unit that calculates local information corresponding to the detected low-sensitivity pixel, a determining unit that determines a sensitivity correction value corresponding to the low-sensitivity pixel based on the local information, and a low-sensitivity pixel is corrected based on the sensitivity correction value. And a correction means for performing the correction.

The calculating means may calculate, as the local information, a local color balance corresponding to the low-sensitivity pixel detected by the detecting means.

The calculating means uses, as the local information, a local G corresponding to the low-sensitivity pixel detected by the detecting means.
The ratio of the R component to the component or the ratio of the B component to the G component can be calculated.

The calculating means may calculate, as the local information, a local luminance balance corresponding to the low-sensitivity pixel detected by the detecting means.

The determining means may be adapted to apply the local information to a predetermined function to calculate a sensitivity correction value corresponding to the low sensitivity pixel.

The determining means may apply the local information to the look-up table to obtain the sensitivity correction value corresponding to the low-sensitivity pixel.

The transmission limiting filter may be an ND filter.

The color filter may be composed of three primary colors of red, green and blue.

The color filter may be composed of three complementary colors of yellow, magenta and cyan.

The color filter may be composed of three complementary colors of yellow, magenta and cyan, and four colors of green.

The image processing method of the present invention comprises a detection step of detecting low-sensitivity pixels of the pixels of the color / sensitivity mosaic image that are imaged at low sensitivity by the transmission limiting filter, and a low-sensitivity pixel detected in the processing of the detection step. Based on the calculation step of calculating the local information corresponding to the sensitivity pixel and the local information,
It is characterized by including a determination step of determining a sensitivity correction value corresponding to the low sensitivity pixel and a correction step of correcting the low sensitivity pixel based on the sensitivity correction value.

The program of the recording medium of the present invention is detected by the detection step of detecting low-sensitivity pixels of the pixels of the color / sensitivity mosaic image which are imaged with low sensitivity by the transmission limiting filter, and the detection step. A calculating step of calculating local information corresponding to the low sensitivity pixel, a determining step of determining a sensitivity correction value corresponding to the low sensitivity pixel based on the local information, and a correction step of correcting the low sensitivity pixel based on the sensitivity correction value. It is characterized by including and.

The program of the present invention includes a detection step of detecting low-sensitivity pixels of the pixels of the color / sensitivity mosaic image which are imaged at low sensitivity by a transmission limiting filter, and low-sensitivity pixels detected by the processing of the detection step. A calculation step of calculating local information corresponding to the computer, a determination step of determining a sensitivity correction value corresponding to the low sensitivity pixel based on the local information, and a correction step of correcting the low sensitivity pixel based on the sensitivity correction value. It is characterized by making it execute.

In the image processing apparatus and method and the program of the present invention, the low-sensitivity pixel imaged at low sensitivity is detected by the transmission limiting filter among the pixels of the color / sensitivity mosaic image, and the detected low-sensitivity pixel. Is calculated, a sensitivity correction value corresponding to the low sensitivity pixel is determined based on this local information, and the low sensitivity pixel is corrected based on this sensitivity correction value.

[0033]

BEST MODE FOR CARRYING OUT THE INVENTION A configuration example of a digital camera to which the present invention is applied will be described below with reference to FIG.

The lens 11 uses the CCD 16 to form an optical image of the subject.
Focus on. The diaphragm 12 limits the incident light on the CCD 16 to adjust the brightness. The shutter 13 momentarily blocks the light incident on the CCD 16 to stop the exposure. The infrared cut filter 14 removes infrared light from the incident light on the CCD 16. The optical low pass filter 15 is a CCD
The generation of aliasing due to 16 samplings is suppressed. The CCD 16 has an ND filter 61 and a color filter 6
The solid-state imaging device 63 (FIG. 9) covered with 2 has photoelectric conversion and charge transfer functions and outputs a color / sensitivity mosaic image signal to the subsequent stage.

The correlated double sampling unit (CDS) 17 is a CCD
The noise of the color / sensitivity mosaic image signal is removed from 16. The auto gain control (AGC) circuit 18 amplifies the amplitude of the color / sensitivity mosaic image signal in accordance with the brightness of the subject. Analog-to-digital converter (A / D)
Reference numeral 19 converts the color / sensitivity mosaic image signal, which is an analog signal, into, for example, a 10-bit digital signal.

The signal processor 20 generates a wide dynamic range color image signal based on the digitized color / sensitivity mosaic image signal. In addition, the signal processing unit 20
Converts the generated wide dynamic range color image signal into a luminance signal (Y) and a color difference signal (Cb, Cr) and outputs it to the subsequent stage.

The timing generator 21 has a CCD 16
Is driven horizontally and vertically, and the timing of the shutter 13 and the like is also controlled.

FIG. 7 shows an electrical configuration example of the signal processing section 6. The signal processing unit 31, which is an embodiment of the image processing apparatus of the present invention, performs image processing, which will be described later, on the digitized color / sensitivity mosaic image signal input from the A / D 19 to obtain a wide dynamic range color image. Generate an image signal. The signal processing unit 31 includes an image bus 40.
Through the memory control unit 32 and the resolution conversion unit 3
3, a video encoder 34, and a JPEG encoder / decoder 38 are connected.

The memory control section 32 controls the external memo 39 for recording the image signal. Resolution converter 33
Converts the generated wide dynamic range color image signal to a resolution for storage. The video encoder 34 converts the wide dynamic range color image signal into a video signal format (NTSC, PAL, etc.) and outputs the signal to the outside via a digital-analog converter (D / A) 35. The camera detection processing unit 36 performs detection processing for autofocus, auto iris, and white balance adjustment on the digitized color / sensitivity mosaic image signal input via the signal processing unit 31. The JPEG encoder / decoder 38 compresses the generated wide dynamic range color image signal in accordance with the JPEG format, and decompresses the image signal in the JPEG format.

A CPU (Central Processing Unit) 40 includes an image signal processing unit 31 via a host interface unit 37.
To control. The CPU 40 has a human interface (I / F) 41, an SDRAM 42, a flash memory 43, a host interface unit 37, and a JPEG via a CPU bus 45.
An encoder / decoder 38 is connected.

Human interface CPU (HI CPU) 4
1 is the key input operation from the user and the LED (Light Emitting
Diode) Controls processing such as lighting. SDRAM (Synchronous
Dynamic RAM) 42 is a program area memory,
Image signals are temporarily recorded. The flash memory 43 is a program memory in which a control program executed by the CPU 40 is recorded.

Next, the outline of the operation of the digital camera to which the present invention is applied will be described with reference to FIG. Figure 8
Shows a functional block diagram of a digital camera.

The image pickup section 51 corresponds to the lens 11 to the analog-digital conversion section (A / D) 19 shown in FIG. Further, the spectral correction unit 52 and the wide dynamic range color image generation unit 53 correspond to the image signal processing unit 31 of FIG. 7.

The image pickup section 51 generates a digital color / sensitivity mosaic image signal corresponding to the subject, and the spectral correction section 5
Output to 2. The spectral correction unit 52 considers the spectral transmittance characteristics of the ND filter 61 included in the imaging unit 51, and
By correcting the pixels of the sensitivity mosaic image signal for which the transmission of incident light is limited by the ND filter 61 at the time of shooting, a color / sensitivity mosaic image in which the exposure sensitivity ratio in all pixels has a constant value is obtained. It is generated and output to the wide dynamic range color image generation unit 53.

Wide dynamic range color image generator 5
3 has a wide dynamic range R image signal in which all pixels have only an R component having a wide dynamic range based on the corrected color / sensitivity mosaic image signal, and all pixels have only a G component having a wide dynamic range. It produces a wide dynamic range G image signal and a wide dynamic range B image signal in which all pixels have only a wide dynamic range B component. The details of the processing performed by the wide dynamic range color image generation unit 53 are described in the above-mentioned fourth example.
Since it is the same as the conventional method of No. 1, its explanation is omitted.

Here, an example of the color / sensitivity mosaic image signal will be described with reference to FIGS. 9 and 10. Figure 9
Shows a configuration example of the CCD 16. The CCD 16 includes, in order from the lens 11 side, an ND filter 61 that limits transmission of incident light, a color filter 62 that limits the wavelength band of the incident light, and a photodiode (PD) 63 that generates a charge corresponding to the incident light. Is configured. The ND filter 61, the color filter 62, and the photodiode 6
One section of 3 corresponds to one pixel, respectively.

Instead of the ND filter 61, for example, a liquid crystal filter that limits transmission of incident light may be used.

However, the ND filter 61 is not provided for all pixels, but "d (dar
It is provided only at the position marked "k)" and "b (brigh
It is not provided at the position marked "t)". Therefore, the pixel at the position marked "d" has a low sensitivity because the exposure is restricted, and the pixel at the position marked "b" is not restricted in the exposure. High sensitivity, the "d" and "b" shown
The arrangement of is an example, and other arrangements may be used.
Hereinafter, a pixel provided with the ND filter 61 at the time of shooting will be described as a low sensitivity pixel.

In the color filter 62, the three primary colors of red (R), green (R), and blue (B) form a Bayer array. The RGB arrangement may be other than the illustrated Bayer arrangement.

The color type of the color filter 62 is RG.
The color is not limited to B, and may be, for example, three complementary colors of yellow, cyan, and magenta, or may be four colors in which green (G) is added to the three complementary colors of yellow, cyan, and magenta.

FIG. 10 shows a CCD 1 constructed as shown in FIG.
6 shows a color / sensitivity mosaic image output from No. 6.
For example, the pixel Gd (3,0) in the lower right corner is taken with low sensitivity and has only the G component. Further, for example, the pixel Gb (0,3) in the upper left corner is taken with high sensitivity and has only the G component.

Next, FIG. 11 shows a first configuration example of the spectral correction section 52 of FIG. The first configuration example of the spectral correction unit 52 is a configuration example suitable when the spectral transmittance characteristic of the ND filter 61 is linear as shown in FIG.

The local information calculating unit 71 is a low-sensitivity pixel (for example, a pixel marked with "Xd" in FIG. 10) of the pixels of the color / sensitivity mosaic image input from the image pickup unit 51.
X is one of R, G, and B), the local information is calculated using only the low-sensitivity pixels in the vicinity, and the local information is output to the sensitivity information correction value calculation unit 72. Here, the local information indicates color balance or brightness balance.

The color balance as the local information is specifically the ratio (R / G) of the R component to the local G component of the pixel when the pixel is "Rd". When the pixel is “Bd”, it is the ratio of the B component to the local G component of the pixel (B / G). The pixel is "G
If d ″, R for the local G component of the pixel
It is a ratio of components (R / G) or a ratio of B components to local G components of the pixel (B / G).

For example, the color balance R / G (1, of the pixel Rd (1,1) of the color / sensitivity mosaic image shown in FIG.
1) is calculated by the following equation (1). R / G (1,1) = Rd (1,1) / Ave [(Gd (1,0), Gd (1,2)] ... (1) where Ave [A, B] is the pixel A And the average value of the pixel B.

Further, for example, the color balance B / G (2,2) of the pixel Bd (2,2) is calculated by the following equation (2). B / G (2,2) = Bd (2,2) / Ave [(Gd (1,2), Gd (3,2)] ... (2)

Further, for example, the color balance R / G (1,2) of the pixel Gd (1,2) is calculated by the following equation (3). Alternatively, the color balance B / G of the pixel Gd (1,2)
(1,2) is calculated by the following equation (4). R / G (1,2) = Rd (1,1) / Gd (1,2) (3) B / G (1,2) = Bd (2,2) / Gd (1,2) ... (4)

Specifically, the brightness balance as the local information is the ratio (R / Y) of the R component to the local brightness value Y of the pixel when the pixel is "Rd".
When the pixel is “Bd”, it is the ratio (B / Y) of the B component to the local luminance value Y of the pixel. When the pixel is “Gd”, it is the ratio of the G component to the local luminance value Y of the pixel.

For example, the brightness balance R / Y (1, of the pixel Rd (1,1) of the color / sensitivity mosaic image shown in FIG.
1) is calculated by the following equation (5). R / Y (1,1) = Rd (1,1) / {x · Rd (1,1) + y · Ave [(Gd (1,0), Gd (1,2)] + z · Ave [( Bd (0,0), Gd (2,2)]} (5) where x, y, and z are predetermined constants.

Further, for example, the brightness balance B / Y (2,2) of the pixel Bd (2,2) is calculated by the following equation (6). B / Y (2,2) = Bd (2,2) / {x · Ave [(Rd (1,1), Rd (3,3)] + y · Ave [(Gd (1,2), Gd (3,2)] + z · Bd (2,2)} (6)

Further, for example, G / Y (1,2) as the luminance balance of the pixel Gd (1,2) is calculated by the following equation (7). G / Y (1,2) = Gd (1,2) / {x.Rd (1,1) + y.Gd (1,2) + z.Bd (2,2)} (7)

The sensitivity information correction value calculation unit 72 calculates the sensitivity correction value for each low sensitivity pixel of the pixels of the color / sensitivity mosaic image,
Specifically, the ratio (Xd / Xb) of the low-sensitivity exposure to the high-sensitivity exposure in the low-sensitivity pixel (X is either R, G, or B) is calculated based on the corresponding local information, and the sensitivity is corrected. It is output to the unit 73.

For example, the color balance R / as local information
Based on G (1,1), the sensitivity correction value Rd / Rb (1,
1) is calculated. Further, for example, based on the color balance B / G (2,2) as the local information, the sensitivity correction value Bd / Bb
Calculate (2, 2). Furthermore, for example, color balance R / G (1,2) or B / R (1,2) as local information
Based on the above, the sensitivity correction value Gd / Gb (1,2) is calculated.

Alternatively, for example, the sensitivity correction value Rd / R is based on the brightness balance R / Y (1,1) as the local information.
Calculate b (1,1). Further, for example, the sensitivity correction value Bd / Bb (2,2) is calculated based on the brightness balance B / Y (2,2) as the local information. Further, for example, based on the brightness balance G / Y (1,2) as local information,
The sensitivity correction value Gd / Gb (1,2) is calculated.

It is assumed that the function for calculating the sensitivity correction value based on the local information is prepared in advance corresponding to the spectral transmittance characteristic of the ND filter 61 used.

The sensitivity correction unit 73 sets each low sensitivity pixel based on the sensitivity correction value from the sensitivity correction value calculation unit 72 so that the exposure sensitivity ratio of each low sensitivity pixel of the color / sensitivity mosaic image becomes a constant value. to correct. Therefore, the sensitivity correction unit 73
From the color / sensitivity mosaic image (that is, the ND filter 61
(The same as the color / sensitivity mosaic image picked up in the ideal case where the spectral transmittance characteristic of 1 is shown in FIG. 3) is output.

Next, the wide dynamic range color image generation processing by the digital camera to which the present invention is applied will be described with reference to the flowchart of FIG.

In step S1, the spectral correction section 52
Acquires the color / sensitivity mosaic image signal corresponding to the subject, which is generated by the imaging unit 51. In step S2, the spectral correction unit 52 sequentially determines all pixels of the color / sensitivity mosaic image signal as the target pixel one pixel at a time. In step S3, the spectral correction unit 52 determines whether the pixel of interest is a low sensitivity pixel. If it is determined that the pixel of interest is a low sensitivity pixel, the process proceeds to step S4.

In step S4, the spectral correction unit 52
Corrects the pixel of interest, which is a low-sensitivity pixel,
The exposure sensitivity ratio in the pixel of interest is unified to a fixed value. The spectral correction processing in step S4 will be described with reference to the flowchart in FIG.

In step S11, the local information calculating section 71 calculates the local information corresponding to the target pixel which is a low sensitivity pixel. In step S12, the sensitivity correction value calculation unit 72 calculates the sensitivity correction value corresponding to the target pixel which is a low sensitivity pixel by applying the local information calculated by the local information calculation unit 71 to a function prepared in advance. To do. In step S13, the sensitivity correction unit 73 corrects the target pixel based on the sensitivity correction value from the sensitivity correction value calculation unit 72 so that the exposure sensitivity ratio of the target pixel, which is a low sensitivity pixel, becomes a constant value. The process returns to step S5 in FIG.

In step S5, the spectral correction unit 52
Determines whether all the pixels of the color / sensitivity mosaic image signal have been determined as the target pixel. If it is determined that all the pixels of the color / sensitivity mosaic image signal have not been determined as the target pixel, the process returns to step S2, and the subsequent processes are repeated. Then, in step S5,
When it is determined that all the pixels of the sensitivity mosaic image signal have been determined as the target pixel, the processing proceeds to step S6. At this stage, all the low sensitivity pixels of the color / sensitivity mosaic image signal are corrected, and the exposure sensitivity ratios of all the pixels in all the pixels of the color / sensitivity mosaic image signal are set to a constant value.

In step S6, the wide dynamic range color image generation unit 53 determines all the pixels based on the color / sensitivity mosaic image output from the spectral correction unit 52 in which the exposure sensitivity ratios of all pixels are constant. Wide dynamic range R image signal in which pixels have only wide dynamic range R component, and all pixels are in wide dynamic range G
It produces a wide dynamic range G image signal having only components and a wide dynamic range B image signal in which all pixels have only wide dynamic range B components. The description of the wide dynamic range color image generation processing by the digital camera to which the present invention has been applied is completed.

Next, FIG. 14 shows a second configuration example of the spectral correction section 52 of FIG. The second configuration example of the spectral correction section 52 is a configuration example suitable when the spectral transmittance characteristic of the ND filter 61 is non-linear as shown in FIG.

Similar to the local information calculating unit 71 in FIG. 11, the local information calculating unit 81 is arranged in the vicinity of each low sensitivity pixel among the pixels of the color / sensitivity mosaic image input from the image pickup unit 51. The local information (color balance or brightness balance) is calculated using only the low sensitivity pixels, and the sensitivity correction unit 8
Output to 2.

The sensitivity correction unit 82 applies the local information corresponding to the low-sensitivity pixel from the local information calculation unit 81 to the look-up table (LUT) 83 to obtain the corresponding sensitivity correction value, specifically, the pixel concerned. The ratio of low-sensitivity exposure to high-sensitivity exposure (Xd / Xb) (X is either R, G, or B) is acquired. Further, the sensitivity correction unit 82 corrects each low-sensitivity pixel based on the sensitivity correction value acquired from the lookup table 83 so that the exposure sensitivity ratio of each low-sensitivity pixel of the color / sensitivity mosaic image becomes a constant value. . Therefore, the sensitivity correction unit 82 captures a color / sensitivity mosaic image in which the exposure sensitivity ratios of all pixels are constant (that is, the spectral sensitivity characteristics of the ND filter 61 are imaged in the ideal case shown in FIG. 3). Color / sensitivity mosaic image).

When the local information calculating section 81 calculates the color balance as the local information, the lookup table 83 is
As shown in FIGS. 16 to 19, the sensitivity ratio corresponding to the input color balance is output. Here, FIG. 16 shows the sensitivity ratio Bd / to the color balance B / G.
The correspondence relationship of Bb is shown. FIG. 17 shows the color balance R
9 shows the correspondence of the sensitivity ratio Rd / Rb to / G. FIG. 18 shows the sensitivity ratio Gd / for the color balance B / G.
The correspondence of Gb is shown. FIG. 19 shows the color balance R
The correspondence relationship of the sensitivity ratio Gd / Gb to / G is shown.

When the local information calculator 81 calculates the luminance balance as the local information, the lookup table 83 outputs the sensitivity ratio corresponding to the input luminance lance, as shown in FIGS. It is done like this. Here, FIG. 20 shows the correspondence of the sensitivity ratio Bd / Bb to the brightness balance B / Y. FIG. 21 shows a correspondence relationship between the brightness balance R / Y and the sensitivity ratio Rd / Rb. FIG. 22 shows the correspondence relationship between the color balance G / Y and the sensitivity ratio Gd / Gb.

The look-up table 83 may be created from the result of actual image pickup, or the ND filter 6 may be used.
It may be created from the modeled spectral transmittance characteristics of No. 1 and the spectral distribution of the light source, but it is assumed that it is created before the processing of the sensitivity correction unit 82 is started.

Next, the spectral correction processing by the second example of the configuration of the spectral correction section 52 will be described with reference to FIG.
This spectral correction processing is an alternative to the spectral correction processing by the first configuration example of the spectral correction unit 52 described above with reference to FIG.

In step S21, the local information calculating section 81 calculates the local information corresponding to the target pixel which is a low sensitivity pixel. In step S22, the sensitivity correction unit 82
Inputs the local information calculated by the local information calculation unit 811 into the lookup table 83 to obtain the corresponding sensitivity correction value. In step S23, the sensitivity correction unit 82
Corrects the pixel of interest based on the sensitivity correction value so that the exposure sensitivity ratio of the pixel of interest, which is a low sensitivity pixel, becomes a constant value. The process returns to step S5 in FIG.
Above, the description of the spectral correction processing by the second configuration example of the spectral correction unit 52 is completed.

As described above, the color / sensitivity mosaic image output by the spectral correction unit 52 has a constant exposure sensitivity ratio in all pixels regardless of the spectral sensitivity characteristic of the ND filter 61. .

Therefore, in the wide dynamic range color image generation unit 53 subsequent to the spectral correction unit 52, the input color / sensitivity mosaic image is imaged using the ND filter 61 having an ideal spectral transmittance characteristic. As a result, a wide dynamic color image having uniform sensitivity of all pixels and excellent color reproducibility is generated.

In addition to the digital camera as in the present embodiment, the present invention is applicable to any apparatus that executes image processing, such as a personal computer that operates as an image processing apparatus by executing a dedicated application program. It is possible to

The series of processes described above can also be executed by hardware.

In the present specification, the steps for describing the program recorded on the recording medium are not limited to the processing performed in time series according to the order described, but may be performed in parallel if they are not necessarily performed in time series. Alternatively, it also includes processes that are individually executed.

[0086]

As described above, according to the present invention, the problem caused by the spectral transmittance of the ND filter in the color / sensitivity mosaic image is corrected, and the sensitivity of all pixels is uniform and the color reproducibility is improved. It is possible to generate an excellent wide dynamic color image.

[Brief description of drawings]

FIG. 1 is a diagram showing a spectral sensitivity characteristic of a color CCD.

FIG. 2 is a diagram showing a spectral distribution of incident light.

FIG. 3 is a diagram showing an ideal spectral transmittance characteristic of an ND filter.

FIG. 4 is a diagram showing a spectral transmittance characteristic that is linear with respect to the wavelength of an ND filter.

FIG. 5 is a diagram showing a non-linear spectral transmittance characteristic of an ND filter.

FIG. 6 is a block diagram showing a configuration example of a digital camera to which the present invention has been applied.

7 is a block diagram showing a configuration example of a signal processing unit 20 of FIG.

FIG. 8 is a functional block diagram of a digital camera to which the present invention has been applied.

9 is a diagram showing a configuration example of a CCD 16 of FIG.

FIG. 10 is a diagram showing an example of a color / sensitivity mosaic image.

11 is a block diagram showing a first configuration example of a spectral correction unit 52 in FIG.

FIG. 12 is a flowchart illustrating a wide dynamic range color image generation process of a digital camera to which the present invention has been applied.

FIG. 13 is a spectrum correction unit 5 in step S4 of FIG.
6 is a flowchart illustrating a spectral correction process according to the first configuration example of FIG.

14 is a block diagram showing a second configuration example of the spectral correction unit 52 in FIG.

15 is a spectrum correction unit 5 in step S4 of FIG.
6 is a flowchart illustrating a spectral correction process according to the second configuration example of FIG.

FIG. 16: Sensitivity ratio Bd / Bb to color balance B / G
Shows the correspondence relationship of.

FIG. 17: Sensitivity ratio Rd / Rb to color balance R / G
Shows the correspondence relationship of.

FIG. 18: Sensitivity ratio Gd / Gb to color balance B / G
Shows the correspondence relationship of.

FIG. 19: Sensitivity ratio Gd / Gb to color balance R / G
Shows the correspondence relationship of.

FIG. 20: Sensitivity ratio Bd / B to brightness balance B / Y
The correspondence relationship of b is shown.

FIG. 21: Sensitivity ratio Rd / R to brightness balance R / Y
The correspondence relationship of b is shown.

FIG. 22: Sensitivity ratio Gd / Gb to color balance G / Y
Shows the correspondence relationship of.

[Explanation of symbols]

16 CCD, 20 signal processing unit, 31 image signal processing unit, 40 CPU, 43 flash memory, 51
Imaging unit, 52 Spectral correction unit, 53 Wide dynamic range color image generation unit, 61 ND filter,
62 color filter, 71 local information calculation unit, 72 sensitivity correction value calculation unit, 73 sensitivity correction unit, 81 local information calculation unit, 82 sensitivity correction unit, 83 lookup table

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Tomoo Mitsunaga             1-14-10 Higashi Gotanda, Shinagawa-ku, Tokyo Stock             Ceremony company Sony Kihara Laboratory F-term (reference) 5B057 BA11 CA01 CA08 CA16 CB01                       CB08 CB16 CE06 CE11 CE17                       CH01 CH07 CH11                 5C065 AA03 BB01 CC01 DD02 DD17                       EE06 EE17

Claims (13)

[Claims] 1. A color image captured using a color filter for each pixel and a transmission limiting filter for limiting the transmission of incident light.
In an image processing device for generating a wide dynamic range color image based on a sensitivity mosaic image, a detection for detecting a low sensitivity pixel imaged at a low sensitivity by the transmission limiting filter among the pixels of the color / sensitivity mosaic image Means, calculating means for calculating local information corresponding to the low-sensitivity pixels detected by the detecting means, determination means for determining a sensitivity correction value corresponding to the low-sensitivity pixels based on the local information, An image processing apparatus comprising: a correction unit that corrects the low-sensitivity pixel based on a sensitivity correction value. 2. The calculation means, as the local information,
The image processing apparatus according to claim 1, wherein a local color balance corresponding to the low-sensitivity pixel detected by the detection unit is calculated. 3. The calculation means, as the local information,
The image processing apparatus according to claim 2, wherein the ratio of the R component to the G component or the ratio of the B component to the G component, which corresponds to the low-sensitivity pixel detected by the detection unit, is calculated. . 4. The calculation means, as the local information,
The image processing apparatus according to claim 1, wherein a local brightness balance corresponding to the low-sensitivity pixel detected by the detection unit is calculated. 5. The image processing according to claim 1, wherein the determining unit applies the local information to a predetermined function to calculate a sensitivity correction value corresponding to the low-sensitivity pixel. apparatus. 6. The determining means applies the local information to a lookup table to obtain a sensitivity correction value corresponding to the low-sensitivity pixel.
The image processing device according to item 1. 7. The image processing apparatus according to claim 1, wherein the transmission limiting filter is an ND filter. 8. The color filters are red, green,
The image processing apparatus according to claim 1, wherein the image processing apparatus is composed of three primary colors of blue and blue. 9. The color filters are yellow, magenta,
The image processing apparatus according to claim 1, wherein the image processing apparatus is configured by three complementary colors of cyan and cyan. 10. The image processing apparatus according to claim 1, wherein the color filter includes three complementary colors of yellow, magenta, and cyan, and four colors of green. 11. An image of an image processing device for generating a wide dynamic range color image based on a color / sensitivity mosaic image captured using a pixel-by-pixel color filter and a transmission limiting filter for limiting transmission of incident light. In the processing method, a detection step of detecting a low-sensitivity pixel imaged at a low sensitivity by the transmission limiting filter among the pixels of the color / sensitivity mosaic image, and the low-sensitivity pixel detected in the processing of the detection step A calculating step of calculating corresponding local information, a determining step of determining a sensitivity correction value corresponding to the low sensitivity pixel based on the local information, and a correcting step of correcting the low sensitivity pixel based on the sensitivity correction value An image processing method comprising: 12. A program for generating a wide dynamic range color image based on a color / sensitivity mosaic image captured using a color filter for each pixel and a transmission limiting filter for limiting the transmission of incident light, A detection step of detecting low-sensitivity pixels imaged at low sensitivity by the transmission limiting filter among the pixels of the color / sensitivity mosaic image, and local information corresponding to the low-sensitivity pixels detected in the processing of the detection step. A determining step of determining a sensitivity correction value corresponding to the low sensitivity pixel based on the local information, and a correction step of correcting the low sensitivity pixel based on the sensitivity correction value. A recording medium having a computer-readable program recorded thereon. 13. A computer for generating a wide dynamic range color image based on a color / sensitivity mosaic image captured using a pixel-by-pixel color filter and a transmission limiting filter for limiting transmission of incident light, A detection step of detecting low-sensitivity pixels of the pixels of the sensitivity mosaic image that are imaged at low sensitivity by the transmission limiting filter; and calculating local information corresponding to the low-sensitivity pixels detected in the processing of the detection step. A program for executing a calculation step, a determination step for determining a sensitivity correction value corresponding to the low sensitivity pixel based on the local information, and a correction step for correcting the low sensitivity pixel based on the sensitivity correction value.