JP2003158669A - Image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device in which luminance inversion at a small area can be prevented to obtain a natural image while saving memory areas in the case of image pickup with a magnified dynamic range where a plurality of picked up original images with different exposures are composited into one image. SOLUTION: In a digital camera that composites two picked up original images with different exposures into one image to extend the dynamic range, a weight sum coefficient calculation circuit 202 receives a first original image signal S1 obtained by picking up the image at a first exposure via an LPF 201, a weight sum circuit 203 receives a second original image signal S2 obtained by picking up the image at a second exposure greater than the first exposure together with the signal S1, the weight sum coefficient calculation circuit 202 generates weight sum signals W1, W2 with respect to the signals S1, S2 on the basis of the low frequency component of the signal S1 and the weight sum circuit 203 composes the signals S1, S2 on the basis of the sum signals W1, W2.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a solid-state image pickup device such as an electronic still camera (digital camera) or a video camera, and particularly to an image pickup device having a wide dynamic range by combining a plurality of picked-up images having different exposures. Regarding

[0002]

2. Description of the Related Art In recent years, an image pickup device such as a digital camera has been developed which picks up an image of a subject with an image pickup device such as a CCD and records the picked up image as digital data. In this type of image pickup device, many attempts of dynamic range expansion image pickup have been proposed, and, for example, there is known one in which two (or more) original images obtained by image pickup with different exposures are combined. Among them, the following methods are known as methods of synthesizing processing.

(A) The two original image data are simply added and averaged (that is, one-to-one weighted synthesis).

(A) A dynamic range deviation area is obtained from the luminance data of one original image, and this deviation area is replaced with another original image data. This is a so-called inset combining process in which another image is embedded in a part of the image.

(C) From two original image data and exposure shift data, one wide dynamic range image (a multi-bit image having a bit number sufficiently larger than that of the original image so as to cover the entire range) A bit image) is generated, and various image processing (for example, blurring processing by a low-pass filter or the like is performed) is performed on the generated bit image) to generate a recording target image.

[0006]

In the above (c), since a wide dynamic range image (including all image information) is once generated and post-processing is performed, there is a possibility that the best processing can be performed in a sense. have. However, since it is essential to generate a wide dynamic range image in the middle, the calculation process for this is complicated, and in addition, it has a problem that a large memory area is required. This is not desirable when considering real-time processing, downsizing, and cost reduction. In addition, when the process of blurring an image is included, the resolution also deteriorates.

On the other hand, the above-mentioned (a) and (a) are essential in that they are easy to apply to a digital camera because they do not require the generation of an intermediate wide dynamic range image and the arithmetic processing is simple. Although they have merits, they have the following drawbacks in performance.

In (a), although the dynamic range is wide, the gradation is compressed, and the contrast is lowered when viewed as an image as it is. Alternatively, if it is premised on image correction at the time of viewing, quantization noise will increase. In (a), the phenomenon of reversal of the brightness (brightness) depending on the position (area) of the image may occur. That is, FIG. 5 (a)
As shown in, it is assumed that there is a region of brightness depending on the position, and the brightness is higher than the brightness. If this subject is imaged in the first exposure with the short exposure time T1 and the second exposure with the long exposure time T2, respectively, as shown in FIG. 5B, the luminance region has the imaging characteristics of the first exposure. The output V1 is included in the D range (effective luminance range) of the above, but deviates from the image pickup characteristic of the second exposure (saturation level W is output), while the luminance region is imaged by the second exposure. It is included in the characteristic D range and produces the output V2.
It deviates from the imaging characteristic of exposure (black level B is output). In such a case, the output signal obtained by simply synthesizing them has a higher output level in the area than the area shown in FIG. 5C.

Such a phenomenon can be said to be the original effect of expanding the dynamic range, and is not necessarily a defect depending on the pattern (region). For example, it is effective for a composition process for backlighting by the window (an effect similar to that of daytime sync photography using a strobe is obtained). However, a high-frequency and high-contrast subject (an edge portion or a striped pattern like a resolution chart) causes an unnatural image or a false signal-like image with an inverted pattern.

Further, in any of the above-mentioned methods (a), (b), and (c), the exposure timings of the two image pickups cannot theoretically be the same, so that they are caused by camera shake (camera shake) and movement of the subject. If the image blur occurs, the positions of the two original images are misaligned, which causes a problem that the image processing fails in a region where the information of the two images does not correspond to each other and the image quality is significantly deteriorated.

The present invention has been made in consideration of the above circumstances, and an object thereof is to provide a method of directly processing an original image suitable for a camera, in which a brightness reversal phenomenon in a small area is saved while saving memory. It is an object of the present invention to provide an image pickup apparatus that can prevent the occurrence of the above-mentioned problem and can obtain a natural image with little deterioration in image quality even if there is some image blur.

[0012]

(Structure) In order to solve the above problems, the present invention adopts the following structure.

That is, according to the present invention, an image pickup system capable of picking up an image of a subject and outputting a predetermined image signal, exposure control means for controlling exposure to the image pickup system, and different exposure for the image pickup system by the exposure control means. And an image generating unit that generates a single combined image signal by combining a plurality of original image signals obtained by giving an amount, and the image generating unit is configured to generate an image of the plurality of original image signals. Weighting calculation means for calculating a weighting coefficient for each original image signal based on a low frequency component for one of the image pickup luminance regions, and weighting for adding each original image signal based on the weighting coefficient calculated by the weighting calculation means It is characterized in that it includes addition means.

Further, the present invention provides an image pickup system capable of picking up an image of a subject and outputting a predetermined image signal, exposure control means for controlling exposure to the image pickup system, and a plurality of output image signals output by the image pickup system. An image generation unit that generates one combined image signal that is combined as the original image signal information, and a plurality of original image signals that the image generation unit uses to generate the first exposure signal for the imaging system by the exposure control unit. The exposure control means gives a second exposure amount different from the first exposure amount to the first original image signal, which is an output image signal corresponding to the amount, and gives it to the imaging system. And an image pickup control means for controlling so as to obtain a second original image signal which is a corresponding output image signal, wherein the image generation means is adapted to obtain an image pickup luminance region of the first original image signal. Low frequency composition of signal Based on the weight calculating means for calculating a weighting factor for the first and second original image signals,
It is characterized by including a weighted addition means for adding the first and second original image signals based on the weighting coefficient calculated by the weighting calculation means.

According to the present invention, the first original image signal obtained corresponding to the first exposure amount and the second original image signal different from the first exposure amount are different.
An image generation method for generating a combined image signal obtained by combining these first and second original image signals based on a second original image signal obtained corresponding to the exposure amount of A weighting calculation step of calculating a predetermined weighting coefficient based on a low frequency component of the imaged luminance range of the image signal, and first and second original images based on the weighting coefficient calculated by the weighting calculation step. A weighted addition step of adding the signals to generate the composite image signal.

(Operation) According to the present invention, in dynamic range expansion image pickup in which one image is combined from a plurality of image pickup original images with different exposures, at least image generating means including weighting calculation means and weighted addition means is provided, and at least A weighted signal for combination is generated based on the low frequency component of one original image, and a plurality of captured original images are weighted and combined.

As a result, a large weight can be generated on the average of the area, so that an effective dynamic range expansion (which is close to the conventional b) can be realized while brightness inversion is included, while an extreme weight is locally applied. Since the occurrence is prevented, the reversal of the luminance can be prevented. Further, the weighted combination based on the low-frequency component gradually (gradually) switches (embeds) information between different original images, so that the gradual boundary region acts as a buffer region for image blur. , Even if there is some image blur, the image quality does not deteriorate significantly. That is, in a method of directly processing an original image suitable for a digital camera (memory can be saved and relatively simple processing), a natural image that does not cause a brightness inversion phenomenon in a small area and has high tolerance for image blur Can be obtained.

[0018]

DETAILED DESCRIPTION OF THE INVENTION The details of the present invention will be described below with reference to the illustrated embodiments.

FIG. 1 is a block diagram showing the basic arrangement of a digital camera according to an embodiment of the present invention.

In the figure, 101 is a lens system including various lenses, and 102 is a lens driving mechanism for driving the lens system 101. An exposure control mechanism 103 controls the exposure, and includes an aperture and a shutter. Reference numeral 104 is a filter system including a low-pass filter and an infrared cut filter, and 105 is a CCD color image pickup device. A light beam that has passed through the exposure control mechanism 103 is guided to the image pickup device 105 via the filter 104. Therefore, an image corresponding to the subject is formed on the image sensor 105.

107 is a gain control amplifier, A
This is a pre-process circuit including a D / D converter and the like, and an image pickup signal obtained by the image pickup device 105 is input to the preprocess circuit 107, and a digitized pixel signal is output from the preprocess circuit 107. Reference numeral 108 denotes a digital process circuit for performing color signal generation processing, matrix conversion processing, and various other digital processing. Color image data is generated by processing the digitized image signal in this digital process circuit 108. To be done. 109 is a card interface connected to the digital process circuit 108, and 110 is a CF (Co
mpact Flash Memory Card) and a memory card such as smart media, and 111 is an LCD image display system. The memory card 110 stores color image data, and the LCD display system 111 displays color image data.

Further, in the figure, 112 is a system controller (CPU) for controlling each part in an integrated manner, 113 is an operation switch system including various SWs, 114 is an operation display for displaying an operation state, a mode state and the like. Reference numeral 115 is a lens driver for controlling the lens driving mechanism 102, 116 is a strobe as a light emitting means, 117 is an exposure control driver for controlling the exposure control mechanism 103 and the strobe 116, and 118 is various setting information and the like. 3 shows a non-volatile memory (EEPROM) for doing so.

In the digital camera of this embodiment, the system controller 112 centrally controls all the operations, and in particular, the shutter device included in the exposure control mechanism 103 and the CCD image pickup device 10 by the CCD driver 106.
5 is controlled to perform exposure (charge accumulation) and signal reading, and the signal is taken into the digital process circuit 108 via the pre-process circuit 107, subjected to various signal processing, and then stored via the card interface 109 to the memory. It is designed to be recorded on the card 110. Note that C
The CD image sensor 105 is, for example, an interline type having a vertical overflow drain structure and is progressive (sequential).
It is a scanning type.

FIG. 2 is a block diagram showing the arrangement of the image generation mechanism included in the digital process circuit 108.

The first image signal S1 is input to a weighting coefficient calculation circuit 202 via a low pass filter (LPF) 201, and the weighting coefficient calculation circuit 202 calculates weighting coefficients in accordance with the LPF output as described later. It In addition, the first and second image signals S1 and S2 are input to the weighted addition circuit 203, and based on the weighting coefficient determined by the weighting coefficient calculation circuit 202, these image signals S1 and S2.
2 is to be synthesized.

By the way, the image pickup range is not simply determined by the image pickup element alone, but is decided by the whole including the signal processing of the image pickup apparatus using the image pickup element. On the luminance side, the noise level of the output of the image pickup device in the state of being incorporated in the image pickup device becomes a limit, so that an image pickup range exceeding at least that cannot be obtained. Conventionally, the photoelectric conversion characteristics of an image pickup device when an image pickup apparatus using a general image pickup device is configured have been as schematically shown in FIG. 3, for example.

In this figure, the horizontal axis represents the amount of incident light and the vertical axis represents the signal level logarithmically. In the figure, UL indicates a high luminance side limit level, LL indicates a low luminance side limit level, and NL indicates a noise level. U
L is a level substantially corresponding to the saturation level of the image sensor, while LL is not the noise level NL itself, but a predetermined limit S / that can be appreciated even when coexisting with noise.
It is defined as the signal level with N ratio. Alternatively, the visual evaluation may be performed based on the criterion of “black level is lower than this level”, and LL may be determined as the limit level. Then, the effective luminance range is between UL and LL, and the difference DR (UL-LL-on the logarithmic axis) becomes the imaging range.

The above-mentioned DR varies depending on the design and manufacture of the image pickup device, but in many cases it is about 5 to 6 EV (30 to 36 dB), and the digital camera of this embodiment is such a one. Further, the image pickup device is assumed to be monochrome in order to simplify the explanation.

Although general processing can be used to drive the camera, in view of minimizing the time difference between the two original images, for example, Japanese Patent Application No. 2000-07713 by the present applicant.
It is particularly desirable to use the driving method described in No. 6 in which a CCD for progressive scanning (full frame reading) and a mechanical shutter are combined to capture two images without a time lag.

In the camera using the CCD image pickup device 105 as described above, signals are read out and recorded in the memory card 110 or displayed on the LCD image display system 111 as in the conventional camera. . The video signal generation processing is different from the conventional one, but it is assumed that the processing is exactly the same as that for the conventionally known monochrome CCD image pickup device except the part directly related to the present invention described below. Then, the video signal generation processing related to the image synthesis is performed under the control of the system controller 112 by the digital process 108 having the image generation mechanism shown in FIG.

The photoelectric conversion characteristics of the pixel information signal of the original image read from the image pickup device 105 are as shown in FIG. 4 for the first and second original images. That is,
The first original image is exposed with a shorter exposure time and the second original image is exposed with a longer exposure time. The exposure is performed in parallel to the left and right by an amount corresponding to the difference in the exposure time.

The pixel signal generation processing of the camera of this embodiment is performed as follows by the image generation mechanism of FIG.
Here, as described in FIG. 2, the signal of the first original image with the smaller exposure amount (faster shutter speed) is S1, and the signal of the second original image with the larger exposure amount (lower shutter speed) is S2
And

(1) LP for the signal S1 of the first original image
F processing is performed. The result is LS1.

(2) Calculate the weighting coefficient.

As f (LS1) ≡ (LS1−LL) / {k × (UL−LL)} (where k is an arbitrary constant, an example value is 0.18) (3) Perform image generation calculation (weighted addition).

Sout = w1.times.S1 + w2.times.S2 LPF processing corresponds to obtaining a local average value by blurring the image. Then, in the sufficiently bright area (f (LS1) ≧ 1) on average of the first original image, w1
Becomes 1 and w2 becomes 0. This means that only the signal S1 of the first original image is used for such an area. The first original image is sufficiently dark on average (f (LS1) ≦
In the region (which is 0), w1 becomes 0 and w2 becomes 1.
This is the signal S of the second original image for such an area.
This means that only 2 is used. That is, it means that the same processing as so-called inset synthesis is performed, in which the dark portion of the first original image is replaced with the second original image.

When the LPF output of the first original image is at an intermediate level, that is, in the region of 0 <f (LS1) <1, the first original image and the second original image are weighted by w1 and w2, respectively, and image combination is performed. Done. Therefore, in the intermediate region of the LPF output, the luminance characteristic for the first original image with a small exposure amount and the luminance characteristic for the second original image with a large exposure amount are continuously and progressively connected, which results in local luminance. Inversion will be suppressed.

The pixel signal obtained as a result of the above-described processing is processed by the circuit in the subsequent stage similarly to the conventional monochrome image pickup signal, and finally recorded in the memory card 110 or LC.
It is displayed on the D image display system 111. Here, since the second original image having a long exposure time is strongly affected by smear, the first original image having a short exposure time is better used for calculating the weighting coefficient.

Although the case of monochrome has been described above, the present invention can be applied in the same manner to the case of color.
That is, as an example, a processing example in the case where an RGB component signal (a state in which each pixel has all RGB information) after color signal synchronization is used in a three-plate image pickup system or a single-plate image pickup system will be shown.

In the LPF processing of (1), as S1,
The luminance signal Y1 calculated by the luminance matrix Y Y = 0.299R + 0.587G + 0.114B (the G1 signal may be substituted) is used.

In the image generation operation of (3), similar weighted addition is performed for each of RGB.

As described above, according to the present embodiment, by synthesizing the first original image and the second original image, the dynamic range for the image pickup signal is expanded. At that time, since the weighting coefficient is calculated based on the low frequency component of the original image, extreme weighting does not occur locally. Therefore, it is possible to prevent a defect that a pseudo signal such as inversion occurs with respect to a black and white edge portion or a pattern like a resolution chart.

Further, since frequency processing (blurring) is not performed on the image signal itself, the problem of deterioration of resolution does not occur. Further, since it is not necessary to generate a wide dynamic range image in the middle, it does not cause a complicated arithmetic process or an increase in memory area. Therefore, it is effective for real-time processing, downsizing, cost reduction, etc. in a digital camera.

The present invention is not limited to the above embodiment. The setting of the shift amount of the exposure time in the first image and the second image is arbitrary, and may be a setting that eliminates the common luminance range as long as it is set according to the luminance distribution of the subject. Further, the exposure level may be changed by a parameter (for example, diaphragm) other than the exposure time. Furthermore, although two images are used in the embodiment, three images are used.
More than one image may be used. For example, using an ABC3 image with large, medium, and small exposure levels, the same process as above is performed with the BA image based on the low-luminance deviation degree of the low-frequency component of the intermediate B image, and the BC based on the high-luminance deviation degree. The same process as described above may be performed on the image (actually, these processes are performed at once).

In addition, various modifications can be made without departing from the scope of the present invention.

[0046]

As described above in detail, according to the present invention, in dynamic range expansion image pickup in which one image is combined from a plurality of image pickup original images having different exposures, the low frequency component of at least one original image is used. By generating a weighted signal for synthesis by using the above method, it is possible to prevent a luminance inversion phenomenon in a small area while saving memory, and obtain a natural image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a basic configuration of a digital camera according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration example of an image generation mechanism used in the digital camera of FIG.

FIG. 3 is a diagram showing photoelectric conversion characteristics of an image sensor.

FIG. 4 is a diagram showing photoelectric conversion characteristics with respect to first and second original image signals.

FIG. 5 is a diagram for explaining a conventional problem.

[Explanation of symbols]

101 ... Lens system 102 ... Lens drive mechanism 103 ... Exposure control mechanism 104 ... Filter 105 ... CCD color image sensor 106 ... CCD driver 107 ... Preprocess circuit 108 ... Digital process circuit 109 ... Card interface 110 ... Memory card 111 ... LCD image display system 112 ... System controller (CPU) 113 ... Operation switch system 114 ... Operation display system 115 ... Lens driver 116 ... Strobe 117 ... Exposure control driver 118 ... Nonvolatile memory (EEPROM)

Claims (5)

[Claims] 1. An imaging system capable of imaging a subject and outputting a predetermined image signal, exposure control means for controlling exposure to the imaging system, and different exposure amounts given to the imaging system by the exposure control means. And an image generating unit that generates one combined image signal obtained by combining a plurality of original image signals obtained respectively, wherein the image generating unit is one of the plurality of original image signals. A weighting calculation means for calculating a weighting coefficient for each original image signal based on a low frequency component for one imaging luminance region,
An image pickup apparatus comprising a weighted addition means for adding each original image signal based on the weighting coefficient calculated by the weighting calculation means. 2. An image pickup system capable of picking up an image of a subject and outputting a predetermined image signal, exposure control means for controlling exposure to the image pickup system, and a plurality of output image signals output by the image pickup system as original image signals. Image generation means for generating one combined image signal combined as information, and a plurality of original image signals used for generation by the image generation means, the exposure control means giving a first exposure amount to the imaging system. A first original image signal which is an output image signal corresponding to this is obtained, and a second exposure amount different from the first exposure amount is given to the imaging system by the exposure control means, and an output corresponding to this is given. An image pickup control means for controlling so as to obtain a second original image signal which is an image signal, and the image generation means, wherein the image generation means lowers the signal relative to the image pickup luminance range of the first original image signal. Based on frequency components Weighting calculation means for calculating a weighting coefficient for the first and second original image signals, and weighted addition means for adding the first and second original image signals based on the weighting coefficient calculated by the weighting calculation means. An imaging device characterized in that it is a digital camera. 3. The image pickup apparatus according to claim 2, wherein the second exposure amount is larger than the first exposure amount. 4. The weighting calculation means, when the low frequency component of the first original image signal is equal to or higher than a predetermined level H, sets the weighting coefficient to the first original image signal to 1 and the weighting coefficient to the second original image signal. If the coefficient is 0 and the low frequency component of the first original image signal is below a predetermined level L (L <H), the weighting coefficient for the first original image signal is 0, and the weighting coefficient for the second original image signal is Is 1, and the low-frequency component of the first original image signal is L
In the case of the level to the H level, the first original image signal is k, and the second original image signal is 1-k (k is closer to 1 as it approaches the H level and closer to 0 as it approaches the L level). The image pickup device according to claim 2, wherein the image pickup device gives a weighting coefficient. 5. A first original image signal obtained corresponding to a first exposure amount and a second original image signal obtained corresponding to a second exposure amount different from the first exposure amount. Based on these first
An image generating method for generating a combined image signal by combining a first original image signal and a second original image signal, wherein a predetermined weighting coefficient is calculated based on a low frequency component of the first original image signal with respect to an image pickup luminance region. A weighting calculation step, and a weighted addition step of generating the combined image signal by adding the first and second original image signals based on the weighting coefficient calculated by the weighting calculation step. Image generation method.