JP2004056568A - Image composing method and image pickup device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a white-balanced image having a wide dynamic range by composing of a high sensitive image and a low sensitive image. <P>SOLUTION: In composing images of high output image data and low output image data, composite image data are obtained by multiplying the total gain corresponding to a scene by added data of the high output image data and the low output image data. The total gain is suitably multiplied in the range where the high output image data exceeds a certain value. The range is a range where the value of the total gain p exceeds the value of an optional number α-a coefficient value k × (high output image data high after gamma correction/threshold th), wherein, suitably, if the coefficient value k = 0.2, and the optional number α = 1, p = 0.8 in a scene with high contrast, p = 0.86 in a scene on a cloudy day or in shade, and p = 0.9 in a scene in an indoor fluorescent lamp. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image synthesizing method and an imaging apparatus, and more particularly to an image synthesizing method and an imaging apparatus capable of obtaining an image signal having a wide dynamic range while maintaining white balance.
[0002]
[Prior art]
For example, when an indoor landscape is imaged with an imaging device such as a digital still camera, the image of the subject present in the room is well reflected, but the blue sky seen from the window is blown out, resulting in an overall unnatural image. It may become. This is because the dynamic range of the image is narrow, and in order to solve this problem, conventionally, the dynamic range of the image is widened by capturing and synthesizing two images.
[0003]
For example, the first short-time exposure image (low-sensitivity image) is taken with the high-speed shutter turned off, and then the second long-time exposure image (high-sensitivity image) is taken with the low-speed shutter turned off. By synthesizing the two images, the scenery outside the window shown in the low-sensitivity image overlaps the high-sensitivity image in which the indoor scenery is well reflected.
[0004]
In the conventional imaging device described in Japanese Patent Application Laid-Open No. 2000-307963, when two images are combined, a moving subject portion does not match exactly between a low-sensitivity image and a high-sensitivity image, so a mask is used. For each part, the high-sensitivity image and the low-sensitivity image are replaced and image synthesis is performed.
[0005]
[Problems to be solved by the invention]
In the above-described conventional technology, the image signals of two captured images are combined using a mask, but the white balance deviation is not taken into consideration. For this reason, there is a problem in that the white balance differs between the high-sensitivity image and the low-sensitivity image in the composite image, resulting in a composite image having a sense of incongruity depending on the shooting scene.
[0006]
In recent years, for example, both a high-sensitivity pixel and a low-sensitivity pixel are mounted on a solid-state imaging device, and a high-sensitivity image (hereinafter, also referred to as a high-output image) captured with a high-sensitivity pixel and a low-sensitivity image captured with a low-sensitivity pixel. An image pickup apparatus such as a digital still camera that outputs a single image data by combining the image data (also referred to as a low output image) has been proposed. There is a need to solve technical problems.
[0007]
An object of the present invention is to provide an image composition method and an imaging apparatus that can synthesize and output an image with a wide dynamic range while maintaining white balance.
[0008]
[Means for Solving the Problems]
An image composition method that achieves the above object is an image composition method for image composition of high output image data and low output image data. In the image composition method, a total gain corresponding to a scene is added to the addition data of the high output image data and the low output image data. It is characterized in that it is multiplied to obtain composite image data. With this configuration, it is possible to synthesize an image with a wide dynamic range with white balance.
[0009]
Preferably, in a range where the high output image data exceeds a certain value, the sum of the high output image data and the low output image data is multiplied by a total gain corresponding to the scene. The range exceeding a certain value is a range in which the value of the total gain p exceeds a value of an arbitrary number α−coefficient value k × (high output image data high / threshold th after gamma correction), Further, the coefficient value k = 0.2, the arbitrary number α = 1, p = 0.8 for a high-contrast scene, p = 0.86 for a cloudy or shaded scene, In a scene under a fluorescent lamp, p = 0.9. Thereby, it is possible to obtain a good composite image suitable for the scene.
[0010]
An imaging apparatus that achieves the above object is an imaging apparatus that synthesizes and outputs high-output image data and low-output image data, and adds a total gain corresponding to the scene to the addition data of the high-output image data and the low-output image data. Image combining means for multiplying and outputting as composite image data is provided. With this configuration, it is possible to synthesize and output an image having a wide dynamic range with white balance.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0012]
FIG. 1 is a configuration diagram of a digital still camera according to an embodiment of the present invention. In this embodiment, a digital still camera will be described as an example. However, the present invention can be applied to other types of imaging devices such as a digital video camera. Further, the image synthesis processing of the present embodiment is executed by software by the digital signal processing unit 26 described later, but this can also be realized by a hardware circuit.
[0013]
The digital still camera shown in FIG. 1 includes a photographic lens 10, a solid-state image sensor 11, a diaphragm 12 provided between them, an infrared cut filter 13, and an optical low-pass filter 14. The CPU 15 that controls the entire digital still camera controls the light emitting unit 16 and the light receiving unit 17 for flash, controls the lens driving unit 18 to adjust the position of the photographing lens 10 to the focus position, and stops the aperture driving unit. The opening amount of the diaphragm 12 is controlled via 19 to adjust the exposure amount to an appropriate exposure amount.
[0014]
Further, the CPU 15 drives the solid-state image sensor 11 via the image sensor driving unit 20 and outputs the subject image captured through the photographing lens 10 as a color signal. In addition, a user instruction signal is input to the CPU 15 through the operation unit 21, and the CPU 15 performs various controls according to the instruction.
[0015]
The electric control system of the digital still camera includes an analog signal processing unit 22 connected to the output of the solid-state imaging device 11, and an A / D conversion that converts RGB color signals output from the analog signal processing unit 22 into digital signals. The circuit 23 is provided and these are controlled by the CPU 15.
[0016]
Further, the electric control system of this digital still camera includes a memory control unit 25 connected to the main memory 24, a digital signal processing unit 26 described later in detail, and compresses the captured image into a JPEG image and decompresses the compressed image. Mounted on the back side of the camera, the compression / decompression processing unit 27, the integration unit 28 for integrating the photometric data and adjusting the white balance gain, the external memory control unit 30 to which the removable recording medium 29 is connected. And a display control unit 32 to which the liquid crystal display unit 31 is connected. These are connected to each other by a control bus 33 and a data bus 34, and are controlled by a command from the CPU 15.
[0017]
The digital signal processing unit 26, the analog signal processing unit 22, the A / D conversion circuit 23, and the like shown in FIG. 1 can be mounted on the digital still camera as separate circuits. It is preferable to manufacture on the same semiconductor substrate using LSI manufacturing technology to form one solid-state imaging device.
[0018]
FIG. 2 is a pixel arrangement diagram of the solid-state imaging device 11 used in the present embodiment. The pixel 1 in the CCD portion that captures an image with a wide dynamic range has a pixel arrangement described in, for example, Japanese Patent Laid-Open No. 10-136391, and each pixel in the odd row is horizontally aligned with each pixel in the even row. A vertical transfer path (not shown) that is arranged with a ½ pitch shift and transfers signal charges read from each pixel in the vertical direction is meandered so as to avoid each pixel in the vertical direction. ing.
[0019]
Each pixel 1 according to this embodiment is divided into a low-sensitivity pixel 2 occupying about 1/5 of the area of the pixel 1 and a high-sensitivity pixel 3 occupying the remaining about 4/5 in the illustrated example. The signal charge of each low-sensitivity pixel 2 and the signal charge of each high-sensitivity pixel 3 can be distinguished and transferred to the vertical transfer path. It should be noted that the ratio and the position at which the pixel 1 is divided are determined by design, and FIG. 2 is merely an example.
[0020]
In the imaging apparatus according to the present embodiment, a low-sensitivity image (an image obtained by the low-sensitivity pixel 2) and a high-sensitivity image (an image obtained by the high-sensitivity pixel 3) are simultaneously acquired by one imaging, and each image is acquired. Are sequentially read out from the pixels 2 and 3, and are synthesized and output as will be described in detail later.
[0021]
The solid-state imaging device 11 has been described by taking a CCD having a honeycomb pixel arrangement as shown in FIG. 2 as an example, but may be a Bayer CCD or a CMOS sensor.
[0022]
FIG. 3 is a detailed configuration diagram of the digital signal processing unit 26 shown in FIG. The digital signal processing unit 26 employs a logarithmic addition method in which high-sensitivity image signals and low-sensitivity image signals are subjected to gamma correction and then subjected to addition processing, and is output from the A / D conversion circuit 23 shown in FIG. An offset correction circuit 41a that takes an RGB color signal that is a digital signal of a high-sensitivity image and performs an offset process, a gain correction circuit 42a that white balances an output signal of the offset correction circuit 41a, and a color signal after gain correction A gamma correction circuit 43a that performs gamma correction, an offset correction circuit 41b that performs an offset process by taking in RGB color signals that are digital signals of low-sensitivity images output from the A / D conversion circuit 23 shown in FIG. A gain correction circuit 42b for white balance of the output signal of the correction circuit 41b, and a color signal after gain correction And a gamma correction circuit 43b for performing gamma correction for. When linear matrix processing or the like is performed on the signal after offset correction, it is performed between the gain correction circuits 42a and 42b and the gamma correction circuits 43a and 43b.
[0023]
The digital signal processing unit 26 further captures both output signals of the respective gamma correction circuits 43a and 43b and performs an image composition processing circuit 44 for performing image composition processing as will be described in detail later, and an RGB color signal after image composition. An RGB interpolation calculation unit 45 that obtains RGB three-color signals at each pixel position by interpolation calculation, an RGB / YC conversion circuit 46 that obtains the luminance signal Y and the color difference signals Cr and Cb from the RGB signals, and the luminance signal Y and the color difference signal A noise filter 47 that reduces noise from Cr and Cb, a contour correction circuit 48 that performs contour correction on the luminance signal Y after noise reduction, and a color difference matrix that multiplies the color difference signals Cr and Cb by color correction. And a color difference matrix circuit 49 to be performed.
[0024]
The RGB interpolation calculation unit 44 is not required if it is a three-plate image sensor, but the solid-state image sensor 11 used in this embodiment is a single-plate solid-state image sensor, and R, G, B from each pixel. Therefore, in a pixel that outputs no color, that is, a pixel that outputs R, the level of the G and B color signals at this pixel position indicates the G and B signals of the surrounding pixels. Is obtained by interpolation calculation.
[0025]
The above-described image synthesis processing circuit 44 synthesizes the high sensitivity image signal output from the gamma correction circuit 43a and the low sensitivity image signal output from the gamma correction circuit 43b in units of pixels based on the following equation (1). Output.
[0026]
[Equation 1]
data = [high + MIN (high / th, 1) × low] × MAX [(−k × high / th) + α, p]
Here, high: data after gamma correction of high sensitivity (high output) image signal low: data after gamma correction of low sensitivity (low output) image signal p: total # gain (total gain)
k: coefficient th: threshold α: value determined by the scene (≈1)
It is.
[0027]
If the data after gamma correction is 8-bit data (256 gradations), the threshold value th is designated by, for example, “219” of the value 0 to 255 and the user of the digital still camera or the designer of the digital still camera. The value to be
[0028]
The first term of Equation 1 adds the low-sensitivity image data low to the high-sensitivity image data high when the high-sensitivity image data high exceeds the threshold th, and when the high-sensitivity image data high is less than the threshold th, A value obtained by multiplying the ratio of the high sensitivity image data high to the threshold th by the low sensitivity image data low is added to the high sensitivity image data high.
[0029]
In the present embodiment, the addition data obtained in the first term is not used as the synthesized image data as it is, but the second term (MAX [(− k × high / th) + α, p]) is added to the first term. A value obtained by multiplying is used as composite image data.
[0030]
In the second term, the coefficient “k” is preferably a value “0.2” in the solid-state imaging device 11 of the embodiment shown in FIG. When the signal charge saturation ratios of the high-sensitivity pixel 3 and the low-sensitivity pixel 2 are different as in the solid-state imaging device 11 shown in FIG. 2, the coefficient k can be conveniently obtained by the following equation 2.
[0031]
[Equation 2]
Coefficient k = 1−Sh / (Sh + Sl)
Here, Sh: signal charge saturation amount of high sensitivity pixel Sl: signal charge saturation amount of low sensitivity pixel
In the example shown in FIG. 2, the area ratio of the photodiode does not become the saturation ratio as it is, but it can be regarded as an area ratio for convenience, and when the above example is applied,
k = 1-4 / (4 + 1) = 1-0.8
= 0.2
It becomes.
[0033]
The solid-state imaging device having both high-sensitivity pixels and low-sensitivity pixels is not limited to that illustrated in FIG. 2, and for example, as shown in FIG. It is conceivable to change the aperture area of the microlens provided on top of the above and provide the high-sensitivity pixel 3 and the low-sensitivity pixel 2. In this case, the saturation amount of the signal charge of the high-sensitivity pixel and the low-sensitivity pixel is the same, and thus Equation 2 cannot be applied. However, the value of the coefficient k is obtained experimentally or from the opening area of a microlens or the like. Equation 1 can be applied by obtaining the coefficient value. The value of the coefficient k is a value that is determined by the configuration of the solid-state imaging device, and is not arbitrarily changed by the user, but is set to a fixed value when the imaging apparatus is shipped.
[0034]
In Equation 1, an experimentally determined value is used as the value of the total gain p in this embodiment. p is a gain with respect to the entire composite image data. In this embodiment, the dynamic range of the image is controlled by controlling the value of p. The smaller the value of the total gain p, the wider the dynamic range, and the larger the value of p, the narrower the dynamic range. Specifically, depending on the scene, p = 0.8 for high-contrast scenes (such as midsummer sunny weather), p = 0.86 for cloudy or shaded, and p = 0.9 for indoor fluorescent lighting. To change the value of p. Thereby, when the data after gamma correction is 8-bit data, the 8-bit gradation value can be used more effectively.
[0035]
The value of p may be that the digital still camera itself automatically determines and automatically sets the scene of the captured image based on the detection values of various sensors, and the user designates the scene type with the operation unit 21 shown in FIG. By doing so, the value of p may be set.
[0036]
FIG. 4 is a diagram showing how the dynamic range changes when the value of p is changed. When the total gain p is increased, the characteristic line a has a small dynamic range, and when the total gain p is decreased, the characteristic line a changes to a characteristic line B having a large dynamic range.
[0037]
In Equation 1, the value of α may be a value corresponding to the scene of the captured image, but may be a fixed value “1”.
[0038]
Thus, according to the present embodiment, since the high-sensitivity image data and the low-sensitivity image data are added and then multiplied by the total gain according to the scene, it is possible to generate an image with a wide white balance and a wide dynamic range. Become. In addition, since the logarithmic addition method is employed to combine the images after reducing the number of bits of the high-sensitivity image data and the low-sensitivity image data, the circuit scale can be reduced and the cost can be reduced.
[0039]
In the above-described embodiment, an example in which a high-output image (high-sensitivity image) and a low-output image (low-sensitivity image) captured by a digital still camera are combined in the digital still camera has been described. The high-sensitivity image data and the low-sensitivity image data are stored in a memory and taken out from the image pickup device, and the high-sensitivity image data and the low-sensitivity image data (CCD-RAW data) are read into a personal computer or the like and described in the above-described embodiment. The present invention can also be applied to the case where image synthesis processing similar to that performed by the digital signal processing unit 26 is performed, and a composite image having a wide dynamic range with white balance adjustment can be generated.
[0040]
In the embodiment described above, an image captured with low-sensitivity pixels is referred to as a low-sensitivity image, and an image captured with high-sensitivity pixels is referred to as a high-sensitivity image. However, the present invention is limited to combining images with different sensitivities. The present invention is not limited to this, and can also be applied to a case where a plurality of captured images that are images captured by the same pixel and have different aperture amounts are combined. For example, when a plurality of still images with high contrast are taken continuously with exposure, the image taken with a wide aperture aperture is the above-mentioned high output image because the output level from each pixel of the solid-state image sensor is high, An image picked up with a narrow aperture with a small opening amount is the above-described low output image because the output level from each pixel is lower than that of the high output image. The above embodiment can also be applied to the case where these two image data are combined.
[0041]
【The invention's effect】
According to the present invention, it is possible to synthesize an image with a wide dynamic range in accordance with a scene and with white balance.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a digital still camera according to an embodiment of the present invention.
2 is a diagram illustrating a pixel arrangement example of the solid-state imaging device illustrated in FIG. 1;
FIG. 3 is a detailed configuration diagram of a digital signal processing unit shown in FIG. 1;
FIG. 4 is a diagram illustrating a change in dynamic range.
FIG. 5 is a pixel arrangement diagram according to another embodiment of the solid-state imaging device.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Pixel 2 Low sensitivity pixel 3 High sensitivity pixel 10 Lens 11 Solid-state image sensor 15 CPU
26 Digital signal processing units 43a and 43b Gamma correction circuit 44 Image composition processing circuit A Characteristic line of large total gain (low dynamic range) B Characteristic line of small total gain (dynamic range large)

Claims (5)

In an image composition method for compositing high output image data and low output image data, the sum of the high output image data and low output image data is multiplied by the total gain corresponding to the scene to obtain composite image data. A characteristic image composition method. 2. The image composition method according to claim 1, wherein the sum of the high output image data and the low output image data is multiplied by a total gain corresponding to a scene within a range in which the high output image data exceeds a certain value. . The range in which the high output image data exceeds a certain value is a range in which the value of the total gain p exceeds the value of an arbitrary number α-coefficient value k × (high output image data high / threshold th after gamma correction). The image synthesizing method according to claim 2. The coefficient value k = 0.2, the arbitrary number α = 1, p = 0.8 for a high contrast scene, p = 0.86 for a cloudy or shaded scene, 4. The image composition method according to claim 3, wherein p = 0.9 is set in the lower scene. In an imaging device that synthesizes and outputs high-output image data and low-output image data, the sum of the high-output image data and low-output image data is multiplied by the total gain corresponding to the scene and output as composite image data An image pickup apparatus comprising an image composition unit.

Images