JP3914810B2 - Imaging apparatus, imaging method, and program thereof - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image pickup apparatus using an image pickup device such as a CCD, and more particularly to a technique for synthesizing a plurality of images for expanding a dynamic range.
[0002]
[Prior art]
In recent years, various attempts to obtain a captured image with a wide dynamic range by combining a plurality of images having different exposure amounts have been proposed.
[0003]
One of them is to generate an image of one wide dynamic range (hereinafter referred to as “D range”) from two original image data with different exposure amounts, and perform various image processing on the image and record it. A target image is generated. Here, the wide D range image generated in the intermediate stage is an image having a larger number of bits than the number of bits of the data of the original image so as to cover the entire range. In this way, there is a possibility that the best processing can be performed by performing various kinds of image processing using an image of a wide D range.
[0004]
Another method is to generate an image obtained by performing weighted synthesis by averaging two original image data for each pixel. This method has an advantage that it can be easily applied to an imaging apparatus because the arithmetic processing of the image is simple.
[0005]
Furthermore, one of the other methods is to replace the data in the area deviating from the range of the D range in the luminance data of the original image on one side with the data of the other original image. This method has an advantage that it can be easily applied to the imaging apparatus because the image calculation process is simple as in the above method.
[0006]
[Problems to be solved by the invention]
However, the method of creating a wide D-range image at an intermediate stage has a problem that the creation and subsequent processing are complicated, and a large memory area is required. Considering needs such as real-time processing, miniaturization, and cost reduction, it is not preferable.
[0007]
Even if image information of a wide D range can be obtained temporarily, if signal processing such as luminance compression is performed on an finally generated image having a narrow range, the image quality will eventually deteriorate. Become. Therefore, it cannot be said that a signal processing technique capable of obtaining sufficient image quality has been established from this viewpoint.
[0008]
In addition, in the averaging process, the D range is expanded but the gradation is compressed, and the contrast is lowered when viewed as an image as it is. On the other hand, when some kind of image correction is performed, there is a case where the quantization noise is increased and the image quality is deteriorated.
[0009]
Further, in the deviation data replacement process, an image to be adopted is switched depending on the luminance information, and therefore an unnatural image may be obtained depending on the luminance distribution of the subject image or the like. For example, a high-frequency and high-contrast image of an object such as a stripe pattern on a resolution chart will produce a false signal image with an inverted pattern.
[0010]
The present invention has been made in view of such circumstances, and provides an imaging apparatus, an imaging method, and a program thereof that can be easily applied to an imaging apparatus and can obtain a natural captured image with an expanded D range. The purpose is to do.
[0011]
[Means for Solving the Problems]
The present invention for solving the above-described problem is an index corresponding to chromaticity for each divided region formed of at least one pixel by processing each of the image signals of a plurality of original images obtained by giving different exposure amounts. Indexing means for calculating the image, selection means for selecting one image signal from among the image signals of the plurality of original images for each divided region based on the index, and generating one composite image signal from the selected image signal And an image generating unit that performs image generation.
[0012]
According to the present invention, in the imaging device according to the above-described invention, the selection unit compares each index for each divided region to determine an index having the largest absolute value, and determines an image signal of the original image that provides the index. The imaging device to be selected.
[0013]
In the imaging apparatus according to the invention described above, the indexing unit corresponds to chromaticity from the following equation using the luminance signal Y, the color difference signals RY and BY, and the predetermined value α. It is an imaging device that calculates an index C.
[0014]
C = {(R−Y) ² + (B−Y) ² } / (Y + α) ²
Further, according to the present invention, in the imaging device according to the above-described invention, the indexing unit includes a luminance signal Y of one image, color difference signals RY and BY, a luminance signal Y ^* of another image to be compared, a predetermined value This is an imaging device that calculates an index C corresponding to chromaticity from the following equation using the value α.
[0015]
C = {(R−Y) ² + (B−Y) ² } * (Y ^* + α) ²
Further, according to the present invention, in the imaging device according to the above-described invention, the indexing unit includes a luminance signal Y of one image, color difference signals RY and BY, a luminance signal Y ^* of another image to be compared, a predetermined value An index C corresponding to chromaticity is calculated from the following equation using the value α of the imaging device.
[0016]
C = {| R−Y | + | B−Y |} * (Y ^* + α)
Further, the present invention processes each of a plurality of image signals obtained by giving different exposure amounts, calculates an index corresponding to chromaticity for each divided region composed of at least one pixel, and based on the index In this imaging method, one image signal is selected from image signals of a plurality of original images for each divided region, and one composite image signal is generated from the selected image signal.
[0017]
The present invention also processes each of the image signals of a plurality of original images obtained by giving different exposure amounts to a computer, and calculates an index corresponding to chromaticity for each divided area composed of at least one pixel. A procedure for selecting one image signal from among a plurality of image signals of original images for each divided region based on an index, and a procedure for generating one composite image signal from the selected image signal It is a program.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus to which the present invention is applied.
[0019]
The imaging apparatus includes a lens system 101 composed of various lenses, a lens driving mechanism 102 for driving the lens system 101, an exposure control mechanism 103 for controlling a diaphragm of the lens system 101, a mechanical shutter 104, a CCD imaging device 105, A CCD driver 106 for driving the CCD image sensor 105, a preprocessing unit 107 including an A / D converter, a digital processing unit 108 for performing matrix conversion processing and other digital processing, a card interface 109, a memory card 110, An LCD image display system 111 is provided.
[0020]
Further, the imaging apparatus controls a system controller 112 for comprehensively controlling each unit, an operation switch system 113 including various SWs, an operation display system 114 for displaying an operation state and a mode state, and a lens driving mechanism. A lens driver 115 for light emission, a strobe 116 as light emitting means, an exposure control driver 117 for controlling the strobe 116, and a nonvolatile memory (EEPROM) for storing various setting information.
[0021]
In this image pickup apparatus, the system controller 112 performs all the control in an integrated manner. In particular, exposure (charge accumulation) is controlled by controlling the drive of the CCD image pickup device 105 by the exposure control mechanism 103, the mechanical shutter 104, and the CCD driver 106. The signal is read out, taken into the digital process unit 108 via the pre-processing unit 107, and subjected to various signal processing, and then recorded in the memory card 110 via the card interface 109.
[0022]
Next, a processing method for enlarging the D range using the imaging apparatus will be described.
[0023]
First, the system controller 112 controls the drive of the CCD image sensor 105 by the exposure control mechanism 103, the mechanical shutter 104, and the CCD driver 106, and obtains a first image signal taken under the first exposure condition from the CCD image sensor 115. Next, the second image signal photographed under the second exposure condition is read out from the CCD image sensor 115.
[0024]
Here, it is necessary to read out the first and second image signals while minimizing the time difference between them in order to prevent the “processing failure” that occurs when the subject moves during the photographing of the two images. It is. This can be realized, for example, by using a driving method described in Japanese Patent Application No. 2000-077136 in which a progressive scan (full frame readout) compatible CCD and a mechanical shutter are combined.
[0025]
By the way, the saturation level of the CCD image pickup device 105 is a limit for the high luminance side signal of the image signal read from the CCD image pickup device 105, and the low luminance side signal is noise of the image pickup device output in a state where it is incorporated in the image pickup apparatus. Since the level is limited, an imaging range exceeding at least these levels cannot be obtained.
[0026]
FIG. 2 is a diagram schematically illustrating photoelectric conversion characteristics of a general image sensor.
[0027]
In this figure, the horizontal axis indicates the incident light amount, and the vertical axis indicates the signal level logarithmically. In the drawing, the limit level of the signal on the high luminance side is indicated by UL, and the limit level of the signal on the low luminance side is indicated by LL. UL is a value that substantially corresponds to the saturation level of the image sensor, while LL is not uniquely defined, unlike NL that is a noise level. For example, a predetermined S / N ratio that an image can withstand viewing is obtained. It is determined as a level to have. Even if LL is not the above-mentioned standard, visual evaluation may be performed to determine a level “the level below this is blackout”, and the limit level may be set to LL.
[0028]
The effective luminance range is between UL and LL determined in this way, and the difference DR (= UL−LL) on the logarithmic axis is the imaging range. This DR varies depending on the designed structure of the image pickup apparatus, but in the case of a normal image pickup apparatus, 5 to 6 EV (30 to 36 dB) is often used, and the image pickup apparatus of the present embodiment is also of this class. It has a range.
[0029]
Therefore, the photoelectric conversion characteristics of the two image signals read from the CCD image sensor 105 are schematically shown in FIG. In this figure, the first image signal is imaged with a short exposure time, and the second image signal is a signal imaged with a long exposure time. For this reason, the photoelectric conversion characteristics of the two image signals have a relationship of moving in the horizontal axis direction by an amount corresponding to the difference in exposure time. As a result, these two image signals include image signals regarding the luminance of the subject in the enlarged luminance range even in the imaging apparatus having the same imaging range DR.
[0030]
In the present invention, the shift amount of the exposure time can be arbitrarily selected, and the exposure level may be changed by a parameter other than the exposure time, such as a diaphragm.
[0031]
Next, a method for obtaining one composite image from two image signals having different exposure levels will be described.
[0032]
FIG. 4 is a diagram illustrating a configuration of an imaging apparatus related to image synthesis. The image composition processing is executed by the image generation processing unit 122 provided in the system controller 112 exchanging signals with the matrix conversion processing unit 120 provided in the digital process unit 108. The image generation processing unit 122 includes an indexing unit 123, an image signal selection unit 124, and an image generation unit 125.
[0033]
First, the image composition processing unit 122 inputs a dot sequential signal (R, G, B) that is a color signal of the first and second image signals to the digital processing unit 108. Then, the matrix conversion processing unit 120 is operated to generate a synchronized component signal. Here, the component signal of the first original image is S1, and the component signal of the second original image is S2. The specific configuration of the representative component signal Sn (n = 1, 2 is shown) is a luminance signal (Yn) and a color difference signal (Rn−Yn, Bn−Yn). In the following embodiments, An embodiment using the luminance signal (Yn) and the color difference signals (Rn−Yn, Bn−Yn) will be described.
[0034]
In the first embodiment according to the present invention, the indexing unit 123 calculates the chromaticity Cn, which is an index corresponding to chromaticity, for each pixel from the component signal Sn obtained by the above-described procedure, using Expression (1). .
[0035]
Cn = {(Rn−Yn) ² + (Bn−Yn) ² } / (Yn + α) ² (1)
Here, α: constant Here, the color saturation Cn defined by the equation (1) is a concept introduced in the present invention, and its physical meaning will be described based on the color difference coordinates of FIG. The color difference coordinates are coordinates having the horizontal axis as the color difference signal RY and the vertical axis as the color difference signal BY, and the point Dn (Rn-Yn, Bn-Yn) corresponding to the color difference signal is the origin (0, 0). ) As a starting point and a color difference vector with a declination of θ.
[0036]
In this coordinate system, the declination θ represents the hue, and the coordinate origin (0, 0) represents a colorless state. Therefore, the size of the color difference vector, that is, the size of the line segment connecting the point Dn and the origin corresponds to the product of the saturation representing the color density and the luminance representing brightness (lightness) as a physical quantity. If this size is used, it is considered that the color intensity can be identified.
[0037]
However, as described above, brightness is included in the magnitude of the color difference vector, so that it is not possible to determine only the color density using this index. Therefore, in the expression (1), the magnitude of the color difference vector is divided by the luminance signal Y and is normalized to be used as an index representing the color density. In general, a conversion for normalizing a color image to a constant lightness is called chromaticity conversion. The color saturation in the present invention is also a value obtained by standardizing a color difference with a luminance signal. Therefore, this index is called an index corresponding to chromaticity.
[0038]
The index should be originally defined as √Cn, but the reason why the index is defined as a value obtained by squaring the color saturation in this embodiment is to enable the calculation process of the index to be performed quickly and easily. is there. In addition, the denominator should be the square of the luminance signal Y in its original meaning. However, the small offset α is added to this when the value of the luminance signal Yn is small. This is because the influence of the relative increase in error due to the components is reduced, and a zero division error is avoided. When the luminance level is very small due to the action of the offset α (when the numerator denominator is close to a very small indefinite state), the color saturation is regarded as small.
[0039]
Next, the image signal selection unit 124 compares the color saturation C1 of the first original image thus obtained with the color saturation C2 of the second original image for each pixel. Then, the component signal of the original image having a large color saturation is selected and used as the component signal of the newly synthesized image. That is, S1 is selected when C1> C2 (or C1 >> C2), and S2 is selected when C1 <C2 (or C1 << C2). When C1 = C2 (or C1≈C2), (S1 + S2) / 2 is set as a new component signal.
[0040]
Then, the image generation unit 125 generates one composite image based on the selected image signal. The pixel signal thus obtained is processed by a subsequent circuit in the same manner as a conventional imaging signal, and is recorded on the memory card 110 or displayed on the LCD image display system 111. As described above, since an image having a large color density is selected for each pixel based on the color component of the original image, the influence of gradation compression is reduced compared to processing based on the luminance component. A natural image that is not easily received and does not generate a false signal can be obtained by simple processing.
[0041]
In the present embodiment, the color saturation is calculated for each pixel. However, the present invention is not limited to this mode, and the original image is divided into regions including at least one pixel and is divided into the divided regions. The index may be calculated.
[0042]
Next, in the second embodiment according to the present invention, the indexing unit 123 calculates the color saturation Cn for each pixel from the component signal Sn according to the equation (2) instead of the equation (1).
[0043]
Cn = {(Rn−Yn) ² + (Bn−Yn) ² } * (Ym + α) ² (2)
Where α is a constant, where m is a number different from n. That is, when n = 1, m = 2, and when n = 2, m = 1.
[0044]
Now, assuming that the relationship of C1> C2 is established, substituting equation (2) into this relationship equation yields equation (3).
{(R1-Y1) ² + (B1-Y1) ² } * (Y2 + α) ²
> {(R2-Y2) ² + (B2-Y2) ² } * (Y1 + α) ² (3)
And if Formula (3) is deformed, Formula (4) is obtained.
{(R1-Y1) ² + (B1-Y1) ² } / (Y1 + α) ²
> {(R2-Y2) ² + (B2-Y2) ² } / (Y2 + α) ² (4)
Formula (4) is equivalent to comparing the magnitudes of the color saturations of the first embodiment. That is, when comparing two values, even if the magnitudes are compared using the values defined in Equation (2), the same result as comparing the values defined in Equation (1) is obtained. Will get.
[0045]
Therefore, by using the value defined by the expression (2) as the color saturation, it is not necessary to use the division calculation function, so that the calculation can be speeded up and simplified.
[0046]
Next, in the third embodiment according to the present invention, instead of Equation (1), the indexing unit 123 calculates the color saturation Cn for each pixel from the component signal Sn by Equation (5).
[0047]
Cn = {| Rn−Yn | + | Bn−Yn |} / (Yn + α) (5)
However, α: The index defined by the constant equation (5) is based on the same concept as in the equation (1), and the numerator is a value related to the size of the color difference vector (however, each component of the color difference vector) Is the sum of the sizes of By using this equation (5), the calculation of the square is not necessary, so that the calculation can be speeded up and simplified as compared with the first embodiment.
[0048]
Next, in the fourth embodiment according to the present invention, instead of Equation (1), the indexing unit 123 calculates the color saturation Cn for each pixel from the component signal Sn by Equation (6).
[0049]
Cn = {| Rn−Yn | + | Bn−Yn |} * (Ym + α) (6)
Where α is a constant, and n is a number different from m.
[0050]
Expression (6) is obtained by modifying the index used in the third embodiment according to the concept described in the second embodiment. Therefore, by using this formula (6), the calculation and division of the square are not required, and the calculation can be further speeded up and simplified.
[0051]
Note that the comparison and selection of the two pixels may be performed for all the pixels of the image signal, or may be processed for pixels in a luminance region other than the common luminance region shown in FIG.
[0052]
In the above-described embodiments, the processing for combining two images has been described. However, the present invention is not limited to this embodiment, and three or more images may be used. For example, it is possible to perform a process similar to the above using three images A, B, and C having exposure levels of large, medium, and small, and select the one having the maximum color saturation Cn.
[0053]
Furthermore, in the present embodiment, the color saturation is defined using the luminance and the color difference signal. However, the present invention is not limited to this form. For example, the color intensity is determined based on the amount defined in the L ^* a ^* b ^* space. It may be defined.
[0054]
The present invention may be configured using hardware such as a circuit, or may be realized by incorporating a processing program into a microcomputer.
[0055]
【The invention's effect】
As described above, by applying the present invention, it is possible to provide an imaging apparatus capable of obtaining a natural captured image with an expanded D range.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus to which the present invention is applied.
FIG. 2 is a diagram schematically illustrating photoelectric conversion characteristics of a general image sensor.
FIG. 3 is a diagram schematically illustrating photoelectric conversion characteristics of two image signals.
FIG. 4 is a diagram illustrating a configuration of an imaging apparatus related to image synthesis.
FIG. 5 is a diagram showing the physical meaning of color saturation.
[Explanation of symbols]
105 ... CCD image sensor 107 ... pre-process unit 108 ... digital process unit 112 ... system controller

Claims (7)

Indexing means for processing each of the image signals of the plurality of original images obtained by giving different exposure amounts, and calculating an index corresponding to chromaticity for each divided region composed of at least one pixel;
Selection means for selecting one image signal from among the image signals of the plurality of original images for each of the divided regions based on the index;
An image pickup apparatus comprising: image generation means for generating one composite image signal from the selected image signal. The selection means includes
2. The imaging apparatus according to claim 1, wherein the index having the largest absolute value is determined by comparing the respective indexes for each of the divided regions, and the image signal of the original image that gives the index is selected. The indexing means includes
Using the luminance signal Y, the color difference signals RY and BY, and a predetermined value α,
The imaging apparatus according to claim 1, wherein an index C corresponding to chromaticity is calculated from the following equation.
C = {(R−Y) ² + (B−Y) ² } / (Y + α) ² The indexing means includes
Using the luminance signal Y of one image, the color difference signals RY and BY, the luminance signal Y ^* of another image to be compared, and a predetermined value α,
The imaging apparatus according to claim 1, wherein an index C corresponding to chromaticity is calculated from the following equation.
C = {(R−Y) ² + (B−Y) ² } * (Y ^* + α) ² The indexing means includes
Using the luminance signal Y of one image, the color difference signals RY and BY, the luminance signal Y ^* of another image to be compared, and a predetermined value α,
The imaging apparatus according to claim 1, wherein an index C corresponding to chromaticity is calculated from the following equation.
C = {| R−Y | + | B−Y |} * (Y ^* + α) Process each of the image signals of the plurality of original images obtained by giving different exposure amounts, calculate an index corresponding to chromaticity for each divided region composed of at least one pixel,
One image signal is selected from the image signals of the plurality of original images for each of the divided regions based on the index,
An imaging method, wherein one composite image signal is generated from the selected image signal. In the processing program of the imaging device,
On the computer,
A procedure for processing each of the image signals of the plurality of original images obtained by giving different exposure amounts, and calculating an index corresponding to chromaticity for each divided region composed of at least one pixel,
A procedure for selecting one image signal from among the image signals of the plurality of original images for each of the divided regions based on the index,
Generating a composite image signal from the selected image signal;
A program for running

Images