JP4014436B2 - Imaging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image pickup apparatus using an image pickup device such as a CCD, and more particularly to a technique for synthesizing a plurality of images for expanding a dynamic range.
[0002]
[Prior art]
In recent years, various attempts to obtain a captured image with a wide dynamic range by combining a plurality of images having different exposure amounts have been proposed.
[0003]
One of them is to generate an image of one wide dynamic range (hereinafter referred to as “D range”) from two original image data with different exposure amounts, and perform various image processing on the image and record it. A target image is generated. Here, the wide D range image generated in the intermediate stage is an image having a larger number of bits than the number of bits of the data of the original image so as to cover the entire range. In this way, there is a possibility that the best processing can be performed by performing various kinds of image processing using an image of a wide D range.
[0004]
Another method is to generate an image obtained by performing weighted synthesis by averaging two original image data for each pixel. This method has an advantage that it can be easily applied to an imaging apparatus because the arithmetic processing of the image is simple.
[0005]
Furthermore, one of the other methods is to replace the data in the area deviating from the range of the D range in the luminance data of the original image on one side with the data of the other original image. This method has an advantage that it can be easily applied to the imaging apparatus because the image calculation process is simple as in the above method.
[0006]
[Problems to be solved by the invention]
However, the method of creating a wide D-range image at an intermediate stage has a problem that the creation and subsequent processing are complicated, and a large memory area is required. Considering needs such as real-time processing, miniaturization, and cost reduction, it is not preferable.
[0007]
Even if image information of a wide D range can be obtained temporarily, if signal processing such as luminance compression is performed on an finally generated image having a narrow range, the image quality will eventually deteriorate. Become. Therefore, it cannot be said that a signal processing technique capable of obtaining sufficient image quality has been established from this viewpoint.
[0008]
In addition, in the averaging process, the D range is expanded but the gradation is compressed, and the contrast is lowered when viewed as an image as it is. On the other hand, when some kind of image correction is performed, there is a case where the quantization noise is increased and the image quality is deteriorated.
[0009]
Further, in the deviation data replacement process, an image to be adopted is switched depending on the luminance information, and therefore an unnatural image may be obtained depending on the luminance distribution of the subject image or the like. For example, a high-frequency and high-contrast image of an object such as a stripe pattern on a resolution chart will produce a false signal image with an inverted pattern.
[0010]
The present invention has been made in view of such circumstances, and an object thereof is to provide an imaging apparatus that can be easily applied to an imaging apparatus and can obtain a natural captured image with an expanded D range. .
[0011]
[Means for Solving the Problems]
The present invention for solving the above-described problem is to process each of a plurality of image signals of original images obtained by giving different exposure amounts, and regarding the target pixel to be processed at the time point, the target pixel and the target pixel Indexing means for calculating an index representing the degree of contrast related to the pixel of interest based on each pixel value of a pixel group consisting of adjacent pixels, and images of the plurality of original images for each pixel of interest based on the index A selection unit that selects one image signal from the signals; and an image generation unit that generates one composite image signal from the selected image signal. The indexing unit includes the image signal S of the pixel of interest. The image pickup apparatus calculates an index C representing the degree of contrast from the following equation using the image signal S _{i of} the neighboring pixels and a predetermined value α.
C = Σ | S−S _i | / {Σ (S + S _i ) + α}
[0012]
Further, the present invention processes each of the image signals of a plurality of original images obtained by giving different exposure amounts, and includes the target pixel and a neighboring pixel associated with the target pixel as a processing target at the time. Indexing means for calculating an index representing the degree of contrast related to the target pixel based on each pixel value of the pixel group; and one of the image signals of the plurality of original images for each target pixel based on the index A selection unit configured to select an image signal; and an image generation unit configured to generate one composite image signal from the selected image signal. The indexing unit includes a maximum value Smax of the image signal of the pixel group, the pixel The imaging apparatus calculates an index C representing the degree of contrast from the following equation using the minimum value Smin of the image signals of the group and the average value Sav of the image signals of the pixel group.
C = (Smax−Smin) / Sav
[0017]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus to which the present invention is applied.
[0018]
The imaging apparatus includes a lens system 101 composed of various lenses, a lens driving mechanism 102 for driving the lens system 101, an exposure control mechanism 103 for controlling a diaphragm of the lens system 101, a mechanical shutter 104, a CCD imaging device 105, A CCD driver 106 for driving the CCD image sensor 105, a preprocessing unit 107 including an A / D converter, a digital processing unit 108 for performing matrix conversion processing and other digital processing, a card interface 109, a memory card 110, An LCD image display system 111 is provided.
[0019]
Further, the imaging apparatus controls a system controller 112 for comprehensively controlling each unit, an operation switch system 113 including various SWs, an operation display system 114 for displaying an operation state and a mode state, and a lens driving mechanism. A lens driver 115 for light emission, a strobe 116 as light emitting means, an exposure control driver 117 for controlling the strobe 116, and a nonvolatile memory (EEPROM) for storing various setting information.
[0020]
In this image pickup apparatus, the system controller 112 performs all the control in an integrated manner. In particular, exposure (charge accumulation) is controlled by controlling the drive of the CCD image pickup device 105 by the exposure control mechanism 103, the mechanical shutter 104, and the CCD driver 106. The signal is read out, taken into the digital process unit 108 via the pre-processing unit 107, and subjected to various signal processing, and then recorded in the memory card 110 via the card interface 109.
[0021]
Next, a processing method for enlarging the D range using the imaging apparatus will be described.
[0022]
First, the system controller 112 controls the drive of the CCD image sensor 105 by the exposure control mechanism 103, the mechanical shutter 104, and the CCD driver 106, and obtains a first image signal taken under the first exposure condition from the CCD image sensor 115. Next, the second image signal photographed under the second exposure condition is read out from the CCD image sensor 115.
[0023]
Here, it is necessary to read out the first and second image signals while minimizing the time difference between them in order to prevent the “processing failure” that occurs when the subject moves during the photographing of the two images. It is. This can be realized, for example, by using a driving method described in Japanese Patent Application No. 2000-077136 in which a progressive scan (full frame readout) compatible CCD and a mechanical shutter are combined.
[0024]
By the way, the saturation level of the CCD image pickup device 105 is a limit for the high luminance side signal of the image signal read from the CCD image pickup device 105, and the low luminance side signal is noise of the image pickup device output in a state where it is incorporated in the image pickup apparatus. Since the level is limited, an imaging range exceeding at least these levels cannot be obtained.
[0025]
FIG. 2 is a diagram schematically illustrating photoelectric conversion characteristics of a general image sensor.
[0026]
In this figure, the horizontal axis indicates the incident light amount, and the vertical axis indicates the signal level logarithmically. In the drawing, the limit level of the signal on the high luminance side is indicated by UL, and the limit level of the signal on the low luminance side is indicated by LL. UL is a value that substantially corresponds to the saturation level of the image sensor, while LL is not uniquely defined, unlike NL that is a noise level. For example, a predetermined S / N ratio that an image can withstand viewing is obtained. It is determined as a level to have. Even if LL is not the above-mentioned standard, visual evaluation may be performed to determine a level “the level below this is blackout”, and the limit level may be set to LL.
[0027]
The effective luminance range is between UL and LL determined in this way, and the difference DR (= UL−LL) on the logarithmic axis is the imaging range. This DR varies depending on the designed structure of the image pickup apparatus, but in the case of a normal image pickup apparatus, 5 to 6 EV (30 to 36 dB) is often used, and the image pickup apparatus of the present embodiment is also of this class. It has a range.
[0028]
Therefore, the photoelectric conversion characteristics of the two image signals read from the CCD image sensor 105 are schematically shown in FIG. In this figure, the first image signal is imaged with a short exposure time, and the second image signal is a signal imaged with a long exposure time. For this reason, the photoelectric conversion characteristics of the two image signals have a relationship of moving in the horizontal axis direction by an amount corresponding to the difference in exposure time. As a result, these two image signals include image signals regarding the luminance of the subject in the enlarged luminance range even in the imaging apparatus having the same imaging range DR.
[0029]
In the present invention, the shift amount of the exposure time can be arbitrarily selected, and the exposure level may be changed by a parameter other than the exposure time, such as a diaphragm.
[0030]
Next, a method for obtaining one composite image from two image signals having different exposure levels will be described. Here, in the following description, unless otherwise specified, processing using a monochrome signal is performed.
[0031]
FIG. 4 is a diagram illustrating a configuration of an imaging apparatus related to image synthesis. The image composition processing is executed by the image generation processing unit 122 provided in the system controller 112 exchanging signals with the digital process unit 108. The image generation processing unit 122 includes an indexing unit 123, an image signal selection unit 124, and an image generation unit 125.
[0032]
In the first embodiment according to the present invention, the image composition processing unit 122 reads the image signal S1 of the first original image and the image signal S2 of the second original image via the digital process unit 108. An image signal of one pixel at the position of i row and j column of the original image is represented as Sn (i, j). Note that n represents 1 or 2, and therefore Sn means the image signal S1 of the first original image and the image signal S2 of the second original image.
[0033]
Next, a local region including the image signal Sn (i, j) to be processed is considered. This local area can be rephrased as "a pixel group consisting of a pixel of interest whose coordinates are (i, j) and its neighboring pixels". Contrast is not obtained by considering one pixel of interest alone. Since it cannot be defined, a pixel group that includes the target pixel and the neighboring pixels associated with the target pixel is considered, and the target pixel (or the target pixel) is obtained with a contrast value obtained by applying a predetermined calculation to the pixel group. It is set to be regarded as a contrast value for the pixel group). In the region shown in FIG. 5, the pixel and five pixels above, below, left, and right are targeted.
[0034]
The indexing unit 123 uses each data in the local region to generate a high frequency component Hn (i, j) expressed by Equation (1) and a low frequency component Ln (i, j) expressed by Equation (2). calculate.
[0035]
Hn (i, j) = Σ | Sn (i, j) −Sn (k, l) | (1)
Ln (i, j) = Σ {Sn (i, j) + Sn (k, l)} (2)
Here, k and l represent the four points on the top, bottom, left and right of the pixel at the position of i row and j column shown in FIG.
[0036]
Based on Hn (i, j) and Ln (i, j), the contrast component Cn (i, j) shown in Expression (3) is calculated.
[0037]
Cn (i, j) = Hn (i, j) / {Ln (i, j) + α} (3)
However, α: constant Next, the meaning of the equations (1) to (3) will be described.
[0038]
First, the high-frequency component Hn (i, j) is considered to indicate the difference between the image signals Sn (i, j) and the surrounding points, that is, the contrast that is a change in pixels. Therefore, when this value is large, it indicates that the image in this local region has a high contrast, and when this value is small, it indicates that the image in this local region has a low contrast.
[0039]
In addition, the operation {Sn (i, j) + Sn (k, l)} constituting the low frequency component Ln (i, j) is an intermediate between the image signal Sn (i, j) and the image signal of the surrounding points. A value that is twice the value is given. Therefore, when the value of Ln (i, j) is large, it indicates that the image signal in this local region is large on average, and when this value is small, the image signal in this local region is small on average. It is shown that.
[0040]
Therefore, the contrast component Cn (i, j) obtained by dividing the high frequency component Hn (i, j) by the low frequency component Ln (i, j) can be considered as an index value based on contrast. . Here, the minute offset value α is added to the denominator to reduce the influence of the error due to the noise component included when the value of the low frequency component Ln (i, j) is small, and further, the low frequency component Ln ( This is to substantially exclude the case where i, j) is very small.
[0041]
Next, the image signal selection unit 124 compares the contrast component C1 of the first original image thus obtained and the contrast component C2 of the second original image for each pixel. Then, an image signal of an original image having a large contrast component is selected and used as an image signal of a newly synthesized image. That is, S1 is selected when C1> C2 (or C1 >> C2), and S2 is selected when C1 <C2 (or C1 << C2). When C1 = C2 (or C1≈C2), (S1 + S2) / 2 is set as a new image signal.
[0042]
Then, the image generation unit 125 generates one composite image based on the selected image signal. The pixel signal thus obtained is processed by a subsequent circuit in the same manner as a conventional imaging signal, and is recorded on the memory card 110 or displayed on the LCD image display system 111. In this way, since it is configured to select a pixel having a large contrast for each pixel based on the image signal of the original image, it is less susceptible to gradation compression than processing based on the luminance component, etc. A natural image free from false signals can be obtained by simple processing.
[0043]
Next, in the second embodiment according to the present invention, the contrast component Cn (i, j) is calculated by the equation (4) instead of the equation (3).
[0044]
Cn (i, j) = Hn (i, j) * {Lm (i, j) + α) (4)
Where α is a constant, where m is a number different from n. That is, when n = 1, m = 2, and when n = 2, m = 1.
[0045]
Now, assuming that the relationship of C1> C2 is established, substituting equation (4) into this relationship gives equation (5).
H1 (i, j) * {L2 (i, j) + α)
> H2 (i, j) * {L1 (i, j) + α) (5)
Then, when Expression (5) is modified, Expression (6) is obtained.
H1 (i, j) / {L1 (i, j) + α)
> H2 (i, j) / {L2 (i, j) + α) (6)
Expression (6) is the same as comparing the magnitudes of the contrast components in the first embodiment. That is, when comparing two values, even if the magnitudes are compared using the value defined in Equation (4), the same result as comparing the values defined in Equation (3) is obtained. Will get.
[0046]
Therefore, by using the value defined by Equation (4) as a contrast component, it is not necessary to use a division process as a calculation function, so that the calculation can be speeded up and simplified.
[0047]
Next, in the third embodiment according to the present invention, the contrast component Cn (i, j) is defined using the equation (7).
[0048]
Cn (= | Smax−Smin | / Sav (7)
Here, Smax: the maximum value of the image signal in the local area, Smin: the minimum value of the image signal in the local area, Sav: the average value of the image signals in the local area.
[0049]
According to this equation (7), it is possible to create an index with less error based on all image signal information in the region.
[0050]
Next, in the fourth embodiment according to the present invention, block processing similar to DCT (Discrete Cosine Transfer) processing in standard JPEG processing is performed. That is, the contrast component Cn (i, j) is defined by the equation (3) or the equation (4) where the alternating current component in DCT processing is Hn (i, j) and the direct current component is Ln (i, j). is there. According to this method, it is possible to create an index with less error based on all image signal information in the region.
[0051]
Next, a method for synthesizing an image based on a color signal according to the fifth embodiment of the present invention will be described.
[0052]
In the configuration shown in FIG. 4, first, the image composition processing unit 122 inputs a dot sequential signal (R, G, B) that is a color signal of the first and second image signals to the digital processing unit 108. Then, the matrix conversion processing unit 120 (not shown) is operated to generate a synchronized component signal, and the luminance signal Y shown in Expression (8) is calculated using this component signal.
[0053]
Y = 0.299R + 0.587G + 0.114B (8)
Then, this luminance signal is handled as the image signal Sn (i, j) in each of the above-described embodiments.
[0054]
Hereinafter, the indexing unit 123 calculates the contrast component Cn based on the equations (1) to (4), and the image signal selection unit 124 compares C1 and C2 and selects an image to be selected in the same procedure as in the above embodiment. Determine the signal. Then, the image generation unit 125 creates one composite image using the component signal extracted as a result.
[0055]
According to the present embodiment, the color image is also configured to select a pixel having a high contrast for each pixel based on the luminance signal, so that gradation compression is performed compared to processing based on the luminance component. It is possible to obtain a natural image which is not easily affected and does not generate a false signal by simple processing.
[0056]
In the first and second embodiments described above, calculation is performed using the upper, lower, left, and right pixels of the pixel to be processed as shown in FIG. 5, but the present invention is not limited to this embodiment, and as shown in FIG. Alternatively, the same calculation may be performed using pixels at different positions surrounding the pixels to be processed.
[0057]
Further, the local region to be processed is not limited to 3 × 3 shown in the embodiment and can be arbitrarily set. For example, it may be a 5 × 5 area as shown in FIG.
[0058]
Furthermore, in the present embodiment, the image signal used for image synthesis is selected for each pixel. However, the present invention is not limited to this mode, and a local region including the selected pixel is selected and the image signal is selected. You may comprise so that an image may be produced | generated by combining a local area | region.
[0059]
Note that the comparison and selection of the two pixels may be performed for all the pixels of the image signal, or may be processed for pixels in a luminance region other than the common luminance region shown in FIG.
[0060]
In the above-described embodiments, the processing for combining two images has been described. However, the present invention is not limited to this embodiment, and three or more images may be used. For example, it is possible to perform a process similar to the above using three images A, B, and C having exposure levels of large, medium, and small, and select the one having the maximum contrast component Cn.
[0061]
The present invention may be configured using hardware such as a circuit, or may be realized by incorporating a processing program into a microcomputer.
[0062]
【The invention's effect】
As described above, by applying the present invention, it is possible to provide an imaging apparatus capable of obtaining a natural captured image with an expanded D range.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus to which the present invention is applied.
FIG. 2 is a diagram schematically illustrating photoelectric conversion characteristics of a general image sensor.
FIG. 3 is a diagram schematically illustrating photoelectric conversion characteristics of two image signals.
FIG. 4 is a diagram illustrating a configuration of an imaging apparatus related to image synthesis.
FIG. 5 is a diagram showing a configuration of a local region used for calculating an index value.
FIG. 6 is a diagram showing another configuration of a local region used for calculating an index value.
FIG. 7 is a diagram showing another configuration of a local region used for calculating an index value.
[Explanation of symbols]
105 ... CCD image sensor 107 ... pre-process unit 108 ... digital process unit 112 ... system controller

Claims (2)

Each pixel signal of a plurality of original images obtained by giving different exposure amounts is processed, and each pixel of a pixel group including the target pixel and the neighboring pixels associated with the target pixel as a processing target at the time point Indexing means for calculating an index representing the degree of contrast related to the pixel of interest based on a value;
Selection means for selecting one image signal from among the image signals of the plurality of original images for each pixel of interest based on the index;
Image generating means for generating one composite image signal from the selected image signal,
The indexing means includes
Using the image signal S of the pixel of interest, the image signal S _{i of} the neighboring pixel, and a predetermined value α,
An image pickup apparatus that calculates an index C representing a degree of contrast from the following equation.
C = Σ | S−S _i | / {Σ (S + S _i ) + α} Each pixel signal of a plurality of original images obtained by giving different exposure amounts is processed, and each pixel of a pixel group including the target pixel and the neighboring pixels associated with the target pixel as a processing target at the time point Indexing means for calculating an index representing the degree of contrast related to the pixel of interest based on a value;
Selection means for selecting one image signal from among the image signals of the plurality of original images for each pixel of interest based on the index;
Image generating means for generating one composite image signal from the selected image signal,
The indexing means includes
Using the maximum value Smax of the image signal of the pixel group, the minimum value Smin of the image signal of the pixel group, and the average value Sav of the image signal of the pixel group,
An image pickup apparatus that calculates an index C representing a degree of contrast from the following equation.
C = (Smax−Smin) / Sav

Images