JP2003319404A - Imaging apparatus, method therefor, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus which can provide a natural shot image having an enlarged D range, an imaging method thereof, and a program therefor. <P>SOLUTION: The imaging apparatus includes an indexing means (123) for calculating an index corresponding to a chromaticity for each segment having at least one pixel by processing each of a plurality of image signals obtained for different given quantities of exposure light, a selection means (124) for selecting one of the plurality of image signals for each segment on the basis of the calculated index, and an image generating means (125) for generating a composite image signal from the selected image signal. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus using an image pickup device such as a CCD, and more particularly to a technique for synthesizing a plurality of images for expanding a dynamic range.

[0002]

2. Description of the Related Art In recent years, various attempts have been proposed to obtain a picked-up image having a wide dynamic range by synthesizing a plurality of images having different exposure amounts.

One of them is to temporarily generate an image having a wide dynamic range (hereinafter referred to as "D range") from two original image data having different exposure amounts, and perform various image processing on this image. It is applied to generate a recording target image. Here, the image of the wide D range generated in the intermediate stage is an image having a larger number of bits than the number of bits of the data of the original image so as to cover the entire range. As described above, there is a possibility that the best processing can be performed by performing various kinds of image processing using an image of a wide D range.

Another method is to generate an image in which two original image data are added and averaged for each pixel and weighted and combined. This method has an advantage that it can be easily applied to the image pickup apparatus because the arithmetic processing of the image is simple.

Further, another method is to replace the data in the area deviating from the D range of the luminance data of the original image on one side with the data of another original image. This method has an advantage that it can be easily applied to an image pickup apparatus because the image calculation processing is simple as in the above method.

[0006]

However, in the method of creating a wide D range image in the intermediate stage, there is a problem that the processing for this creation and the subsequent calculation becomes complicated and a large memory area is required. Therefore, it is considered to be unfavorable in consideration of needs such as real-time processing, miniaturization, and cost reduction required for the image pickup apparatus.

Further, even if image information in a wide D range can be temporarily obtained, if signal processing such as luminance compression is applied to an image finally generated with a narrow range, the image quality will eventually deteriorate. Will be done. Therefore, from this point of view, it cannot be said that a signal processing technique capable of obtaining a sufficient image quality has been established yet.

In addition, in the addition and averaging process, the D range is expanded, but the gradation is compressed, and the contrast deteriorates when viewed as an image as it is. On the other hand, when performing some kind of image correction, on the contrary, an increase in quantization noise may occur, resulting in deterioration in image quality.

Further, in the deviation data replacement process, since the image to be adopted is switched depending on the brightness information, an unnatural image may be obtained depending on the brightness distribution such as the pattern of the subject. For example, in a high-frequency and high-contrast image of a subject such as a striped pattern of a resolution chart, an inverted pattern-like image is generated.

The present invention has been made in view of the above circumstances, and is easily applied to an image pickup apparatus, and an image pickup apparatus capable of obtaining a natural picked-up image with an expanded D range,
An object is to provide an imaging method and its program.

[0011]

According to the present invention for solving the above-mentioned problems, at least one image signal of a plurality of original images obtained by applying different exposure amounts is processed to obtain at least 1.
Indexing means for calculating an index corresponding to chromaticity for each of the divided areas composed of pixels, and selection for selecting one image signal from the image signals of a plurality of original images for each divided area based on the index. And an image generation unit that generates one combined image signal from the selected image signal.

According to the present invention, in the image pickup apparatus according to the above-mentioned invention, the selecting means compares the respective indexes for each divided area to determine the index having the largest absolute value and determines the original image to which the index is given. It is an imaging device which selects an image signal.

According to the present invention, in the image pickup apparatus according to the above-mentioned invention, the indexing means has a luminance signal Y and a color difference signal R.
It is an image pickup apparatus that calculates an index C corresponding to chromaticity from the following formula using −Y and BY, and a predetermined value α.

C = {(R−Y) ² + (B−Y) ² } / (Y + α) ^{2 Further} , in the image pickup device according to the present invention as described above, the indexing unit has the brightness of one image. Signal Y, color difference signals RY and BY, luminance signals Y ^* of other images to be compared,
The index C corresponding to the chromaticity is calculated from the following equation using a predetermined value α.
Is an imaging device that calculates

C = {(R−Y) ² + (B−Y) ² } * (Y ^* + α) ^{2 Further} , in the image pickup device according to the present invention as described above, the indexing means is one image. Luminance signal Y, color difference signals RY and BY, luminance signals Y ^* of other images to be compared,
The index C corresponding to the chromaticity is calculated from the following equation using a predetermined value α.
Is an imaging device.

C = {| R−Y | + | B−Y |} * (Y ^* + α) Further, according to the present invention, each of a plurality of image signals obtained by giving different exposure amounts is processed to obtain at least 1. An index corresponding to the chromaticity is calculated for each divided area composed of pixels, and one image signal is selected from the image signals of a plurality of original images for each divided area based on the index, and the selected image is selected. This is an imaging method for generating one combined image signal from a signal.

Further, according to the present invention, each of the image signals of a plurality of original images obtained by giving different exposure amounts to the computer is processed, and the chromaticity corresponds to each divided area formed of at least one pixel. A procedure for calculating an index, a procedure for selecting one image signal from the image signals of a plurality of original images for each divided area based on the index, and a procedure for generating one combined image signal from the selected image signals. It is a program to be executed.

[0018]

1 is a block diagram showing the structure of an image pickup apparatus to which the present invention is applied.

This image pickup apparatus includes a lens system 101 including various lenses, a lens drive mechanism 102 for driving the lens system 101, an exposure control mechanism 103 for controlling the aperture of the lens system 101, a mechanical shutter 104, and a CCD.
An image pickup device 105, a CCD driver 106 for driving the CCD image pickup device 105, a pre-process unit 107 including an A / D converter, a digital process unit 108 for performing matrix conversion processing and other digital processing, a card interface 109, Memory card 110, LC
A D image display system 111 is provided.

The image pickup apparatus further includes a system controller 112 for centrally controlling each unit, an operation switch system 113 including various SWs, an operation display system 114 for displaying an operation state and a mode state, and a lens driving mechanism. A lens driver 115 for controlling the flash, a strobe 116 as a light emitting unit, an exposure control driver 117 for controlling the strobe 116, and a non-volatile memory (EEPROM) for storing various setting information and the like are provided.

In this image pickup apparatus, the system controller 112 centrally controls all the operations, and particularly the exposure control mechanism 103, the mechanical shutter 104, and the CC.
The driving of the CCD image pickup device 105 by the D driver 106 is controlled to perform exposure (charge accumulation) and signal readout.
This is taken into the digital process unit 108 via the pre-process unit 107, subjected to various kinds of signal processing, and then recorded on the memory card 110 via the card interface 109.

Next, a processing method for expanding the D range using the image pickup apparatus will be described.

First, the system controller 112 controls the drive of the CCD image pickup device 105 by the exposure control mechanism 103, the mechanical shutter 104, and the CCD driver 106, and outputs the first image signal photographed under the first exposure condition to the C image signal.
The second image signal read out from the CD image pickup device 115 and subsequently photographed under the second exposure condition is used for the CCD image pickup device 115.
Read from.

Here, the reading of the first and second image signals with the time difference thereof being as small as possible prevents the "processing failure" that occurs when the object moves during the shooting of the two images. Is necessary for. This can be realized by using, for example, a driving method combining a progressive scanning (full frame reading) type CCD and a mechanical shutter, which is described in Japanese Patent Application No. 2000-077136.

By the way, the high-luminance side signal of the image signal read from the CCD image pickup device 105 is the CCD image pickup device 1.
The saturation level of 05 becomes the limit, and the noise level of the output of the image pickup device when the signal on the low luminance side is incorporated in the image pickup device becomes the limit, so it is not possible to obtain an image pickup range that exceeds these levels at least. .

FIG. 2 is a diagram schematically showing photoelectric conversion characteristics of a general image pickup device.

In the figure, the horizontal axis represents the amount of incident light and the vertical axis represents the signal level in logarithmic form. In the figure, the limit level of the signal on the high brightness side is indicated by UL, and the limit level of the signal on the low brightness side is indicated by LL. UL is a value substantially corresponding to the saturation level of the image sensor, and LL is not uniquely defined unlike NL which is a noise level. For example, a predetermined S / N ratio that an image can withstand is appreciated. It is defined as the level to have. It should be noted that the LL is not limited to the above criteria, but a visual evaluation may be performed to determine a “level below this is blackout”, and the limit level may be set to LL.

The area between UL and LL thus determined is the effective luminance range, and the difference DR (=
UL-LL) is the imaging range. This DR varies depending on the designed structure of the image pickup device, but in the case of a normal image pickup device, 5 to 6 EV (30 to 36 dB) is often used, and the image pickup device of the present embodiment also picks up images of this class. It has a range.

Therefore, the photoelectric conversion characteristics of the two image signals read from the CCD image pickup device 105 are schematically shown in FIG. In this figure, the first image signal is a signal imaged with a short exposure time, and the second image signal is a signal imaged with a long exposure time. Therefore, the photoelectric conversion characteristics of the two image signals have a relationship of being moved in the horizontal axis direction by an amount corresponding to the difference in the exposure time. As a result, these two image signals include the image signal about the brightness of the subject in the expanded brightness range even in the imaging device having the same imaging range DR.

In the present invention, the shift amount of the exposure time can be arbitrarily selected, and the exposure level may be changed by a parameter other than the exposure time, for example, an aperture.

Next, a method for obtaining one composite image from two image signals having different exposure levels will be described.

FIG. 4 is a view showing the arrangement of an image pickup apparatus relating to image combination. In the image synthesis processing, the image generation processing unit 122 provided in the system controller 112 and the matrix conversion processing unit 12 provided in the digital processing unit 108.
It is executed by exchanging signals with 0. And
The image generation processing unit 122 includes an indexing unit 123, an image signal selection unit 124, and an image generation unit 125.

First, the image synthesizing processor 122 performs dot-sequential signals (R, G, which are color signals of the first and second image signals).
B) is input to the digital process unit 108. Then, the matrix conversion processing unit 120 is operated to generate the synchronized component signals. Where the first
The component signal of the original image is S1, and the component signal of the second original image is S2. As a specific configuration of the representative component signal Sn (n = 1, 2 is shown), a luminance signal (Yn) and a color difference signal (Rn-Yn, Bn-).
Yn), and an embodiment using the luminance signal (Yn) and the color difference signals (Rn-Yn, Bn-Yn) will be described in the following embodiments.

In the first embodiment of the present invention, the indexing unit 123 uses the component signal S obtained by the above procedure.
Color saturation C, which is an index corresponding to chromaticity for each pixel from n
n is calculated by the equation (1).

Cn = {(Rn−Yn) ² + (Bn−Yn) ² } / (Yn + α) ² (1) where α: constant, where the color saturation Cn defined by the formula (1) is the present invention. The concept introduced in the above, and its physical meaning will be described based on the color difference coordinates in FIG. The color difference coordinates are coordinates in which the horizontal axis represents the color difference signal RY and the vertical axis represents the color difference signal BY.
Point Dn (Rn-Yn, Bn-Yn) corresponding to the color difference signal
Is represented by a color difference vector having an origin (0, 0) as a starting point and a deviation angle of θ.

In this coordinate system, the argument θ represents the hue,
The origin (0, 0) of the coordinates represents a colorless state. Therefore, the size of this color difference vector, that is, the size of the line segment connecting the point Dn and the origin corresponds to the product of the saturation, which is the color density, and the brightness, which represents the brightness (brightness), as a physical quantity. It is considered that the color intensity can be identified by using this size.

However, since the magnitude of the color difference vector includes the brightness as described above, it is not possible to judge only the color depth by this index. Therefore, in the formula (1), the magnitude of the color difference vector is divided by the luminance signal Y to be standardized, and is used as an index representing the color density. Generally, the conversion for normalizing a color image to a constant lightness is called chromaticity conversion, and the color saturation in the present invention is also a value obtained by normalizing the color difference by a luminance signal. Therefore, this index is called an index corresponding to chromaticity.

It should be noted that the above-mentioned index should originally be defined as √Cn, but it is defined as a value obtained by squaring the color saturation in the present embodiment so that the index calculation process can be performed quickly and easily. This is because Further, the denominator should be the square of the luminance signal Y from the original meaning, but the reason why the small offset α is added is that noise generated due to division when the value of the luminance signal Yn is small. This is to reduce the influence of the relative increase of the error due to the component and to avoid the division by zero error. Due to the action of the offset α, when the luminance level is very small (when the numerator and denominator are both very close to indefiniteness), the color saturation is considered to be minute.

Next, the image signal selecting section 124 compares the color saturation C1 of the first original image thus obtained with the color saturation C2 of the second original image for each pixel. Then, the component signal of the original image having a large color saturation is selected and used as the component signal of the image to be newly synthesized. That is, C1
If> C2 (or C1 >> C2), select S1
When C1 <C2 (or C1 << C2), S2 is selected. When C1 = C2 (or C1≈C2), (S1 + S2) / 2 is set as a new component signal.

Then, the image generator 125 generates one composite image based on the selected image signal. The pixel signal thus obtained is processed by the circuit in the subsequent stage in the same manner as the conventional image pickup signal, and is recorded in the memory card 110 or displayed on the LCD image display system 111. As described above, since the image having a large color density is selected for each pixel based on the color component of the original image, the effect of gradation compression is reduced as compared with the processing based on the luminance component. It is possible to obtain a natural image that is difficult to receive and has no false signal by a simple process.

In the present embodiment, the color saturation is calculated for each pixel, but the present invention is not limited to this form, and the original image is divided into regions including at least one pixel and the division is performed. The index may be calculated for each area.

Next, in the second embodiment according to the present invention, the indexing unit 123 calculates the color saturation Cn for each pixel from the component signal Sn by the formula (2) instead of the formula (1).

Cn = {(Rn-Yn) ² + (Bn-Yn) ² } * (Ym + α) ² (2) where α is a constant, and m means a number different from n. That is, n = 1
When, m = 2, and when n = 2, m = 1.

Now, assuming that the relation of C1> C2 is established, substituting equation (2) into this relational equation yields equation (3). ^{{(R1-Y1) 2 +} (B1-Y1) 2} * (Y2 + α) 2> {(R2-Y2) 2 + (B2-Y2) 2} * (Y1 + α) 2 ... (3) Then, the formula (3 ) Is transformed into formula (4). {(R1-Y1) ² + (B1-Y1) ² } / (Y1 + α) ² > {(R2-Y2) ² + (B2-Y2) ² } / (Y2 + α) ² (4) Formula (4) is , Which is equivalent to comparing the magnitude of the color saturation of the first embodiment. That is, when comparing two values,
Even if the magnitudes are compared using the values defined by the equation (2), the same result as the magnitude comparison of the values defined by the equation (1) is obtained.

Therefore, by using the value defined by the equation (2) as the color saturation, it is not necessary to use the operation function of division, so that the operation can be speeded up and simplified.

Next, in the third embodiment according to the present invention, the indexing unit 123 calculates the color saturation Cn for each pixel from the component signal Sn by the formula (5) instead of the formula (1).

Cn = {| Rn−Yn | + | Bn−Yn |} / (Yn + α) (5) where α: the index defined by the constant expression (5) is similar to the expression (1). The numerator is a value related to the magnitude of the color difference vector (however, the sum of the magnitudes of the components of the color difference vector), which is a simpler expression. By using this equation (5), it is possible to speed up and simplify the calculation as compared with the first embodiment because the square calculation is unnecessary.

Next, in the fourth embodiment according to the present invention, the indexing unit 123 calculates the color saturation Cn for each pixel from the component signal Sn by the formula (6) instead of the formula (1).

[0049] Cn = {| Rn-Yn | + | Bn-Yn |} * (Ym + α) (6) However, α is a constant, and n means a number different from m.

Expression (6) is a modification of the index used in the third embodiment according to the concept described in the second embodiment. Therefore, by using the equation (6), the square calculation and the division are not required, so that the calculation can be further speeded up and simplified.

The comparison and selection of the two pixels may be performed for all the pixels of the image signal, or the pixels in the luminance range other than the common luminance range shown in FIG. 3 may be processed.

Further, in each of the above-mentioned embodiments, the processing in the case of synthesizing two images has been described, but the present invention is not limited to this embodiment and may use three or more images. good. For example, it is possible to perform a process similar to the above using three images of A, B, and C whose exposure levels are large, medium, and small to select the one having the maximum color saturation Cn.

Further, in the present embodiment, the color saturation is defined using the luminance and the color difference signal, but the present invention is not limited to this form, and for example, the color saturation is defined based on the quantity defined in the L ^* a ^* b ^* space. It may be one that defines the depth.

The present invention may be configured by using hardware such as a circuit, or may be realized by incorporating a processing program in a microcomputer.

[0055]

As described above, by applying the present invention, it is possible to provide an image pickup apparatus capable of obtaining a natural picked-up image in which the D range is expanded.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image pickup apparatus to which the present invention is applied.

FIG. 2 is a diagram schematically showing photoelectric conversion characteristics of a general image sensor.

FIG. 3 is a diagram schematically showing photoelectric conversion characteristics of two image signals.

FIG. 4 is a diagram showing a configuration of an image pickup apparatus relating to image combination.

FIG. 5 is a diagram showing the physical meaning of color saturation.

[Explanation of symbols]

105 ... CCD image sensor 107 ... Preprocessing section 108 ... Digital Process Department 112 ... System controller

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B057 AA20 BA02 CA01 CA08 CA12                       CA16 CB01 CB08 CB12 CB16                       CC01 CE08 CE11 DA20 DB02                       DB06 DB09 DC25                 5C065 AA03 BB48 CC01 DD02 EE12                       EE18 GG21 GG23 GG27 GG32

Claims (7)

[Claims] 1. Indexing for processing an image signal of each of a plurality of original images obtained by giving different exposure amounts to calculate an index corresponding to chromaticity for each divided region composed of at least one pixel. Means, selecting means for selecting one image signal from among the image signals of the plurality of original images for each of the divided regions based on the index, and generating one combined image signal from the selected image signals. An image pickup apparatus comprising: an image generation unit. 2. The selecting means compares the respective indexes for each of the divided areas to determine an index having the largest absolute value, and selects an image signal of the original image to which the index is given. The image pickup apparatus according to claim 1. 3. The indexing means calculates an index C corresponding to chromaticity from the following equation using a luminance signal Y, color difference signals RY and BY, and a predetermined value α. The imaging device according to claim 1 or 2. C = {(RY) ² + (BY) ² } / (Y + α) ² 4. The indexing means comprises: a luminance signal Y, color difference signals RY and BY of one image;
The image pickup apparatus according to claim 1, wherein the index C corresponding to the chromaticity is calculated from the following equation using the luminance signal Y ^* of another image to be compared and a predetermined value α. ^{C = {(R-Y)} 2 + (B-Y) 2} * (Y * + α) 2 5. The indexing means comprises: a luminance signal Y, color difference signals RY and BY of one image;
The image pickup apparatus according to claim 1, wherein the index C corresponding to the chromaticity is calculated from the following equation using the luminance signal Y ^* of another image to be compared and a predetermined value α. C = {| R−Y | + | B−Y |} * (Y ^* + α) 6. An image signal of a plurality of original images obtained by giving different exposure amounts is processed, and an index corresponding to chromaticity is calculated for each divided area formed of at least one pixel, An image pickup method, characterized in that one image signal is selected from among the image signals of the plurality of original images for each of the divided areas based on an index, and one combined image signal is generated from the selected image signal. . 7. A processing program for an image pickup apparatus, wherein each of image signals of a plurality of original images obtained by giving different exposure amounts to a computer is processed, and a color is divided for each divided area including at least one pixel. A step of calculating an index corresponding to the degree, a step of selecting one image signal from the image signals of the plurality of original images for each of the divided areas based on the index, one combining from the selected image signals A program for executing a procedure for generating an image signal.