JPH07135599A - Image pickup device and its picture processing method - Google Patents

Abstract

PURPOSE:To obtain a digital picture with excellent S/N, high gradation, large dynamic range and high picture quality by the digital image pickup device. CONSTITUTION:A same object is picked up simultaneously or continuously by changing exposure and resulting picture data are converted into digital data, which are stored in a memory. Then the fluctuation of picture data are corrected to implement picture matching processing (step 11), the gain of the picture data is adjusted to make the levels coincident with each other (step 12). Then the picture data are added to synthesize the pictures (step 13), the resulting picture is converted into R, G, B signals by process processing of a next stage (step 14) and outputted to the monitor as a luminance signal Y and a color difference signal C (step 15).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus capable of obtaining a digital image having a low noise, a high gradation and a large dynamic range, and an image processing method thereof.

[0002]

2. Description of the Related Art Recently, digital image pickup apparatuses such as digital video movie cameras and digital still video cameras, which handle image signals of a subject as digital signals, have been put into practical use. In such an imaging device,
The output signal of the solid-state image sensor is sample-held, the gain is adjusted, the analog signal is converted into a digital signal by the A / D conversion circuit, and the signal is recorded in the memory. The image signal from the solid-state image sensor converted into a digital signal is converted into a luminance signal and a color difference signal of the NTSC / PAL system, or processed as R, G, B signals, and a D / A conversion circuit. After being converted into an analog signal by, the signal is output to a monitor or is also input to a computer or the like as digital image information.

Here, the gradation of the digital image pickup device as described above is determined by the A / D conversion circuit, and the A / D conversion circuit of 8 bits to 10 bits is currently put into practical use. However, at this level, the gradation in the low-luminance portion of the subject is insufficient, resulting in a rough image quality and giving an unpleasant feeling.

Further, in the conventional image pickup apparatus, the noise generated in the solid-state image pickup element and the random noise generated in the analog circuit system up to the input of the A / D conversion circuit are S / N of the final image.
However, this is also conspicuous in the low luminance portion, and a good image with high S / N cannot be obtained.

[0005]

The conventional digital image pickup apparatus is constructed as described above, and the gradation is determined by the A / D conversion circuit, and the noise generated in the image pickup element and the analog circuit are generated. Since the S / N of the final image is determined by the random noise that occurs, there is a problem that the image quality of the low-luminance portion deteriorates.

Therefore, it is conceivable that the exposure amount is increased and the image is picked up, the image pickup signal is converted into a digital signal, and then the gain is adjusted by calculation. The parts are saturated, and the information in the high brightness part is lost.

The present invention has been made by paying attention to the above problems, and an image pickup apparatus capable of obtaining a high quality digital image having a good S / N, a high gradation and a large dynamic range. And its image processing method.

[0008]

The image pickup device of the present invention is configured as follows.

(1) Conversion means for digitizing a plurality of image data of the same subject imaged with different exposure amounts, matching processing means for aligning the images of the digitized image data, and digitized. A level adjusting means for adjusting the level of each image data and an image combining means for combining the image data of the matched level are provided.

(2) In the image pickup apparatus of (1) above, the level adjusting means compares the average value of the saturated portion of the image data with the average value of the area excluding its periphery to match the levels.

(3) In the image pickup apparatus of (1) above, the level adjusting means extracts an area of the image data in which the change in luminance is small and compares the average values of the areas to match the levels.

(4) In the image pickup device of (1), the level adjusting means extracts a region close to white in the image data, compares the average values of the regions, and adjusts the levels.

(5) In the image pickup apparatus according to (1), the level adjusting means extracts an arbitrary area of the image data, compares the average values of areas excluding the saturated portion of the area, and sets the level. I tried to match it.

(6) In the image pickup apparatus according to any one of (1) to (5) above, the image synthesizing means sets the image data obtained with the exposure amount larger than the standard as the low-luminance side data, and the standard exposure amount is used. The obtained image data is the data on the high brightness side,
The image data obtained with the exposure amount smaller than the standard is used as the data on the ultra-high brightness side, and the image data is combined in all or in any combination.

(7) In the image pickup apparatus according to (6), the image synthesizing means synthesizes the image data with the data value of the changeover of the combination of the image data as a fixed value.

(8) In the image pickup device according to the above (6), the image synthesizing means can change the content ratio of the high-luminance side data and the low-luminance side data to the data value for switching the combination of the image data. Image data is composed with a certain width.

(9) In the image pickup apparatus according to (8), the image synthesizing means linearly changes the content ratio of the high-luminance side data and the low-luminance side data to synthesize the image.

(10) In the image pickup device according to (8) above,
The image synthesizing means synthesizes the image by changing the content ratio of the high-luminance side data and the low-luminance side data with a cosine curve.

Further, the image processing method of the present invention converts the respective image data obtained by simultaneously or continuously capturing the same subject with different exposure amounts into digital data, and converting the digitized image data. In addition to aligning the images, the levels of the image data are matched and then the image data are combined.

[0020]

According to the present invention, for example, a plurality of image data of a digital image obtained by photographing with standard exposure and a digital image obtained by simultaneously or continuously changing the exposure amount of an image, By performing processing such as correcting the blur component, processing such as gain adjustment that matches the levels of multiple digital images obtained with different exposure amounts, and processing that combines these multiple digital image data , A high S / N ratio, a high gradation, and a large digital range can be obtained.

[0021]

1 is a block diagram showing the structure of an image pickup apparatus according to an embodiment of the present invention. In the figure, 1a and 1b are solid-state image pickup devices that photoelectrically convert image pickup light from the same subject imaged with different exposure amounts, and 2 are these image pickup devices 1a and 1b.
AD (analog-digital) conversion means for digitizing a plurality of image data from b, 3 is a matching processing means for aligning the image with respect to each digitized image data, and here, each digital image. Adjust the position by correcting the blur.

Reference numeral 4 is a level adjusting means for adjusting the levels of the digitized image data, and 5 is an image combining means for combining the image data of the matched levels into one digital image data. The image signal to be processed is subjected to predetermined image processing by a process processing circuit (not shown),
The signals are output as R, G, B color signals, Y, C luminance signals, and color difference signals.

Next, the operation of the image pickup apparatus will be described. FIG. 2 is a flow chart showing the process of the image processing.

First, the image data (1) and (2) of the same subject under the exposure amounts of a plurality of conditions are taken into the memory from the image pickup devices 1a and 1b at the same time or almost the same time. here,
The same time is the case of a camera having an optical system for combining the same subject image with a plurality of solid-state image pickup elements by a prism or the like as in the multi-plate type camera, and in this case, the exposure amount of each solid-state image pickup element is different. As a means for doing so, the accumulation time of each solid-state image pickup device is made different, or an ND filter is inserted in front of the solid-state image pickup device. Further, "at substantially the same time" means a case where images are captured by continuous shooting using a single-chip camera at short intervals and under different exposure conditions (aperture aperture of shutter, shutter speed are different).

The plurality of image data are obtained by the solid-state image pickup device 1.
It is read as outputs a and 1b, sample-held, gain-adjusted, A-D converted, and recorded in each memory.

The image data (1) and the image data (2) are image data stored in the memory after A / D conversion, and in this embodiment, the image data (1) was obtained by standard exposure. The data and the image data (2) are data obtained with an exposure amount three times as large as the standard exposure, and both are digitized by a 10-bit (bit) A-D conversion circuit. Therefore, the gradation has approximately 1024 levels.

Although not all 10 bits are actually used for image data but some offset is added, it is assumed here that all 10 bits are image information.

Image data (1) and image data (2)
Then, the image matching process is performed (step 11).
This is not necessary when a multi-plate optical system is used as an image capturing means, but in the case of a single-plate system,
Since the image data (1) and the image data (2) are photographed at slightly different times, the positional relationship of the images is slightly blurred. From this, the image data (1) and the image data (2) are compared, and the coordinates are corrected so that the position of the image data (1) and the position of the image data (2) match. . As a method of this processing, "blur prevention" that is currently in practical use in video movie cameras, etc.
The following method is adopted.

Next, after the positional relationships of the two images are matched, the image data (2) is added to the image level of the image data (1).
The gain adjustment processing is performed so that the data of (1) match (step 12). At this time, it suffices to multiply the image data (2) taken with an exposure amount three times the exposure amount of the image data (1) by 1/3, but in reality, the exposure amount of the camera is an error due to the system, that is, Since there are errors such as mechanical errors of the diaphragm and shutter, errors of the photometric system, etc., highly accurate gain adjustment by comparison of actual image data is required.

In the above, since the data to be compared must exclude the saturated part, for example, 10 of the image data (2) is used.
00 levels or less (Be sure to exclude the saturation value from the saturation value 1024 without being affected by noise etc. 1
024), image data (1)
The level of data to be compared is 333 or less.

Further, since the influence of the noise component becomes large in the data in the low luminance part, it is necessary to exclude this.
In the present embodiment, the lower limit stage is 100 for image data (1).
The stage and the image data (2) are set to 300 stages. Normally, the level of 100% white is set to 1/3 to 1/4 of the saturation, so the level used for gain adjustment is
It is performed in a portion having a white level of 30-40% or more.

Then, the average value of the area defined as described above is taken in each image, the average value of the image data (1) is A, the average value of the image data (2) is B, and the image data (2 ), Where C is the data value of each pixel, A ÷ B × C
And the level of the image data (2) is made the same as that of the image data (1).

Here, as another method of measuring the gain ratio of the image data (1) and the image data (2), a method of obtaining a higher accuracy than the above method is used. It is possible to adopt a method in which the average value of each image is compared with each other by extracting a specific area and limiting the area to a small area.

As yet another method, a region close to white in the screen may be extracted and the average value of only that region may be compared. According to this method, each color pixel (complementary color solid-state image sensor is used). If so, the levels of Y, MG, G, and Cy are balanced, and if the colors are pure, the levels of R, G, and B) are balanced, so that the accuracy can be further improved.

Further, the comparison area is determined in advance such as the central portion of the screen or the upper portion of the screen, and unless it is saturated, the average values in the determined area are compared in advance, so that the accuracy is slightly lowered. However, the processing time can be significantly reduced.

When the image matching process and the gain adjusting process are completed as described above, the image adding process for adding the image data (1) and the image data (2) is performed (step 1).
3). In the addition of this data, the image data (1) is used as high-luminance side data, and the image data (2) is used as low-luminance side data.

FIG. 3 shows the above-mentioned addition processing method. The image data (2) obtained by the above gain adjustment processing is almost 1
Since it is / 3, the maximum level of the image data (2) is almost 341. IN (1) of this graph (a) is the data of each pixel of the image data (1), IN of (b)
The data of each image of the image data (2) in (2), and N1
Is a coefficient for the pixel data of the image data (1), N2
Is a coefficient related to the pixel data of the image data (2).

Then, according to the above graphs (a) and (b), (IN (1) × N1) + (IN (2) × N2) is calculated to form one image. In this case, the data of image data (2) is used for the data less than 330, which is slightly lower than the saturation value 341 of the image data (2), and the data of image data (1) is used for the data of 330 or more. By doing this, the S / H of the portion of 1/3 or less of the saturation of the image is obtained.
Is 1/3, the gradation is tripled, and an image of high quality and high information is created.

However, since the information around the data value 330 is disturbed due to the gain adjustment error or the like, the values around the data value 330 are represented as false colors, and a band of false color may be generated. The sex remains.

Therefore, as shown in FIGS. 4 and 5, the data width of the switching portion between the image data (1) and (2) is given, and the information of the image data (1) and (2) is included, and the ratio thereof is It is effective to gradually change. Figure 4
The content ratio is linearly changed, and FIG. 5 is changed with a cosine curve. Needless to say, N1 + N2 is always 1. By doing so, it is possible to reduce the influence of a gain error or the like at the switching portion between the image data (1) and the image data (2) and suppress the occurrence of false images.

Next, the image data synthesized as described above is converted into R, G, B signals in the process process (1) of the next stage and output (step 14). Further, it is converted into a luminance signal Y and a color difference signal C in the process processing (2) (step 15). Then, these signals can be input to the monitor television or input to the computer as image data.

In the above example, the switching level of the image data (1) and the image data (2) is set to 1/4 of the saturation level.
Although it came to about 1/3, it is also possible to have a part that switches to the high-brightness part of the gamma curve attached in the later process processing, in which case the effect of the above-mentioned gain error etc. should be further reduced. You can

By performing the image processing as described above, it is possible to obtain a high quality digital image having a good S / N, a high gradation and a large dynamic range.

FIG. 6 is a flow chart showing the operation of another embodiment of the present invention. The image data (1) and the image data (2) are the same as those in the above-mentioned embodiment. The image data (3) is image data taken with an exposure of 1/3 of the standard exposure amount.

The three image data are aligned by the image matching process (step 11), and then the gains of the three images are adjusted (step 12). At this time, the image data (2) is almost ⅓ and the image data (3) is tripled. As a result, the image data (3)
The maximum value of is expanded to 3072.

Next, the image data (1) and the image data (2) are combined in the same manner as in the above-mentioned embodiment, and the image data (3) and the image data (1) are also the image data (1) and the image data. They are combined in the same manner as the combination of (2) (step 13). In this case, the switching level between the image data (1) and the image data (3) is in the range of 700 to 1000.

By carrying out such image processing, the dynamic range of the output digital image is greatly expanded as compared with the image output in the above-mentioned embodiment. Therefore, even a subject image in the ultra-high brightness portion can be clearly projected, and a low-luminance image can be realized with a clear image having a large amount of information with little noise.

As another embodiment, only the above image data (1) and image data (3) may be combined. In this case, the processing method is the same as that in each of the above-described embodiments. Alternatively, only the image data (2) and the image data (3) may be combined.

Further, in the above-mentioned embodiments, the case of 3 times and 1/3 times of the standard exposure has been described, but if it is set to 4 times and 1/4 times, for example, a higher quality image can be obtained. .

[0050]

As described above, according to the present invention, by synthesizing a plurality of images of the same subject imaged with different exposure amounts, the S / N is good, the gradation is high, and the dynamic range is wide. There is an effect that a high quality digital image can be obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of one embodiment.

FIG. 3 is an explanatory diagram showing an example of image addition processing.

FIG. 4 is an explanatory diagram showing another example of image addition processing.

FIG. 5 is an explanatory diagram showing another example of image addition processing.

FIG. 6 is a flowchart showing the operation of another embodiment of the present invention.

[Explanation of symbols]

 1a, 1b Solid-state image sensor 2 A-D conversion means 3 Matching processing means 4 Level adjusting means 5 Image combining means

Claims (11)

[Claims] 1. A conversion means for digitizing a plurality of image data of the same subject imaged with different exposure amounts, a matching processing means for aligning the images of the digitized image data, and a digitized image. An image pickup apparatus comprising: a level adjusting means for adjusting the level of each image data; and an image combining means for combining the image data of the matched level. 2. The image pickup apparatus according to claim 1, wherein the level adjusting means compares the average value of a region excluding a saturated portion of the image data with an area excluding the saturated portion to adjust the level. 3. The image pickup apparatus according to claim 1, wherein the level adjusting means extracts an area of the image data in which the luminance change is small and compares the average values of the areas to adjust the levels. 4. The image pickup apparatus according to claim 1, wherein the level adjusting means extracts a region close to white in the image data, compares the average values of the regions, and adjusts the levels. 5. The level adjusting means extracts an arbitrary area of the image data, compares the average values of areas excluding the saturated portion of the area, and adjusts the levels. Imaging device. 6. The image synthesizing means sets image data obtained with an exposure amount larger than a standard as low-luminance side data, and image data obtained with a standard exposure amount as high-luminance side data, which is less than the standard. 6. The image pickup apparatus according to claim 1, wherein the image data obtained by the exposure amount is used as data on the ultra-high brightness side, and the image data is combined in all or in any combination. 7. The image pickup apparatus according to claim 6, wherein the image synthesizing unit synthesizes the image data by using a data value for switching a combination of the image data as a fixed value. 8. The image synthesizing means synthesizes image data by giving a data value for switching a combination of image data a certain width in which a content ratio of high-luminance data and low-luminance data can be changed. The image pickup apparatus according to claim 6, wherein: 9. The image pickup apparatus according to claim 8, wherein the image synthesizing unit synthesizes the image by linearly changing the content ratio of the high-luminance side data and the low-luminance side data. 10. The image pickup apparatus according to claim 8, wherein the image synthesizing unit synthesizes the image by changing a content ratio of the high-luminance side data and the low-luminance side data with a cosine curve. 11. The image data obtained by simultaneously or continuously capturing the same subject with different exposure amounts is converted into digital data, and the image is aligned with the digitized image data. The image processing method is characterized in that the levels of the image data are matched and then the image data are combined.