JPH05145857A - Wide dynamic range camera - Google Patents

Abstract

PURPOSE:To shorten one image pickup time in the wide dynamic range camera. CONSTITUTION:Micro lenses 12 are arranged in a checkered pattern for all picture elements of an image pickup element 10 capable of high-speed nondestructive read out. The nondestructive read out of picture information of the picture element group corresponding to the presence or absence of the micro lens 12 is performed at the end of 1/8, 1/2, and 1 exposure period. For each read out, the integrated processing of the picture stored in frame memories 22A and 22B is performed by adders 20A and 20B. After the sub-Nyquist recovery processing of sub-Nyquist sampling interpolation processing circuits 24A and 24B, the picture signals after the interpolation processing corresponding to the presence or absence of the micro lens 12 is added to be displayed on a display part 34.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a wide dynamic range video (TV) used in the field of electronic image technology.
The present invention relates to a camera, and more particularly, to shortening an image pickup time in such a camera.

[0002]

2. Description of the Related Art Conventionally, as a method of widening the dynamic range of an image pickup device or a picked-up image, a method of accumulating image signals at high speed a number of times to take in an image signal having a high SN ratio is disclosed in, for example, the applicant of the present invention. Japanese Patent Application No. 3-2599
Proposed in No. 86. At this time, the cumulative addition number, the exposure time, and the aperture value are optimized according to the dynamic range of the subject.

Alternatively, for example, Japanese Patent Application No. 62-23413
As disclosed in No. 3, the height of the OFD (overflow drain) gate or the transfer gate of the image pickup device is changed in several steps or continuously during one exposure period to widen the image pickup output signal of the device. It is also known to have a dynamic range.

[0004]

However, each of the conventional methods has a problem that it takes time to capture one image. That is, it takes a long time to read out the image signal after one exposure, and as a result, the imaging time which is the exposure time and the reading time becomes long.

Therefore, it is difficult for a wide dynamic range camera to pick up a moving image like a normal camera. Further, when a moving object is imaged by such a wide dynamic range camera, there is a problem that the image quality is deteriorated due to a large error or camera shake or the like.

The present invention has been made in view of the above points, and it is possible to shorten the image pickup time by one, and thereby to improve the deterioration of the image quality such as the real-time property of the moving image and the reduction of the shake. It is intended to provide a wide dynamic range camera.

[0007]

In order to achieve the above object, a wide dynamic range camera of the present invention is provided with an image pickup device capable of high-speed nondestructive read and a part or all of pixels of the image pickup device. Readout means for performing nondestructive read-out of the image information of the arranged microlenses for condensing on the photosensitive portion of the pixels and the pixel information corresponding to each presence or absence of the microlenses or each condensing rate for a plurality of times during one exposure period. It is characterized by having and.

Here, it is preferable that the microlenses are arranged in a checkered pattern with respect to all pixels of the image pickup device.
Further, it is possible to further include an arithmetic means for performing integration processing with a previously read image and sub Nyquist sampling recovery processing for each reading.

[0009]

That is, according to the wide dynamic range camera of the present invention, the microlenses for condensing are arranged in a checkered pattern on the image pickup surface of the image pickup device capable of high-speed nondestructive reading, and one reading period is set by the reading means. During this, non-destructive read is performed multiple times at high speed. The read-out image signal is subjected to integration processing with the previously read-out image and recovery processing of sub-Nyquist sampling for each of the presence or absence of the microlens and the light collection ratio by the calculating means, and a plurality of types having different sensitivities are obtained. Get a full size image of.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1A is a block diagram of a wide dynamic range camera according to the first embodiment of the present invention. In the figure, reference numeral 10 is an image sensor capable of high-speed reading and non-destructive reading, for example, a charge modulator (Ch
arge Modulation Device: hereinafter abbreviated as CMD) can be used. A solid-state image pickup device using this CMD image pickup device is disclosed in JP-A-61-84059 and International Electr held in 1986.
"A NEW MOS IMAGE SENSOR OPERATING" on pages 353 to 356 of the proceedings of onDevice Meeting (IEDM)
The content is disclosed in a paper entitled "IN A NON-DESTRUCTIVE READOUT MODE".

Further, reference numeral 12 is a microlens arranged on the image pickup surface of the image pickup device 10 every other pixel in a checkered pattern corresponding to each pixel 10A as shown in FIG. 1B. is there. Here, in the present embodiment, the microlens 12 has a light collection ratio such that the aperture of the light receiving surface of each pixel 10A is, for example, twice as large as that of the pixel without the pixel with the microlens 12. You are using what you have. As a result, the pixel with the microlens 12 has twice the sensitivity of the pixel without the microlens 12.

Reference numeral 14 is an amplifier for amplifying an image signal read nondestructively from the image pickup device 10, and 16 is an analog / digital (A) for converting the amplified image signal into a digital signal.
/ D) converter. 16 is a pixel clock (CL
The switch 18 is switched by a K) signal.

Reference numeral 20A is an adder, and 22A is a first frame memory (FM) for storing the output of the adder 20A.
1) 22A. The frame memory 22A has a capacity half that of a normal frame memory. The image signal stored in the frame memory 22A is the adder 2
0A and is integrated with the signal read from the next image sensor 10. Reference numeral 24A denotes a sub-Nyquist sampling interpolation processing circuit 24A which performs interpolation processing of missing pixels on the image signal of the frame memory 22A which has been subjected to a predetermined number of times of integration processing. Similarly, 20B is an adder, 22
B is a second frame memory (FM2), and 24B is a sub-Nyquist sampling interpolation processing circuit 24B.

Reference numeral 26 denotes the interpolation processing circuits 24A and 24B.
It is an adder that obtains an image signal having a wide dynamic range by adding the two image signals interpolated by each pixel signal at the same position (address). Reference numeral 28 is a third frame memory (FM3) for storing an image signal of a wide dynamic range, and 30 is a wide area for performing an adaptive process on the image signal stored in the frame memory 28 to the dynamic range of the display system in the subsequent stage. It is a processing unit for displaying a dynamic (D) range image. And 32 is digital /
An analog (D / A) converter 32, and a display unit 34 such as a monitor. Next, the operation of the wide dynamic range camera configured as described above will be described.

In this embodiment, a plurality of times, for example, 3 times, during one exposure period (for example, 1 frame: 1/30 seconds).
The non-destructive reading is performed at high speed from the image pickup device 10 capable of high-speed reading and non-destructive reading.

That is, first, nondestructive read is performed at high speed once once the 1/8 exposure time has elapsed. The read signal is stored in different frame memories 22A and 22B for each pixel with or without the microlens 12 by switching the switch 18 by the pixel CLK signal. As a result, the second frame memory 22B for storing the signals read out from the pixels without the microlens 12 stores the image signal of the luminance level of 1/8, while the second frame memory 22B stores the signal of the microlens 12. In the first frame memory 22A for storing the signal read from a certain pixel, the microlens 12 gives twice the sensitivity to the corresponding pixel, so that the image signal of the luminance level of 1/4 is stored. Becomes

Next, when the 1/2 exposure time has elapsed, nondestructive read is performed once more at high speed. The read signals are integrated with the signals of the frame memories 22A and 22B by the adders 20A and 20B for each presence / absence of the microlens 12, and then the frame memories 22A and 22B, respectively.
Stored in B. In this case, the signal read from the image sensor 10 becomes an image signal having a brightness level of 1 and a brightness level of 1/2 depending on the presence or absence of the microlens 12.

Next, when one exposure time has elapsed, reset reading (normal reading) of the image pickup device 10 is performed. The read signals are added by the adders 20A and 20B to the frame memory 2 depending on the presence or absence of the microlens 12, respectively.
After being integrated with the signals of 2A and 22B, they are stored in the frame memories 22A and 22B, respectively. In this case, the signal read from the image sensor 10 becomes an image signal having a brightness level of 2 and a brightness level of 1 depending on the presence or absence of the microlens 12.

In this way, when one exposure period ends, the image data of the first and second frame memories 22A and 22B are input to the corresponding sub-Nyquist sampling interpolation processing circuits 24A and 24B, respectively, and the missing pixels are lost. Is interpolated. As the interpolation at this time, for example, four-point interpolation using pixel data of upper, lower, left, and right of the corresponding missing pixel as shown in FIG. 2A can be considered.

The interpolated image signals are added by the adder 26 for each pixel signal at the same position (address) to obtain an image signal having a wide dynamic range. As a result, when one exposure period ends, 1
A combined image signal of image data having different brightness levels of / 8, 1/4, 1/2, 1 and 2 is obtained,
This means that a wide dynamic range 16 times higher than usual can be achieved by three times (high speed) cumulative addition.

The wide dynamic range image thus obtained is stored in the third frame memory 28, and after the wide dynamic range image display processing unit 30 in the subsequent stage is subjected to the adaptive processing on the dynamic range of the display system. , D
The signal is converted into an analog signal by the / A converter 32 and displayed on the display unit 34 such as a monitor.

In this embodiment, as compared with the system in which a normal CCD is used for image pickup, almost no read time is required, so that an image having a wide dynamic range of 16 times can be obtained in 1/6 image pickup time.

FIG. 2B is a block diagram of the wide dynamic range camera according to the second embodiment of the present invention. In the figure, the same parts as those in FIG. 1A are designated by the same reference numerals, and the description thereof will be omitted.

In the figure, reference numerals 26A and 26B are switches which are switched by a frame clock (CLK) signal, and 38A to 38F are frame memories.
40A to 40F are dispersion calculators for obtaining the dispersion of the brightness levels of the image signals stored in the corresponding frame memories, and 42 is a frequency from low brightness to high brightness based on the outputs of these dispersion calculators 40A to 40F. It is a determination unit that determines the frame memory output with the least distribution bias.
Reference numeral 44 is a switch for selectively supplying the outputs of the frame memories 38A to 38F to the D / A converter 32,
Reference numeral 46 denotes a switch for switching the switch 44 according to an auto / manual selection signal according to the determination result of the determination section 42 or manual input.

Next, the operation of the second embodiment having such a configuration will be described. First, nondestructive read-out is performed once at a high speed from the image sensor 10 similar to that of the first embodiment, when 1/8 exposure time in one exposure period elapses. The read signal is stored in the first frame memory (FM1) 38A and the second frame memory (FM2) 38B for each pixel with or without the microlens 12.

Next, when 1/2 exposure time has elapsed,
Further, non-destructive read is performed once at high speed. The read signal is stored in the third frame memory (FM3) 38C and the fourth frame memory (FM4) 38D for each pixel with or without the microlens 12.

Next, when one exposure time has elapsed, reset reading is performed, and depending on the presence / absence of the microlens 12, they are stored in the fifth frame memory (FM5) 38E and the sixth frame memory (FM6) 38F, respectively. ..

Then, the first to sixth frame memories 3
The outputs of 8A to 38F are the corresponding distributed computing units 4 respectively.
The brightness level dispersion is obtained from 0A to 40F, and the determination unit 42 outputs the frame memory output that does not rely on the frequency distribution from the low brightness to the high brightness (the smallest) based on the outputs of the dispersion calculators 40A to 40F. The switch 44 is controlled to be switched so that is displayed on the display unit 34 via the D / A 32. Alternatively, the switch 46 can be switched by an auto manual selection signal so that the observer can manually switch and display the output of each frame memory. As described above, in the second embodiment, it is possible to display a plurality of images having different brightness levels obtained during one exposure time.

Next, as a third embodiment of the present invention, an embodiment in which the shape and arrangement of the microlenses of the first and second embodiments are changed is shown in FIGS. 3 (A) to 3 (C). It will be explained based on.

The third embodiment is similar to the first and second embodiments in that it has a microlens 12 having a double light collection rate.
Two types of microlenses, A (FIG. 3A) and a microlens 12B (½ of FIG. 3B) having a light collection rate of 1/2, are arranged as shown in FIG. 3C. It was done.
That is, as shown in FIG. 3C, the microlenses 12A are alternately arranged between the pixels in which the microlenses are not arranged.
And a microlens 12B are arranged.

In this embodiment, when exposure is performed three times as in the first and second embodiments, 1/16, 1/8, 1
Since image signals obtained by synthesizing image data having different brightness levels of / 4, 1/2, 1 and 2 can be obtained, a wide dynamic range 32 times wider than that of the above-described embodiment can be achieved.

As described above, by appropriately arranging microlenses having different light-condensing rates in combination according to the purpose of use such as the brightness of the subject and the image quality, and setting an appropriate digital filter in the interpolation processing section 24. , A wider dynamic range can be achieved.

In the above embodiment, an example of one image pickup element is shown, but a color camera can be constructed by combining the RGB dichroic prism and the three image pickup elements. Also, the first and second embodiments may be combined.

[0034]

As described in detail above, according to the present invention,
(1) It is possible to provide a wide dynamic range camera capable of shortening the image pickup time and thus improving deterioration of image quality such as real-time property of moving images and reduction of shake. That is, the imaging time can be shortened as compared with the conventional method.

Since the sub-sample is picked up for each sensitivity level, the horizontal and vertical resolutions do not deteriorate.
In other words, the CCD can be used without degrading the horizontal and vertical resolution.
Can take twice as much information.

[Brief description of drawings]

FIG. 1A is a block configuration diagram of a first embodiment, and FIG. 1B is a front view showing a relationship of arrangement of microlenses with respect to an image pickup element.

2A is a diagram for explaining four-point interpolation processing in the sub-Nyquist sampling interpolation processing circuit in FIG. 1A, and FIG. 2B is a block configuration of the second embodiment. It is a figure.

3A is a cross-sectional view showing the shape of a microlens having a double light collection ratio, and FIG. 3B is a cross-sectional view showing the shape of a microlens having a half light collection ratio, FIG. 9C is a front view showing the relationship of arrangement of the microlenses (A) and (B) in the third embodiment.

[Explanation of symbols]

10 ... Image sensor, 12, 12A, 12B ... Micro lens, 18, 36A, 36B, 44, 46 ... Switch, 2
0A, 20B, 26 ... Adder, 22A, 22B, 38A
-38F ... Frame memory, 24A, 24B ... Sub Nyquist sampling interpolation processing circuit, 40A-40F ... Distributed arithmetic unit, 42 ... Judgment part.

Claims (3)

[Claims] 1. An image sensor capable of high-speed non-destructive reading, a microlens arranged for some or all pixels of the image sensor and condensing on a photosensitive portion of the pixel, and the presence or absence of the microlens. A wide dynamic range camera comprising: a reading unit that performs nondestructive reading of image information of a pixel group corresponding to each condensing rate a plurality of times during one exposure period. 2. The wide dynamic range camera according to claim 1, wherein the microlenses are arranged in a checkered pattern with respect to all pixels of the image pickup device. 3. The wide dynamic range camera according to claim 1, further comprising arithmetic means for performing integration processing with a previously read image and recovery processing of sub-Nyquist sampling for each reading.