JPH0564083A - Solid-state electronic image pickup device - Google Patents

Abstract

PURPOSE:To obtain image signals of different exposure amounts from a solid- state electronic image pickup device with single plate without exposing for two times. CONSTITUTION:Many photodiodes 60A and 60B are arrayed in both vertical and horizontal directions and filters are provided on the photodiodes 60A arranged every two columns in the vertical direction. The first signal charge accumulated in the photodiode 60A which is provided with a filter is inputted, by way of a vertical transfer path, into the first horizontal transfer path 63 and thus it is obtained from the first horizontal transfer path 63, and the second signal charge accumulated in the photodiode 60B which is not provided with a filter is inputted, by way of a vertical transfer path, into the second horizontal transfer path 65 and thus it is obtained from the second horizontal transfer path 65. Without exposing two times, image signals of different exposure amounts are obtained from the first and the second signal charges of a single plate CCD, and so, an inserting synthesization image processing is made possible.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a solid-state electronic image pickup device such as a CCD.

[0002]

2. Description of the Related Art Solid-state electronic image pickup devices such as CCDs are used in image pickup optical systems such as movie video cameras and electronic still cameras. Since the solid-state electronic image pickup device has characteristics such as light weight and small size, a camera using the solid-state electronic image pickup device can be made lighter and smaller.

[0003]

However, since the solid-state electronic image pickup device has a relatively narrow dynamic range,
When the brightness difference between the bright portion and the dark portion included in the visual field is large, it is difficult to properly expose and photograph both.

For example, there are cases where the background is very bright and the main subject in the center is dark, there is backlighting, and the outside scene is reflected through the window when shooting in a room with a window. When shooting such scenes, if the exposure is adjusted so that the main subject (often a person) is properly exposed, the amount of incident light exceeds the dynamic range of the solid-state electronic image pickup device in the bright background area. Is saturated, so the bright background is not captured and the image in that part is simply white (overexposure).

Therefore, in general, a device is devised to increase the brightness of the main subject by strobe light emission (daytime synchrocross flash photography). However, a stroboscopic device is required for daytime synchro-stroboscopic photography, the camera becomes large, and its operation is troublesome. In addition, when the dark part is in the distance and when it extends from a short-distance location to a distant area, the daytime syncrostrography is not always effective.

Therefore, the inventor has developed a technique for obtaining an appropriate image signal for a subject having a large luminance difference between a bright area and a dark area without necessarily emitting a strobe light, and applied for a patent on the same date as the present application. (The title of the invention is “video camera, shooting method using the same, operating method thereof, and image processing apparatus and method”; serial number 91
167; Applicant Fuji Photo Film Co., Ltd. and Fuji Film Microdevice Co., Ltd.) (hereinafter referred to as “inset synthetic patent application”).

[0007] The basic idea of the invention of this embedded composite patent application is to capture an image of a subject with two different exposure amounts to obtain image signals respectively representing two images with different exposure amounts, and to represent an image with a large exposure amount. In the image signal,
An image signal representing a region having a relatively high brightness in the image is replaced with an image signal representing a corresponding region in the image with a small exposure amount to create a combined image signal.

It is an object of the present invention to provide a solid-state electronic image pickup device capable of obtaining an appropriate video signal for a subject having a large brightness difference between a bright area and a dark area without necessarily emitting a strobe light. It is an object of the present invention to obtain an image signal representing two images having different exposure amounts, which are used in the invention of the synthetic patent application.

In particular, it is an object of the present invention to be able to obtain two image signals having different exposure amounts without using a single plate and exposing twice.

[0010]

A solid-state electronic image pickup device according to a first aspect of the present invention is arranged in a vertical direction and a horizontal direction, and has a large number of photoelectric conversion elements for accumulating an amount of signal charges according to an amount of incident light. A filter that is provided in the photoelectric conversion element in the horizontal direction at predetermined intervals and that limits the amount of light incident on the photoelectric conversion element, is provided adjacent to each column in the vertical direction of the photoelectric conversion element, and responds to a transfer signal. And a vertical transfer unit that separately transfers the first signal charge accumulated in the photoelectric conversion element provided with the filter and the second signal charge accumulated in the photoelectric conversion element not provided with the filter, A first horizontal transfer section for horizontally transferring the first signal charge input from the vertical transfer section, and a first horizontal transfer section for horizontally transferring the second signal charge input from the vertical transfer section. Characterized in that it comprises a horizontal transfer unit.

It is preferable to further include moving means for moving all of the photoelectric conversion elements or the filter in the vertical direction by one row of the photoelectric conversion elements.

A solid-state electronic image pickup device according to a second aspect of the present invention comprises a large number of photoelectric conversion elements arranged in the vertical and horizontal directions and accumulating signal charges of an amount corresponding to the amount of incident light, and horizontal at predetermined intervals in the vertical direction. Provided on the photoelectric conversion element in the direction, and is provided on the photoelectric conversion element not provided with a first filter that limits the amount of light incident on the photoelectric conversion element according to the first transmittance, and the first filter is not provided. , Second
Second filter for limiting the amount of light incident on the photoelectric conversion element depending on the transmittance of the photoelectric conversion element, provided adjacent to each column in the vertical direction of the photoelectric conversion element, and provided with the first filter according to a transfer signal. And a vertical transfer unit for separately transferring the first signal charge accumulated in the photoelectric conversion element and the second signal charge accumulated in the photoelectric conversion element provided with the second filter, respectively. A first horizontal transfer unit for horizontally transferring the first signal charge input from
In addition, a second horizontal transfer unit for horizontally transferring the second signal charges input from the vertical transfer unit is provided.

All of the above photoelectric conversion elements or the above first
It is preferable to further include a moving unit that moves the filter and the second filter in the vertical direction by one row of the photoelectric conversion elements.

[0014]

According to the first invention, there are a photoelectric conversion element provided with a filter and a photoelectric conversion element not provided with a filter.

According to the first invention, the first signal charges accumulated in the photoelectric conversion element provided with the filter and the second signal charges accumulated in the photoelectric conversion element not provided with the filter are distinguished from each other. Transferred through the vertical transfer unit.

According to the first invention, the first signal charge is obtained from the first horizontal transfer section and the second signal charge is obtained from the second horizontal transfer section.

According to the second invention, there is a photoelectric conversion element provided with the filter having the first transmittance and a photoelectric conversion element provided with the filter having the second transmittance.

According to the second invention, the first signal charge and the second signal charge accumulated in each photoelectric conversion element are transferred by the vertical transfer section.

Also according to the second aspect, the first signal charge is obtained from the first horizontal transfer section and the second signal charge is obtained from the second horizontal transfer section.

[0020]

In both the first and second inventions, the first signal charge is obtained from the first horizontal transfer path and is represented by the first signal charge from the second horizontal transfer path. A second signal charge representing an exposure dose different from the exposure dose is obtained. It is possible to obtain two image signals with different exposure amounts from one solid-state electronic image pickup device without exposing twice, and by combining these image signals, it is possible to cope with an image pickup having a portion with a large luminance difference. Become.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Before describing a solid-state electronic image pickup device according to the present invention, an image fitting process using two types of image signals obtained from the solid-state electronic image pickup device according to the present invention will be described.

The basic concept of the image fitting process will be described with reference to FIGS. 1 to 4.

FIG. 1 shows an example of a scene that falls within the field of view of the camera when shooting is performed in a room with windows.

The image SL of the room RM (reference number SL is also used to represent a part or region of this image) is relatively dark.
The outside landscape is visible through the window WN, and the image SH of this landscape (also used to indicate the part or area of this image SH) is relatively bright. Generally speaking, the average brightness of the image SH seen through the window WN is about 5 to 10 times the average brightness of the image SL of the room RM. The dynamic range of an image sensor such as a pick-up tube or CCD is such that the average brightness of a relatively bright part is 2 to 3 of the average brightness of a relatively dark part.
It can only handle double the number of scenes. That is, when the brightness difference is 2 to 3 times or more, when the dark part is properly exposed, the sensor saturates the bright part (whiteout occurs when reproduced), and when the bright part is properly exposed, the dark part is darkened. The image is barely displayed and the ratio of noise components in the video signal becomes extremely high.

FIG. 2 shows a histogram of photoelectric conversion characteristics and brightness of the image sensor.

The solid line SL1 shows the photoelectric conversion characteristic of the image sensor when the relatively dark image SL is properly exposed, and the solid line SL2 shows the relatively dark image SL imaged under the photoelectric conversion characteristic SL1. The frequency distribution of luminance is shown.

On the other hand, the broken line SH1 shows the photoelectric conversion characteristic of the image sensor when the relatively bright image SH is properly exposed, and the broken line SH2 is the relative image picked up under the photoelectric conversion characteristic SH1. The frequency distribution of the brightness of the bright image SH is shown.

As is apparent from this figure, the relatively bright image SH is completely included in the saturated region of the photoelectric conversion characteristic SL1, and under the photoelectric conversion characteristic SH1, the output representing the relatively dark image SL is output. Signal levels concentrate in very low places.

FIG. 3A shows an image taken with an appropriate exposure amount (the exposure amount is relatively large) with respect to the brightness of the room RM. The image SLa of the room RM is a proper image because it is properly exposed. On the other hand, the landscape image SHa that appears through the window WN is white.

FIG. 3B shows an image taken with an appropriate exposure amount (the exposure amount is relatively small) with respect to the image SHb of the window WN. The image SHb of the landscape seen through the window WN is good, but the image SLb of the room RM is considerably dark.

According to this process, an image of a saturated area in an image with a relatively large exposure amount (not necessarily saturated, it is sufficient if the area is relatively brighter than other areas). , The image is fitted by being replaced with the image of the corresponding region in the image with a relatively small exposure amount. That is, the image SHa of the window WN in FIG. 3 (A) is replaced with the image SHb of the window WN shown in FIG. 3 (B). As a result, as shown in FIG. 1, both the indoor RM and the window WN become good images, and an image close to the scene seen by human eyes can be obtained. FIG. 4 shows a histogram of the image that has been fitted and synthesized in this way. The two brightness distributions SL2 and SH2 are very close.

An image of a relatively dark area in an image having a relatively small exposure amount may be replaced with an image of a corresponding area in an image having a relatively large exposure amount. For example, the image SLb of the indoor RM in FIG.
Replace with the image SLa of the indoor RM shown in (A).

FIG. 5 shows a still video stream using this process.
The structure of a camera (electronic still camera) is shown.

The image pickup optical system includes an image pickup lens 11, a diaphragm 12, a shutter 13 and a solid-state electronic image pickup device (image sensor).
CCD 14 is included. The exposure control circuit 10 includes a CPU, and this circuit 10 includes an aperture 12 and a shutter.
It controls the clearing of charges in the CCD 13 and the CCD 14 and the reading of signals.

In this embodiment, preliminary photographing (preliminary exposure) and 2
The main shooting (main exposure) is performed once.

Preliminary photographing is for determining an appropriate exposure amount in each of the two main photographing. The exposure control circuit 10 determines the exposure amount (aperture value and shutter speed) for preliminary shooting based on the level of the output signal of the photometric sensor 27. As shown in FIG. 1, the photographed subject has a dark portion and a bright portion. The output signal of the photometric sensor 27 represents such average brightness of the subject. The operations of the aperture 12 and the shutter 13 are controlled by the aperture value and the shutter speed for the preliminary shooting determined based on the average brightness of the subject. The video signal output from the CCD 14 by this preliminary photographing is amplified by the preamplifier 15 and further A /
It is converted into digital image data by the D converter 19 and given to the exposure control circuit 10.

The exposure control circuit 10 determines the exposure amount for two main photographings based on the digital image data representing the subject image obtained by the preliminary photographing. The subject has a dark portion and a bright portion as described above. One main shooting is performed with an appropriate exposure amount for a dark portion, and the other main shooting is performed with an appropriate exposure amount for a bright portion.

In FIG. 6, the area indicated by the chain line PH indicates the photometric range of the photometric sensor 27.

The exposure control circuit 10 adjusts the diaphragm 12 based on the output signal of the photometric sensor 27 so that the relatively dark region SL of the subject image is properly exposed. This adjustment can be performed, for example, by virtually setting a large number of sample points PS within the photometric range PH of the photometric sensor 27 as shown in FIG. 6 and based on the data at each point. .. Aperture in this case
The method of adjusting twelve is described in detail in the specification of the embedded composite patent application mentioned above.

If the photometric area of the photometric sensor is divided into a plurality of small areas and the divided photometry for obtaining the average luminance of the small areas is performed for each small area, the above-described preliminary photographing is not always necessary. That is, the photometric average brightness for each small area is compared with an appropriate threshold value, and according to the comparison result, the photometric average brightness for the relatively dark part and the photometric average brightness for the bright part of the subject image are classified. .. The exposure amount of the relatively dark part is based on the average value of the photometric average brightness for the relatively dark part, and the exposure amount of the relatively bright part is based on the average value of the photometric average brightness for the relatively bright part. It is determined. Generally, the relatively dark and relatively bright parts of the subject image do not always correspond to the small photometric areas of the photometric sensor, but there is a significant difference in brightness between the dark and bright parts. Therefore, by using an appropriate threshold value, the photometric brightness data of the photometric small area corresponding to the relatively dark area and the photometric brightness data of the photometric small area corresponding to the relatively bright area can be obtained. It is possible to divide.

As methods for determining the aperture value and shutter speed, there are various methods such as aperture priority and shutter speed priority, but the aperture value and shutter speed may be determined in any way. At least the capability and accuracy of the diaphragm 12 and the capability and accuracy of the shutter 13 may be taken into consideration. Also, C
When the electronic shutter function of the CD 14 is used, the allowable exposure time (for example, within 1V = 1/60; V is a vertical scanning period) may be taken into consideration.

In any case, a proper exposure amount (first exposure amount) for the relatively dark portion SL of the subject image and a proper exposure amount (second exposure amount) for the relatively bright portion SH of the image. Amount; which is a value smaller than the first exposure amount) is determined, the aperture 12 and the shutter 13 are controlled using the first exposure amount, and the first actual photographing is performed. The video signal output from the CCD 14 in the first actual shooting is an amplifier
After being amplified in 15, pre-processing such as γ correction is performed in the pre-processing circuit 16, further converted into digital image data in the A / D converter 17, and once converted into the first frame memory 21 via the changeover switch 18. Remembered. Subsequently, the diaphragm 12 and the shutter 13 are controlled using the second exposure amount, and the second actual photographing is performed. The video signal output from the CCD 14 in the second main shooting is also amplified, preprocessed,
A / D converted into digital image data, which is stored in the second frame memory 22 via the changeover switch 18. The changeover switch 18 is controlled by the exposure control circuit 10. It goes without saying that the main photographing based on the second exposure amount may be performed before the main photographing based on the first exposure amount.

The shutter speed control in the main photographing is CC
It may be realized by an electronic shutter function using D14. As is well known, electronic shutters are CCDs.
Exposure is started by discharging (clearing) 14 unnecessary charges,
The exposure is completed by reading out the signal charges accumulated in the CCD 14. In this case, the maximum value of the exposure time (the time from the discharge of unnecessary charges to the reading of the signal charges) is 1
If the aperture value is set to be within / 60 = 1V, it is 2/60
= 2V, the two main shootings are completed. When the electronic shutter function is used, image data of two frames having different exposure amounts can be obtained in a shorter time than when the exposure time is controlled by using the mechanical shutter 13.

A memory for storing a key signal in addition to the two frame memories 21 and 22 in order to perform the above-mentioned fitting and combining process using the image data of two frames thus obtained. 23, a CPU 20, and a multiplexer 24 are provided.

The CPU 20 sets the image data for each pixel stored in the first frame memory 21 to a predetermined threshold value TH (a threshold value for discriminating the level of the image data obtained by the above-described preliminary photographing). Image data having a value equal to or greater than the threshold value, compared with the same value or different value, in any case, a relatively bright or saturated area can be distinguished from other areas) , Which belongs to a relatively bright area SH, data 1 (1 bit) is written as a key signal at the same pixel position as the pixel represented by the image data. For image data less than the threshold value, 0 is written in the position of the pixel in the memory 23. The memory 23 has a capacity capable of storing a key signal for one frame (or one field) (the key signal is 1 bit per pixel).

In this way, the key signal memory 23 is
Whether the image data stored in the frame memories 21 and 22 relates to the pixels belonging to the relatively dark area SL of the subject image (key signal data = 0), or the relatively bright area S
Is it related to pixels belonging to H (key signal data = 1)
The key signal data that represents is written.

The multiplexer 24 is controlled by the key signal data set in the key signal memory 23. When the key signal data is 0, the multiplexer 24 selects the image data read from the frame memory 21, and when the key signal data is 1, the image data read from the frame memory 22 is selected and output.

When the setting of the key signal data in the key signal memory 23 is completed, the image data in the frame memories 21 and 22 and the key signal data in the key signal memory 23 are synchronized (that is, the data related to the same pixel are simultaneously ) It is read and given to the multiplexer 24. As described above, the multiplexer 24 is a frame
Either one of the image data read from the memory 21 or 22 is selectively output. The image data output from the multiplexer 24 is a fitting composite image in which image data of a saturated (relatively bright) area in an image with a large amount of exposure is replaced by image data of a corresponding area in an image with a small amount of exposure. Is represented.

The image data output from the multiplexer 24 is converted into an analog video signal by the D / A converter 25 and output. If this analog video signal is given to a display device such as a CRT, an image that has been embedded and combined is displayed. The video signal may be FM-modulated and recorded on a magnetic recording medium such as a floppy disk or a magnetic tape. Alternatively, the output image data of the multiplexer 24 is separated (Y / C separated) into the luminance data and the color data by the image data processing circuit 26, the data is compressed and encoded, and then the memory card (including the semiconductor memory is doing,
It may be recorded in a memory cartridge or the like). However, it is not necessary to perform the above-described image data embedding / combining process with a still video camera. In this case, the image data of the frame memories 21 and 22 are separately image-processed and then stored in separate areas in the memory card. The image data inset synthesis will be performed by a separately provided image processing device.

The key signal may be created using image data with a small exposure amount stored in the frame memory 22. In this case, a threshold value that is small enough to eliminate relatively dark areas is used. If the image data is greater than or equal to this threshold value, 1 is set as the key signal data, and otherwise 0 is set in the memory 23. By doing so, the image data of the relatively dark area in the image with a small amount of exposure is replaced by the image data of the corresponding area in the image with a large amount of exposure, so that the image data representing the combined image is multiplexed. It is output from 24.

The still video camera shown in FIG. 5 has a strobe device 28. By utilizing the stroboscopic light emission in the strobe device 28, image data suitable for inset composition can be obtained even if the object to be photographed is not necessarily distinguished into a bright portion and a dark portion.

For example, assume a case where two objects having the same degree or a small difference in brightness are located at positions relatively close to and far from each other. If the strobe light emission amount is adjusted so that the object at the near position is properly exposed, the object at the distant position is underexposed. On the contrary, if the stroboscopic light emission amount is adjusted so that the object at the far position is properly exposed, the obtained video signal will be saturated for the object at the near position.

According to this still video camera, the first photographing is performed at the strobe light emission amount that can obtain the exposure suitable for photographing the object at the relatively far position, and the still video camera is at the relatively close position. The second shooting is performed with a strobe light emission amount such that the object is properly exposed. In this way, image data for two frames is obtained, and the image data is embedded and synthesized in exactly the same manner as described above.

Needless to say, when the difference in brightness between the objects at the far position and the close position is large, the strobe light emission amount is adjusted in consideration of this difference in brightness. In any case,
The stroboscopic light emission amount may be controlled so that the average luminance difference between the obtained image data of two frames is relatively small (within 2 to 3 times).

A relatively bright area SH in the subject image
There are various methods for identifying the relatively dark area SL with the above-described processing of simply comparing the image data with the threshold value TH. Other identification methods will be described below. Only a method of extracting a relatively bright area in an image with a large amount of exposure will be described. This is because it is possible to extract a relatively dark area in an image with a small exposure amount by using exactly the same method.

For simplicity, VII-VI in FIG.
Consider the video signal along the horizontal scan line represented by the I line.

FIG. 7A shows an example of the video signal along the horizontal scanning line. The video signal in the relatively bright area SH is saturated. In the relatively dark region SL, there is a small bright spot (for example, a piece of glass, a part of metal, etc. are shining due to light reflection), and due to this bright spot, a steep pulse-like waveform BR is generated in the video signal. It is supposed to appear.

When such a video signal passes through a low pass filter (hereinafter abbreviated as LPF), the symbol br in FIG.
As shown in, the steep pulse-like waveform becomes gentle and its height becomes low. If this video signal is level-discriminated using the threshold value TH set to a level higher than the peak value of the waveform br, only the bright area SH is extracted. In this way, the small bright spots that partially exist in the dark area SL are ignored, and the inset synthesis is prevented from being executed for such a small area.

As described above, the relatively bright area does not necessarily have to be saturated. It can be easily understood from Fig. 7 (B) that the bright region which is not saturated can be extracted by appropriately setting the threshold level TH (the peak value of the bright portion SH is higher than the saturation level). Should be considered low).

Filtering techniques for digital data are well known. The above discussion also applies to the processing in the still video camera shown in FIG. 5 in the CPU 20 which processes the digital image data stored in the frame memory 21 (or 22) to determine the areas SH and SL. The CPU 20 low-pass filters the digital image data and compares the filtered image data with the data representing the threshold value.

FIG. 8 shows another discrimination method. Figure 8
In (A), the same video signal as that shown in FIG. 7 (A) is shown. In this video signal, a front edge (leading edge) showing a sharp rise and a rear edge (trailing edge) showing a sharp fall are detected. The width (or time) t1, t2, etc. from the front edge to the rear edge is measured. The widths t1, t2, etc. are compared with an appropriate reference width W (see FIG. 8B). Only a portion having a width larger than the reference width W is determined to be a relatively bright area (see FIG. 8C). Also by this method, small bright spots existing in a relatively dark region can be excluded from the target region of the fitting synthesis.

It goes without saying that this method can also be implemented in analog or digital manner. In the case of analog execution, a monostable multivibrator having a stable time corresponding to the reference width W can be used. This monostable multivibrator is triggered (set) by the leading edge of the video signal and reset by the trailing edge. If an output is generated from the monostable multivibrator after being set and before being reset (when the time corresponding to the width W has elapsed after being set), that portion is determined to be a bright region. Digitally,
It may be determined whether or not the length from the front edge to the rear edge is larger than the width W.

In the method shown in FIG. 7, the boundary of the bright area may be slightly shifted due to the phase delay inevitably generated in the filtering. On the other hand, the method shown in FIG. 8 has the feature that the boundary between the bright area and the dark area can be accurately determined without shifting.

The detection of the boundary of the area (the boundary extending in the vertical scanning direction of the image) appearing in the video signal on the horizontal scanning line has been described, but the same method as described above is used for the detection of the boundary extending in the horizontal direction of the image. can do. Particularly when digitally processing digital image data, filtering in the vertical direction and width detection are easy. Further, by performing the method shown in FIG. 8 two-dimensionally, it becomes possible to exclude bright spots having a predetermined area or less from the subject of the inset synthesis and to perform the inset synthesis only for a bright area exceeding the certain area.

FIG. 9 shows another example of the fitting synthesis process.
In particular, it shows a circuit configuration that realizes a method of smoothly continuing the vicinity of the boundary between two fitted regions. The circuit shown in FIG. 9 replaces the multiplexer 24 in FIG.

The image data (for example, 8 bits) read out from the frame memory 21 is multiplied by the multipliers 30a, 31a, 32a,
The signals are multiplied by 0, 1, 2, 3, and 4 by 33a and 34a and given to a multiplexer (changeover switch) 37a. The image data read from the frame memory 22 is multiplied by 4, 3, 2, 1 and 0 by the multipliers 30b, 31b, 32b, 33b and 34b, respectively, and is multiplexed.
Given to 37b. The multiplexers 37a and 37b are controlled by key signal data (3 bit data in this embodiment) provided from the key signal memory 23. The multiplexer 37a is a multiplier 30a, 31a, 32a, 33a or 34a.
When is selected, the multiplexer 37b is the multiplier 30b,
Select 31b, 32b, 33b or 34b.

The outputs of the multiplexers 37a and 37b are mutually added by the adder 35, further divided by 4 by the divider 36, and output as combined image data (again, for example, 8 bits).

In one pixel on the detected boundary line between the relatively dark area SL and the bright area SH, the multiplexers 37a and 37b are respectively provided with the multipliers 32a and 32b.
Select. As a result, the arithmetic mean of the image data in the frame memory 21 and the image data in the frame memory 22 becomes the composite image data on the boundary line.

For one pixel adjacent to the boundary line on the side of the area SL darker than the boundary line, the multiplexers 37a and 37b select the multipliers 33a and 33b, respectively. As a result, in this pixel, the average (weighted average) of the value tripled to the image data in the frame memory 21 and the value tripled to the image data in the frame memory 22 becomes the composite image data.

For all pixels inside the area SL which is darker than the above-mentioned pixels adjacent to the boundary line, the multiplexers 37a and 37b have the multipliers 34a and 34 respectively.
Select b. As a result, the image data in the frame memory 21 is output as composite image data.

In one pixel adjacent to the boundary line on the side of the area SH brighter than the boundary line, the multiplexer 37a and
37b selects the multipliers 31a and 31b, respectively. As a result, in this pixel, an average (weighted average) of a value that is 1 times the image data of the frame memory 21 and a value that is 3 times the image data of the frame memory 22 becomes the composite image data.

For all the pixels inside the area SH which is brighter than the above-mentioned pixels adjacent to the boundary line, the multiplexers 37a and 37b are the multipliers 30a and 37a, respectively.
Select 30b. As a result, the image data in the frame memory 22 is output as composite image data.

The key signal data stored in the key signal memory 23 has a pixel position on the boundary line, adjacent to the boundary line, or separated from the adjacent boundary line so as to control the multiplexers 37a and 37b as described above. It is created as 3-bit data by the CPU 20 according to which area it belongs to.

As described above, in the vicinity of the boundary between the regions SL and SH, the composite image data is created by the weighted average of the image data to be fitted (weighted according to the position). The image data will be smoothly continuous near the boundary. As a result, when the composite image is reproduced, the boundary between the two regions feels natural, and pseudo contours are prevented from occurring.

In the above description, the weighting for the weighted average is changed pixel by pixel, but it goes without saying that the weighting may be changed for a plurality of pixels.

FIG. 10 shows an embodiment in which an image inset synthesis process is performed on an analog video signal in real time. The circuit of this embodiment can be applied not only to a still video camera but also to a movie video camera.

The image pickup optical system includes an image pickup lens 41, a diaphragm 42, a beam splitter 43, and two CCDs 44 and 45. An optical image representing a subject passes through a lens 41 and a diaphragm 42, is split by a beam splitter 43, and has two CCDs 44.
And on 45. As described above, in photographing in a room with windows, the ratio of the average brightness of the relatively dark area SL to the average brightness of the relatively bright area SH is 1: 5.
It is about 10 ~. In this still video camera, the light splitting ratio of the beam splitter 43 is set to 5: 1. Light having an amount of light that is 5/6 of the amount of incident light passes through the beam splitter 43 and enters the CCD 44. 1 of the amount of incident light
/ 6 light quantity is reflected by the beam splitter 43 and CC
It is incident on D45.

The video signal output from the CCD 44 is amplified by the preamplifier 46 and then applied to the delay circuit 54 and input to the exposure control circuit 49. The video signal output from the CCD 45 is amplified by a preamplifier 47 and then applied to an automatic gain control amplifier circuit (hereinafter referred to as AGC) 48 and input to an exposure control circuit 49. The exposure control circuit 49 controls the diaphragm 42 via the driver 50 and adjusts the gain of the AGC 48.

The output image signal of the CCD 44 having a large exposure amount is used for controlling the exposure amount. The exposure control circuit 49 is an amplifier 46.
Based on the level of the video signal given from the aperture, the aperture is adjusted so that the relatively dark area SL of the subject image is properly exposed.
Adjust 42. The shutter speed (exposure time) is fixed and is maintained at, for example, 1/60 second (or 1/30 second). That is, no mechanical shutter is provided, and the exposure time is defined by clearing the unnecessary charges of the CCDs 44 and 45 and reading the signal charges.

In this still video camera, a subject image is continuously photographed, and one field (or one frame) from the CCDs 44 and 45 is taken every 1/60 second (or 1/30 second), for example. Video signal is being output.

The exposure amount is adjusted so that the relatively dark region SL of the subject image formed on the CCD 44 is properly exposed, and the division ratio of the beam splitter 43 is set to 5: 1. Therefore, it can be expected that the exposure amount is almost proper even in the relatively bright area SH imaged on the CCD 45. When the image of the relatively dark area SL and the image of the bright area SH are combined, these images are appropriately matched (for example, the image of the relatively bright area SH is relatively AGC 48 is provided in order to prevent the occurrence of a situation where the image is darker than the image in the dark area SL. The exposure control circuit 49 detects the peak level of the video signal of the previous field (or the previous frame) supplied from the amplifier 47, and this peak level is held constant also in the video signal of the next field (or the frame). Adjust the gain of AGC48. In this way, every 1 field (or 1 frame) (every 1/60 second,
Alternatively, the gain of the AGC 48 is adjusted every 1/30 second, and the brightness of the brightest part of the image in the relatively bright area SH is always kept substantially constant.

The output of the amplifier 46, which is a video signal having a large amount of exposure, is also passed through the LPF 51, and only the low frequency component thereof is given to the comparator 52. The threshold voltage V is applied to the comparator 52.
_TH is set. The comparator 52 produces an output when the level of the input video signal exceeds the threshold voltage V _TH . The output of the comparator 52 is input to the pulse width detection circuit 53. The pulse width detection circuit 53 includes the monostable multivibrator or the like as described above, and when the pulse width of the output signal of the comparator 52 exceeds the reference width W, the output signal corresponds to the reference width W for a period of time. Output after delaying. The output signal of the pulse width detection circuit 53 is given to the multiplexer 56 as its control signal.

The delay circuits 54 and 55 are set with a delay time equal to the time corresponding to the reference width W (or the time obtained by adding the delay time due to the operation of the LPF 51 to this time). The output video signal of the amplifier 46 and the output video signal of the AGC 48 are delayed by the delay time by the delay circuits 54 and 55 and input to the multiplexer 56.

The multiplexer 56 normally selects and outputs the output video signal of the delay circuit 54, and when the output signal is given from the pulse width detection circuit 53, selects and outputs the output video signal of the delay circuit 55. As a result, image inset synthesis based on the above-described principle is performed. The video signal output from the multiplexer 56 is subjected to γ correction and the like in the video signal processing circuit 57.

There are various other modifications relating to the inset synthesis process, which are described in detail in the specification of the inset synthesis patent application mentioned above.

FIG. 11 shows an embodiment of the solid-state electronic image pickup device of the present invention and is a schematic view of a CCD. Even if the exposure is not performed twice, two types of image signals having different exposure amounts are obtained from the CCD shown in FIG. 11 and are used for the above-mentioned inset synthesis processing.

A large number of photodiodes 60A and 60B are arranged in the vertical and horizontal directions. A filter for limiting the amount of incident light is provided on the photodiode 60A, and this filter is shown by hatching. Filters are vertical photodiodes in every other row
It is installed at 60A. The transmittance of the filter is preferably 5: 1 with respect to the incident light on the photodiode 60B not provided with the filter.

A vertical transfer path 62 is arranged in the horizontal direction of the photodiodes 60A and 60B via a P well 61.

The vertical transfer path 12 is formed immediately below the electrodes V1 to V6 which are periodically arranged. An electrode V1 is formed adjacent to the photodiode 60A in the first row from above, and an electrode V2 is formed in a direction perpendicular to the electrode V1. An electrode V3 is formed adjacent to the photodiode 60B in the second row and is connected to the electrode V2 in the vertical direction. Further, the electrode V3 is connected to the electrode V4 in the vertical direction.

An electrode V5 is formed adjacent to the photodiode 60A in the third row and is connected to the electrode V4 in the vertical direction. Furthermore, the electrode V5 is connected to the electrode V2 in the vertical direction.

An electrode V6 is formed adjacent to the photodiode 60B in the fourth row and connected to the electrode V2. Further, the electrode V6 is connected to the electrode V4 in the vertical direction. The electrode V4 is connected to the electrode V1 adjacent to the photodiode 10A in the fifth row in the vertical direction.

Electrodes V1, V2, V3 in the vertical direction
It is formed by periodically repeating V4, V5, V2 and V6. Vertical transfer pulses φV1 to φV6 are applied to these electrodes V1 to V6, respectively.

The first electrode V4 is provided at the lowermost electrode V4 of the vertical transfer path 12.
The horizontal transfer path 63 of is connected. First horizontal transfer path 63
An amplifier 64 is connected to the output of the first amplifier via the amplifier 64 according to the applied horizontal transfer pulses φH1 to φH4.
The signal charge of is output.

In the CCD shown in FIG. 11, the first horizontal transfer path 63 and the second horizontal transfer path 65 are connected via a shift gate 67 whose gate is opened by a shift gate pulse. The amplifier 66 is also connected to the output of the second horizontal transfer path 65, and the second signal charge is output via the amplifier 64 according to the horizontal transfer pulses φH1 to φH4.

12 and 13 are time charts showing the state of transfer of the signal charges accumulated in the photodiodes 60A and 60B of the first field, and FIG. 13 shows in detail the portion indicated by reference sign c in FIG. ing. FIG. 14 is a time chart showing how signal charges stored in the second field photodiodes 60A and 60B are transferred. Figure
Reference numeral 15 is a schematic diagram showing how the signal charges are transferred in the vertical transfer path 12, and the signal charges are indicated by hatching.

The output of the first signal charge accumulated in the photodiodes 60A and 60B will be described with reference to FIGS. 12, 13 and 15. Photodiodes 60A and 60B
The accumulation of electric charges in the photodiodes starts from the sweeping out of unnecessary electric charges accumulated in the respective photodiodes 60A and 60B.

The vertical transfer pulses φV1 to φV6 are applied to the electrode V1.
~ V6.

At time t ₁ , the electrodes V1, V3, V4 and
A voltage of V [V] is applied to V5 and V6. Next, at time t ₂ , a voltage of V [V] is applied to the electrodes V4 and V6, and a voltage of 2V [V] is applied to the electrode V3.
As a result, the second signal charge accumulated in the photodiode 60B of the first field is transferred from the photodiode 60B to the potential well generated below the electrodes V3 and V4.

At time t ₃ , the voltage V [V] is applied to the electrode V 5 as well, whereby the signal charges are transferred through the vertical transfer path 12.

At time t ₄ , 2V [V] is applied to the electrode V1.
Voltage is applied. As a result, the first signal charge accumulated in the photodiode 60A of the first field is transferred from the photodiode 60A to the potential well generated below the electrodes V4 and V1.

Thereafter, at time t ₅ , the electrodes V1, V3 and
V4, V5 and V6 voltage V [V] is applied to the voltage of V [V] to the electrodes V1, V4 and V5 are applied at time t _6, the electrodes V1 at time t _7, V2, V
A voltage of V [V] is applied to 4 and V5.

Further, at time t ₈ , a voltage of V [V] is applied to the electrodes V1, V2 and V5, and at time t ₇ , a voltage of V [V] is applied to the electrodes V1, V2, V3, V5 and V6. .. Further, at time t ₁₀ , the electrode V
2, V3 and V6 are given a voltage of V [V], and at time t ₁₁ , electrodes V2, V3, V4 and V6 are V
A voltage of [V] is applied. As the combination of electrodes to which a voltage is applied changes in this way, the potential well generated under the electrodes also changes and moves, so that the signal charge accumulated in the potential well also transfers as the voltage well moves.

As the signal charges are transferred through the vertical transfer path 62, they are given to the horizontal transfer path 63. Photodiode 60A
The first signal charge stored in the first horizontal transfer path 63
Is applied to the first horizontal transfer path 63 and is transferred to the amplifier 64 in accordance with the horizontal transfer pulses φH1 to φH4. In the amplifier 64, the first signal charge accumulated in the photodiode 60A is amplified and output.

When the second signal charge accumulated in the photodiode 60B is applied from the vertical transfer path 62 to the first horizontal transfer path 63, a shift gate pulse is applied to the shift gate 67. By applying the shift gate pulse to the shift gate 67, the second signal charge input to the first horizontal transfer path 63 is further transferred to the second horizontal transfer path 65. The second signal charge input to the second horizontal transfer path 65 is horizontally transferred in response to the horizontal transfer pulses φH1 to φH4. The second signal charge output from the second horizontal transfer path 65 is given to the amplifier 66, amplified, and output.

After the signal charges accumulated in the photodiodes 60A and 60B of the first field are read out, the second
The signal charges accumulated in the photodiodes 60A and 60B of the field are read out. In reading the signal charges accumulated in the photodiodes 60A and 60B in the second field, the vertical transfer pulses φV1 to φV6 are applied to the vertical transfer path 62 and the horizontal transfer paths 13 and 15 are transferred in accordance with the time chart shown in FIG. This is done by applying a pulse. Since the reading of the signal charges in the second field is almost the same as the reading of the signal charges in the first field, the description thereof will be omitted.

Even if the second horizontal transfer path is not exposed twice, the first horizontal transfer path 63 included in the single-plate CCD outputs the first signal charge generated by the light whose incident light amount is limited by the filter, and the second horizontal transfer path. A second signal charge generated by the light whose incident light amount is not limited is output from 15.

FIG. 16 shows a signal processing circuit for an analog video signal represented by signal charges obtained from the two types of horizontal transfer paths as described above. 17A and 17B show the arrangement of RGB color filters arranged in the CCD, and FIG. 17A shows stripe filters in which the arrangement of R, G, and B is different for each column, and ) Indicates mosaic filters respectively. 18 (A) to 18 (C) are graphs showing output signals.

On the CCD 80, RGB color filters having different densities periodically are arranged in an array as shown in FIG. 17 (A) or (B). As a result, the CCD 80 outputs color image signals having different exposure amounts.

The first signal charge representing the color image signal output from the first horizontal transfer path 63 is amplified by the amplifier 64. The output of the amplifier 64 is the output signal a. The second signal charge representing the color image signal output from the second horizontal transfer path 65 is amplified by the amplifier 66. The output of the amplifier 66 is the output signal b.

The output signal a is applied to one input terminal of the comparator 71 and the a ₁ terminal of the changeover switch. The output signal b is provided to the amplifier 72. As shown in FIG. 17A, the amplifier 72 amplifies the output signal b by n times when the incident light amount at which the output signal a is saturated is 1 and the incident light amount at which the output signal b is saturated is n. Accordingly, when the output signal output from the amplifier 72 is c, the output signal a and the output signal c
The graph represented by has the same slope, and thus represents one image by fitting synthesis. The output signal c of the amplifier 72 is applied to the c ₁ terminal of the changeover switch 74.

The other terminal of the comparator 91 has a predetermined voltage E
Is connected to the variable resistor 73 to which is added, and the threshold value is specified. If the signal input to one terminal of the comparator 71 does not exceed the threshold value, the changeover switch 74 has the a ₁ terminal turned on. If it exceeds the threshold value, the changeover switch 74 has the c ₁ terminal turned on. It is controlled by the output signal from the comparator 71 so as to be in the state. As a result, the output of the changeover switch 74 is shown in FIG.
As shown in (C), the output signal a is below the threshold value.
_The output signal is c _{1 when the} threshold value is exceeded. As a result, the two image signals are combined and the fitting combining process is performed. Changeover switch 74
Is output to the color separation circuit 75.

The color separation circuit 75 generates and separates the luminance signal Y and the color signals R, G and B from the input color image signal and outputs the luminance signal Y. The luminance signal Y is sent to the knee circuit 76 by the color signals R, G and B. B is given to each gamma correction circuit.

Kn of the luminance signal Y in the knee circuit 76
The ee process is performed and the result is given to the gamma correction circuit 77.
The luminance signal input in the gamma correction circuit 77 is gamma-corrected and output.

Color signals R and G input to the gamma correction circuit 78
Gamma correction is also performed on B and B, and they are applied to the matrix circuit 79. The matrix circuit 79 inputs the color signals R and G
And B to generate color difference signals RY and BY, and the generated color difference signals RY and BY are output.

The luminance signal Y output from the gamma correction circuit 77 and the color difference signal R output from the matrix circuit 79.
An inset composite image is obtained from -Y and BY.

FIG. 19 shows an example of a circuit for moving the CCD exposure position in the vertical direction. FIG. 20 (A) shows the CCD exposure position of the first field, and FIG. 20 (B) shows the second field. The respective CCD exposure positions are shown.

Referring to FIG. 19, a piezoelectric element 84 is attached to a supporting member 83.
Is fixed. The piezoelectric element 84 has a switching element 81
The DC voltage from the DC voltage source 82 is applied according to the on / off operation of the. A CCD 86 protected by a CCD package 85 is fixed to the piezoelectric element 84. As a result, the position of the CCD 86 moves depending on whether or not the voltage is applied to the piezoelectric element 84.

As shown in FIGS. 20A and 20B, a filter 87 is provided on the photodiodes 60 in odd columns. Referring to FIG. 20 (A), in the first field, no voltage is applied to the piezoelectric element 84, and exposure is performed at a predetermined exposure position. When the exposure in the first field is completed and the reading of the signal charges accumulated in the photodiode 60 is completed, the exposure in the second field is carried out. At the time of exposure in the second field, a DC voltage is applied to the piezoelectric element 84. When a DC voltage is applied to the piezoelectric element 84, stress is generated in the piezoelectric element 84. As a result, as shown in FIG. 20 (B), the CCD
The position of 86 is moved and the exposure in the second field is performed. When the exposure in the second field is completed, the signal charges accumulated in the photodiode 60 are read out.

Since it is considered that the number of pixels in the vertical direction is doubled by moving the CCD shown in FIG. 11 in the vertical direction by using the piezoelectric element as described above, the vertical resolution of the analog video signal obtained from the CCD 86 is increased. Becomes higher.

[Brief description of drawings]

FIG. 1 shows a scene within the field of view of a camera.

FIG. 2 is a graph showing a histogram of photoelectric conversion characteristics and brightness of an image sensor.

FIG. 3A shows an image taken with a relatively large exposure amount, and FIG. 3B shows an image taken with a small exposure amount.

FIG. 4 is a graph showing a luminance histogram of a combined image.

FIG. 5 is a block diagram showing an embodiment applied to a still video camera.

FIG. 6 shows sample points of image data.

FIG. 7A is a waveform diagram showing an example of a video signal, and FIG.
It is a wave form diagram which shows the video signal after passing through PF.

8A to 8C are waveform charts showing a process of detecting a bright region having a certain width or more.

FIG. 9 is a block diagram showing an example of a circuit that calculates a weighted average of image data at a boundary between a bright area and a dark area.

FIG. 10 is a block diagram showing an embodiment for performing inset synthesis of video signals in real time.

FIG. 11 is a schematic view of a CCD showing an embodiment of the present invention.

FIG. 12 is a time chart of signal charge transfer.

13 is a partially enlarged view of the time chart shown in FIG.

FIG. 14 is a time chart of signal charge transfer.

FIG. 15 shows how signal charges are transferred.

16 is a signal processing circuit of an image signal obtained by the CCD shown in FIG.

FIG. 17 (A) is an example of an array of stripe filters,
(B) shows examples of mosaic filter arrays.

18A to 18C show changes in signals of the circuit shown in FIG.

FIG. 19 shows a moving circuit of a CCD.

20 (A) and 20 (B) show movement of CCD position.

[Explanation of symbols]

 60, 60A, 60B Photodiode 62 Vertical transfer path 63, 67 Horizontal transfer path

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Kazuyuki Ippane 1-6 Matsuzakadaira, Yamato-cho, Kurokawa-gun, Miyagi Prefecture Fujifilm Microdevices Co., Ltd.

Claims (4)

[Claims] 1. A large number of photoelectric conversion elements arranged vertically and horizontally and accumulating an amount of signal charges according to the amount of incident light, and provided on the photoelectric conversion elements in the horizontal direction at predetermined vertical intervals. A filter for limiting the amount of light entering the photoelectric conversion element, a filter provided adjacent to each column in the vertical direction of the photoelectric conversion element and accumulated in the photoelectric conversion element provided with the filter according to a transfer signal. A vertical transfer section for separately transferring the signal charge of 1 and the second signal charge accumulated in the photoelectric conversion element not provided with the filter, and the first signal charge input from the vertical transfer section is horizontally A solid-state electronic image pickup device comprising: a first horizontal transfer unit that transfers in the horizontal direction; and a second horizontal transfer unit that horizontally transfers the second signal charge input from the vertical transfer unit. 2. The solid-state electronic imaging device according to claim 1, further comprising moving means for moving all the photoelectric conversion elements or the filters in the vertical direction by one row of the photoelectric conversion elements. 3. A large number of photoelectric conversion elements arranged in the vertical direction and the horizontal direction and accumulating an amount of signal charges according to the amount of incident light. A first filter for limiting the amount of light incident on the photoelectric conversion element by a first transmittance, the photoelectric conversion element provided on the photoelectric conversion element not provided with the first filter, and the photoelectric conversion element having a second transmittance. A second filter for limiting the amount of light incident on the element, provided adjacent to each column in the vertical direction of the photoelectric conversion element, and accumulated in the photoelectric conversion element provided with the first filter according to a transfer signal. A vertical transfer unit for separately transferring the first signal charge and the second signal charge accumulated in the photoelectric conversion element provided with the second filter, and the first transfer unit for inputting from the vertical transfer unit. Solid-state electronic imaging device including: a first horizontal transfer unit that horizontally transfers the signal charges of the second horizontal transfer unit; and a second horizontal transfer unit that horizontally transfers the second signal charges input from the vertical transfer unit. . 4. The solid body according to claim 3, further comprising moving means for moving all the photoelectric conversion elements or the first filter and the second filter in the vertical direction by one row of the photoelectric conversion elements. Electronic imaging device.