JP3071891B2 - Solid-state electronic imaging device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a solid-state electronic imaging device such as a CCD.

[0002]

2. Description of the Related Art A solid-state electronic imaging device such as a CCD is used in an imaging optical system such as a movie video camera and an electronic still camera. Since a solid-state electronic imaging device has features such as light weight and small size, a camera using the solid-state electronic imaging device can be reduced in weight and size.

[0003]

However, since the dynamic range of a solid-state electronic imaging device is relatively narrow,
When the brightness difference between a bright part and a dark part included in the field of view is large, it is difficult to properly expose and photograph both parts.

[0004] For example, there are cases where the background is very bright and the central main subject is dark, when the subject is backlit, or when outside scenery is reflected through a window in shooting indoors with a window. In the shooting of such a scene, if the exposure is adjusted so that the main subject (often a person) is properly exposed, the amount of incident light exceeds the dynamic range of the solid-state electronic imaging device for a bright background portion. Is saturated, so that a bright background is not photographed, and the image in that part simply becomes white (overexposure).

[0005] Therefore, in general, a device is devised to increase the brightness of the main subject by emitting a strobe light (daytime synchro flash photography). However, a flash device is required for daytime synchro flash photography, which increases the size of the camera and its operation is cumbersome. Also, when the dark part is far away, or when it extends from a short distance to a long distance, daytime synchro strobe photography is not necessarily effective.

Accordingly, the inventor has developed a technique for obtaining an appropriate image signal for a subject having a large difference in luminance between a bright area and a dark area without necessarily emitting a strobe light. (Title of Invention "Video camera, shooting method using the same, operating method thereof, and image processing apparatus and method"; reference number 91
167; Applicant: Fuji Photo Film Co., Ltd. and Fuji Film Micro Device Co., Ltd.) (hereinafter referred to as “inlaid synthesis patent application”).

The basic idea of the invention of the inlaid synthesis patent application is to obtain an image signal representing two images having different exposure amounts by imaging a subject with two different exposure amounts, and to represent an image having a large exposure amount. In the image signal,
An image signal representing a region having a relatively high luminance in the image is replaced with an image signal representing a corresponding region in an image having a small amount of exposure to create a combined image signal.

SUMMARY OF THE INVENTION An object of the present invention is to provide a solid-state electronic imaging device capable of obtaining an appropriate video signal for a subject having a large difference in brightness between a bright area and a dark area without necessarily emitting strobe light. An object of the present invention is to obtain image signals representing two images having different exposure amounts used in the invention of the synthetic patent application.

In particular, it is an object of the present invention to be able to obtain two image signals having different exposure amounts on a single plate without having to perform exposure twice.

[0010]

A solid-state electronic imaging device according to a first aspect of the present invention has a large number of photoelectric conversion elements arranged in a vertical direction and a horizontal direction and accumulating an amount of signal charges corresponding to an amount of incident light. A filter that is provided on the photoelectric conversion element in the horizontal direction at a predetermined interval and that limits a light amount incident on the photoelectric conversion element; and a filter that is provided adjacent to each column in the vertical direction of the photoelectric conversion element and that responds to a transfer signal. A vertical transfer unit that distinguishes and transfers the first signal charge stored in the photoelectric conversion element provided with the filter and the second signal charge stored in the photoelectric conversion element not provided with the filter. A first horizontal transfer unit that transfers the first signal charge input from the vertical transfer unit in the horizontal direction, and a second horizontal transfer unit that transfers the second signal charge input from the vertical transfer unit in the horizontal direction Feeding portion, and is characterized in that it comprises a moving means for moving all or the filter of the photoelectric conversion elements in the vertical direction by the photoelectric conversion elements one column.

[0011]

A solid-state electronic image pickup device according to a second aspect of the invention is arranged in a vertical direction and a horizontal direction, and includes a large number of photoelectric conversion elements for accumulating an amount of signal charges according to the amount of incident light. A first filter that is provided on the photoelectric conversion element in a direction and restricts the amount of light incident on the photoelectric conversion element by a first transmittance, and is provided on the photoelectric conversion element that is not provided with the first filter. , Second
A second filter for limiting the amount of light incident on the photoelectric conversion element according to the transmittance of the photoelectric conversion element, provided adjacent to each column of the photoelectric conversion element in the vertical direction, and provided with the first filter according to a transfer signal. A vertical transfer unit that separates and transfers the first signal charge stored in the photoelectric conversion element and the second signal charge stored in the photoelectric conversion element provided with the second filter, respectively. A first horizontal transfer unit for transferring the first signal charges input from the first horizontal direction,
A second horizontal transfer unit that transfers the second signal charge input from the vertical transfer unit in the horizontal direction, and all of the photoelectric conversion elements or the first filter and the second filter in the vertical direction. A moving means for moving the photoelectric conversion element by one row is provided.

[0013]

[0014]

According to the first aspect, there are a photoelectric conversion element provided with a filter and a photoelectric conversion element not provided with a filter.

According to the first invention, the first signal charge stored in the photoelectric conversion element provided with the filter and the second signal charge stored in the photoelectric conversion element not provided with the filter are distinguished. The vertical transfer is transferred.

According to the first aspect, a first signal charge is obtained from the first horizontal transfer unit, and a second signal charge is obtained from the second horizontal transfer unit. According to the first aspect, all of the photoelectric conversion elements or the filter can be moved vertically by one row of the photoelectric conversion elements.

According to the second invention, there is a photoelectric conversion element provided with a filter having a first transmittance and a photoelectric conversion element provided with a filter having a second transmittance.

According to the second aspect, the first signal charge and the second signal charge stored in each photoelectric conversion element are transferred by the vertical transfer unit.

According to the second aspect, the first signal charge is obtained from the first horizontal transfer unit, and the second signal charge is obtained from the second horizontal transfer unit. According to the second aspect of the present invention, all of the photoelectric conversion elements or the first filter and the second filter can be moved vertically by one row of the photoelectric conversion elements.

[0020]

In each of the first and second aspects of the present invention, a first signal charge is obtained from the first horizontal transfer path and is represented by the first signal charge from the second horizontal transfer path. A second signal charge representing an exposure amount different from the exposure amount is obtained. Two image signals having different exposure amounts can be obtained from one solid-state electronic imaging device without performing exposure twice, and by combining these image signals, it is possible to cope with an image having a portion having a large luminance difference. Become. In the first invention, all of the photoelectric conversion elements or the filters can be moved vertically by one row of the photoelectric conversion elements, so that the lines of the photoelectric conversion elements provided with the filters are replaced with odd lines for each field. And even lines. Therefore, even if a filter is provided, interlacing is possible, and it is considered that the number of pixels in the vertical direction is doubled. Vertical resolution is higher. Also, a high-resolution image without flicker can be obtained even for a moving subject. Also in the second invention, the interlacing becomes possible as in the first invention, and it is considered that the number of pixels is doubled in the vertical direction.
Vertical resolution is higher.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Before describing a solid-state electronic imaging device according to the present invention, an image inset process utilizing two types of image signals obtained from the solid-state electronic imaging device according to the present invention will be described.

Referring to FIGS. 1 to 4, the basic concept of the image fitting process will be described.

FIG. 1 shows an example of a scene that fits within the field of view of a camera when shooting in a room with a window.

The image SL of the room RM (symbol SL is also used to represent a part or area of this image) is relatively dark.
The outside scenery can be seen through the window WN, and the image SH of this scenery (this symbol SH is also used to represent a part or area of this image) is relatively bright. Generally speaking, the average brightness of the image SH seen through the window WN is about 5 to 10 times the average brightness of the image SL of the room RM. The dynamic range of an image sensor such as an image pickup tube or a CCD is two to three times higher than the average luminance of a relatively bright part.
It can handle only about twice as many scenes. That is, if the luminance difference is two to three times or more, the sensor saturates for a bright part when a dark part is properly exposed (overexposure occurs during reproduction), and when a bright part is properly exposed, a dark part is darkened. The image hardly appears, and the ratio of the noise component in the video signal becomes extremely high.

FIG. 2 shows a histogram of the photoelectric conversion characteristics and luminance of the image sensor.

The solid line SL1 shows the photoelectric conversion characteristics of the image sensor when the relatively dark image SL is properly exposed, and the solid line SL2 shows the relatively dark image SL taken under the photoelectric conversion characteristics SL1. 9 shows a frequency distribution of luminance.

On the other hand, a broken line SH1 shows the photoelectric conversion characteristics of the image sensor when the relatively bright image SH is properly exposed, and a broken line SH2 shows the relatively imaged under the photoelectric conversion characteristics SH1. The frequency distribution of the brightness of the bright image SH is shown.

As is apparent from this figure, the relatively bright image SH is completely included in the saturation region of the photoelectric conversion characteristic SL1, and the output representing the relatively dark image SL under the photoelectric conversion characteristic SH1. The signal level is concentrated very low.

FIG. 3A shows an image photographed with an exposure amount appropriate for the brightness of the room RM (the exposure amount is relatively large). Since the image SLa of the room RM is properly exposed, a good image is obtained. On the other hand, the scenery image SHA that appears through the window WN is white.

FIG. 3B shows an image taken with an appropriate exposure amount (the exposure amount is relatively small) with respect to the image SHb in the window WN. The image SHb of the scenery seen through the window WN is good, but the image SLb of the room RM is considerably dark.

According to this processing, an image of a saturated area in an image with a relatively large exposure amount (the image need not necessarily be saturated, but may be an area relatively brighter than other parts). The image is fitted by being replaced with the image of the corresponding area in the image having a relatively small exposure amount. That is, the image SHa of the window WN in FIG. 3A is replaced with the image SHb of the window WN shown in FIG. Thereby, as shown in FIG. 1, both the room RM and the window WN become good images, and an image close to the scene seen by human eyes can be obtained. FIG. 4 shows a histogram of the image fitted and synthesized in this manner. The two luminance distributions SL2 and SH2 are fairly close.

An image of a relatively dark area in an image with a relatively small exposure may be replaced with an image of a corresponding area in an image with a relatively large exposure. For example, the image SLb of the room RM in FIG.
(A) is replaced by the image SLa of the room RM shown in FIG.

FIG. 5 shows a still video using this processing.
1 shows a configuration of a camera (electronic still camera).

The image pickup optical system includes an image pickup lens 11, a diaphragm 12, a shutter 13, and a solid-state electronic image pickup device (image sensor).
The CCD 14 is included. The exposure control circuit 10 includes a CPU. The circuit 10 includes an aperture 12 and a shutter.
13 and the CCD 14 to control the charge clearing, signal reading, and the like.

In this embodiment, preliminary photographing (preliminary exposure) and 2
The main shooting (main exposure) is performed twice.

The preliminary photographing is for determining an appropriate exposure amount in the two main photographings. The exposure control circuit 10 determines an exposure amount (aperture value and shutter speed) for preliminary photographing based on the level of the output signal of the photometric sensor 27. As shown in FIG. 1, the photographing subject has a dark part and a bright part. The output signal of the photometric sensor 27 indicates such average luminance of the subject. The operations of the aperture 12 and the shutter 13 are respectively controlled by the aperture value and the shutter speed for preliminary photographing determined based on the average luminance of the subject. The video signal output from the CCD 14 by this preliminary photographing is amplified by the preamplifier 15 and further
The data is converted into digital image data by the D converter 19 and supplied to the exposure control circuit 10.

The exposure control circuit 10 determines an exposure amount for two main shootings based on digital image data representing a subject image obtained by the preliminary shooting. The subject has a dark part and a bright part as described above. One main shooting is performed with an appropriate exposure for a dark part, and the other main shooting is performed with an appropriate exposure for a bright part.

In FIG. 6, a region indicated by a chain line PH indicates a light measuring range of the light measuring sensor 27.

The exposure control circuit 10 adjusts the aperture 12 based on the output signal of the photometric sensor 27 so that the relatively dark area SL of the subject image is properly exposed. This adjustment can be performed, for example, by virtually setting a large number of sample points PS in the photometric range PH of the photometric sensor 27 as shown in FIG. . Aperture in this case
The twelve adjustment methods are described in detail in the above-mentioned inlaid synthetic patent application specification.

If the photometric area of the photometric sensor is divided into a plurality of small areas and divided photometry for calculating the average luminance of each small area is performed, the above-described preliminary photographing is not necessarily required. That is, the photometric average luminance of each small area is compared with an appropriate threshold value, and the photometric average luminance of a relatively dark portion of the subject image and the photometric average luminance of a bright portion are classified according to the comparison result. . The exposure of the relatively dark part is based on the average of the photometric average luminance for the relatively dark part, and the exposure of the relatively bright part is based on the average of the photometric average luminance for the relatively bright part. It is determined. Generally, a relatively dark part and a relatively bright part of the subject image do not always correspond to the small photometric area of the photometric sensor, but a remarkable luminance difference exists between the dark part and the bright part. Therefore, by using an appropriate threshold value, the photometric luminance data of the photometric small area almost corresponding to the relatively dark part and the photometric luminance data of the small photometric area substantially corresponding to the relatively bright part are obtained. It is possible to classify.

There are various methods for determining the aperture value and the shutter speed, such as aperture priority and shutter speed priority. The aperture value and shutter speed may be determined in any manner. At least, the performance and accuracy of the aperture 12, the performance and accuracy of the shutter 13, and the like may be considered. Also, C
When using the electronic shutter function by the CD 14, an allowable exposure time (for example, 1 V = 1/60 or less; V is a vertical scanning period) and the like may be considered.

In any case, an appropriate exposure amount (first exposure amount) for the relatively dark portion SL of the subject image and an appropriate exposure amount (second exposure amount) for the relatively bright portion SH (The amount is smaller than the first exposure amount), the aperture 12 and the shutter 13 are controlled using the first exposure amount, and the first actual photographing is performed. The video signal output from the CCD 14 during the first main shooting is an amplifier
After being amplified at 15, preprocessing such as γ correction is performed at a preprocessing circuit 16, further converted to digital image data at an A / D converter 17, and temporarily stored in a first frame memory 21 via a changeover switch 18. Will be remembered. Subsequently, the aperture 12 and the shutter 13 are controlled using the second exposure amount, and the second main shooting is performed. In the same manner, the video signal output from the CCD 14 during the second main shooting is amplified, pre-processed,
The digital image data is A / D converted and stored in the second frame memory 22 via the changeover switch 18 as digital image data. The changeover switch 18 is controlled by the exposure control circuit 10. It goes without saying that the main photographing based on the second exposure amount may be performed before the main photographing based on the first exposure amount.

The control of the shutter speed in the actual photographing is performed by CC.
It may be realized by an electronic shutter function using D14. An electronic shutter is, as is well known, a CCD
Exposure starts by discharging (clearing) 14 unnecessary charges.
The exposure ends when the signal charges stored in the CCD 14 are read. In this case, the maximum value of the exposure time (the time from the discharge of the unnecessary charges to the reading of the signal charges) is 1
If the aperture value is determined to be within / 60 = 1V, 2/60
= 2V, two main shootings are completed. When the electronic shutter function is used, two frames of image data having different exposure amounts can be obtained in a shorter time than when the exposure time is controlled using the mechanical shutter 13.

A memory for storing a key signal in addition to the two frame memories 21 and 22 in order to perform the above-described inset synthesizing process using the two frames of image data obtained in this manner. 23, a CPU 20, and a multiplexer 24 are provided.

The CPU 20 converts the image data for each pixel stored in the first frame memory 21 into a predetermined threshold value TH (the threshold value for discriminating the level of the image data obtained by the preliminary photographing described above). Image data with a value equal to or greater than the threshold value, which can be the same value or a different value, in any case, as long as it can distinguish a relatively bright or saturated area from other areas. Belongs to a relatively bright area SH, and data 1 (1 bit) is written as a key signal at the same position on the memory 23 as the position of the pixel represented by the image data. For image data less than the threshold value, 0 is written at the position of that pixel in the memory 23. The memory 23 has a capacity capable of storing a key signal for one frame (or one field) (a key signal is one bit per pixel).

Thus, the key signal memory 23 has
Whether the image data stored in the frame memories 21 and 22 relates to pixels belonging to a relatively dark area SL of the subject image (key signal data = 0), or a relatively bright area S
Whether it relates to a pixel belonging to H (key signal data = 1)
Is written.

The multiplexer 24 is controlled by the key signal data set in the key signal memory 23. When the key signal data is 0, the multiplexer 24 selects and outputs the image data read from the frame memory 21. When the key signal data is 1, the multiplexer 24 selects and outputs the image data read from the frame memory 22.

When the setting of the key signal data in the key signal memory 23 is completed, the image data of the frame memories 21 and 22 and the key signal data of the key signal memory 23 are synchronized (that is, data relating to the same pixel is simultaneously transmitted). ) Is read out and supplied to the multiplexer 24. As described above, the multiplexer 24 outputs a frame signal according to the key signal data.
One of the image data read from the memory 21 or 22 is selectively output. The image data output from the multiplexer 24 is an inlaid composite image in which image data in a saturated (relatively bright) area in an image with a large amount of exposure is replaced by image data in a corresponding area in an image with a small amount of exposure. Is represented.

The image data output from the multiplexer 24 is converted into an analog video signal by the D / A converter 25 and output. When the analog video signal is applied to a display device such as a CRT, an image that has been fitted and synthesized is displayed. The video signal can be FM-modulated and recorded on a magnetic recording medium such as a floppy disk or a magnetic tape. Alternatively, the output image data of the multiplexer 24 is separated into luminance data and color data (Y / C separation) by the image data processing circuit 26, the data is compressed and encoded, and then a memory card (with a built-in semiconductor memory) doing,
(Also referred to as a memory cartridge or the like). Needless to say, the above-described image data inset synthesis processing need not be performed by the still video camera. In this case, the image data in the frame memories 21 and 22 are separately processed, and then stored in separate areas in the memory card. The inset synthesis of the image data will be performed by a separately provided image processing device.

A key signal may be created using image data with a small exposure amount stored in the frame memory 22. In this case, a value small enough to exclude a relatively dark portion is adopted as the threshold value. If the image data is equal to or larger than the threshold value, 1 is set as the key signal data in the memory 23, and otherwise, 0 is set in the memory 23. By doing so, image data representing an image synthesized by replacing image data of a relatively dark area in an image with a low exposure amount with image data of a corresponding area in an image with a high exposure amount is output to the multiplexer. Output from 24.

The still video camera shown in FIG. By utilizing the strobe light emission of the strobe device 28, it is possible to obtain image data suitable for fitting synthesis even if the object to be photographed is not necessarily distinguished between a bright part and a dark part.

For example, it is assumed that two objects having the same degree or a small difference in luminance are located at relatively close and far positions. If the amount of strobe light emission is adjusted so that an object at a close position is properly exposed, an object at a distant position will be underexposed. Conversely, if the amount of strobe light emission is adjusted so that an object at a distant position is properly exposed, a video signal obtained for an object at a near position is saturated.

According to the still video camera, the first photographing is performed with the amount of strobe light emission capable of obtaining an exposure suitable for photographing a relatively distant object, and is relatively close. The second photographing is performed with a strobe light emission amount such that the object is properly exposed. In this manner, image data for two frames is obtained, and the inset synthesis of the image data is performed in exactly the same manner as described above.

When the luminance difference between the object at the far position and the luminance at the close position is large, it is needless to say that the strobe light emission amount is adjusted in consideration of the luminance difference. In any case,
The strobe light emission amount may be controlled so that the average luminance difference between the obtained two frames of image data is relatively small (within two to three times).

A relatively bright area SH in the subject image
There are various methods for discriminating between the image data and the relatively dark area SL other than the above-described processing of simply comparing the image data with the threshold value TH. Hereinafter, other identification methods will be described. Only a method of extracting a relatively bright region in an image with a large amount of exposure will be described. This is because it is possible to extract a relatively dark area in an image with a small amount of exposure by exactly the same method.

For the sake of simplicity, VII-VI in FIG.
Consider a video signal along a horizontal scan line represented by the I line.

FIG. 7A shows an example of a video signal along the horizontal scanning line. The video signal in the relatively bright area SH is saturated. In the relatively dark area SL, there is a small bright spot (for example, a piece of glass, a part of metal, etc. shines due to reflection of light). Due to this bright spot, a sharp pulse-like waveform BR is generated in the video signal. Assume that it is appearing.

When such a video signal passes through a low-pass filter (hereinafter abbreviated as LPF), the signal br shown in FIG.
As shown by, the steep pulse-like waveform becomes gentle and its height decreases. If this video signal is level-discriminated using a threshold value TH set to a level higher than the peak value of the waveform br, only a bright area SH is extracted. In this way, small bright spots partially present in the dark area SL are ignored, and execution of inset synthesis in such a small area is prevented beforehand.

As described above, the relatively bright area does not necessarily have to be saturated. It can be easily understood from FIG. 7B that a bright region that is not saturated can be extracted by appropriately setting the threshold level TH (the peak value of the bright portion SH is lower than the saturation level). Should be considered low).

Techniques for filtering digital data are well known. The above discussion also applies to the processing in the CPU 20 for processing the digital image data stored in the frame memory 21 (or 22) to determine the areas SH and SL in the still video camera shown in FIG. The CPU 20 performs low-pass filtering on the digital image data, and compares the filtered image data with data representing a threshold value.

FIG. 8 shows another determination method. FIG.
FIG. 7A shows the same video signal as that shown in FIG. In this video signal, a front edge (leading edge) indicating a steep rising and a trailing edge (trailing edge) indicating a steep falling are detected. The width (or time) t1, t2, etc. from the leading edge to the trailing edge is measured. The widths t1, t2, and the like are compared with an appropriate reference width W (see FIG. 8B). Only a portion having a width larger than the reference width W is determined to be a relatively bright region (see FIG. 8C). According to this method as well, small bright spots existing in a relatively dark area can be excluded from the target area for the inlay synthesis.

It goes without saying that this method can also be executed in analog or digital manner. In the case of executing in an analog manner, a monostable multivibrator having a stabilization time corresponding to the reference width W can be used. The monostable multivibrator is triggered (set) by the leading edge of the video signal and is reset by the trailing edge. If an output is generated from the monostable multivibrator after being set and before being reset (if the time corresponding to the width W has elapsed after being set), that portion is determined to be a bright region. Digitally,
It may be determined whether the length from the front edge to the rear edge is greater than the width W.

In the method shown in FIG. 7, there is a possibility that a boundary of a bright region is slightly shifted due to a phase delay inevitably generated in filtering. On the other hand, the method shown in FIG. 8 has the advantage that the boundary between the bright area and the dark area can be accurately determined without shifting.

Although the detection of the boundary of the region appearing in the video signal on the horizontal scanning line (the boundary extending in the vertical scanning direction of the image) has been described, the same method as described above is used for the detection of the boundary extending in the horizontal direction of the image. can do. Especially when digital image data is digitally processed, filtering and detection of the width in the vertical direction are easy. In addition, by executing the method shown in FIG. 8 two-dimensionally, it is possible to exclude the bright spots having a certain area or less from the object of the inset synthesis and to perform the inset synthesis only on a bright region exceeding the certain area.

FIG. 9 shows another example of the inset synthesizing process.
Particularly, a circuit configuration for realizing a method of smoothly continuing the vicinity of the boundary between two areas to be fitted is shown. The circuit shown in FIG. 9 is replaced by the multiplexer 24 in FIG.

Image data (for example, 8 bits) read from frame memory 21 is applied to multipliers 30a, 31a, 32a,
The signals are multiplied by 0, 1, 2, 3 and 4 by 33a and 34a, respectively, and applied to a multiplexer (selection switch) 37a. The image data read from the frame memory 22 is multiplied by 4, 3, 2, 1 and 0 by multipliers 30b, 31b, 32b, 33b and 34b, respectively, to be a multiplexer.
37b. Multiplexers 37a and 37b are controlled by key signal data (3 bit data in this embodiment) provided from key signal memory 23. Multiplexer 37a is a multiplier 30a, 31a, 32a, 33a or 34a
Is selected, the multiplexer 37b switches the multiplier 30b,
Select 31b, 32b, 33b or 34b.

The outputs of the multiplexers 37a and 37b are added to each other by an adder 35, further divided by 4 by a divider 36, and output as combined image data (for example, 8 bits again).

For one pixel on the detected boundary between the relatively dark area SL and the bright area SH, the multiplexers 37a and 37b are connected to multipliers 32a and 32b, respectively.
Select Thus, on the boundary line, the arithmetic average of the image data of the frame memory 21 and the image data of the frame memory 22 becomes the composite image data.

For one pixel adjacent to the boundary line on the side of the area SL darker than the boundary line, the multiplexers 37a and 37b select the multipliers 33a and 33b, respectively. Thus, in this pixel, the average (weighted average) of the value tripled to the image data in the frame memory 21 and the value tripled to the image data in the frame memory 22 becomes the composite image data.

For all pixels inside the darker area SL than the above-mentioned pixel adjacent to the boundary, the multiplexers 37a and 37b are connected to the multipliers 34a and 34b, respectively.
Select b. As a result, the image data in the frame memory 21 is output as composite image data.

In one pixel adjacent to the boundary on the side of the region SH which is brighter than the boundary, the multiplexer 37a and the
37b selects multipliers 31a and 31b, respectively. Thereby, in this pixel, the average (weighted average) of the value multiplied by 1 for the image data of the frame memory 21 and the value multiplied by 3 for the image data of the frame memory 22 becomes the composite image data.

For all pixels inside the area SH which is brighter than the pixel adjacent to the boundary line, the multiplexers 37a and 37b are connected to the multipliers 30a and 37b, respectively.
Select 30b. As a result, the image data in the frame memory 22 is output as composite image data.

The key signal data stored in the key signal memory 23 is controlled so that the position of the pixel is on the boundary line, next to the boundary line, or farther than the boundary line, so as to control the multiplexers 37a and 37b as described above. The data is created as 3-bit data by the CPU 20 depending on which area the data belongs to.

As described above, in the vicinity of the boundary between the areas SL and SH, the composite image data is created by the weighted average (weighted according to the position) of the two types of image data to be fitted. Image data smoothly continues near the boundary. As a result, when the composite image is reproduced, the boundary between the two regions becomes natural, and the occurrence of the pseudo contour is prevented beforehand.

In the above description, the weighting for the weighted average is changed one pixel at a time, but it goes without saying that the weighting may be changed for a plurality of pixels.

FIG. 10 shows an embodiment in which an image inset and synthesis process is performed on an analog video signal in real time. The circuit of this embodiment can be applied not only to a still video camera but also to a movie video camera.

The imaging optical system includes an imaging lens 41, an aperture 42, a beam splitter 43, and two CCDs 44 and 45. An optical image representing a subject passes through a lens 41 and an aperture 42 and is split by a beam splitter 43 to form two CCDs 44.
And image on 45. As described above, the ratio between the average luminance of the relatively dark area SL and the average luminance of the relatively bright area SH is 1 to 5 in shooting in a room with a window.
~ 10 or so. In this still video camera, the light splitting ratio of the beam splitter 43 is set to 5: 1. Light having a light quantity of の of the light quantity of the incident light is incident on the CCD 44 through the beam splitter 43. 1 of the amount of incident light
/ 6 light is reflected by the beam splitter 43 and CC
It is incident on D45.

The video signal output from the CCD 44 is amplified by a preamplifier 46 and then applied to a delay circuit 54 and input to an exposure control circuit 49. The video signal output from the CCD 45 is amplified by a preamplifier 47 and then applied to an automatic gain control amplification circuit (hereinafter referred to as AGC) 48 and input to an exposure control circuit 49. The exposure control circuit 49 controls the aperture 42 via the driver 50 and adjusts the gain of the AGC 48.

For controlling the exposure amount, an output video signal of the CCD 44 having a large exposure amount is used. The exposure control circuit 49 includes an amplifier 46.
Based on the level of the video signal given from the camera so that the relatively dark area SL of the subject image is properly exposed.
Adjust 42. The shutter speed (exposure time) is fixed, and is maintained at, for example, 1/60 seconds (or 1/30 seconds). That is, no mechanical shutter is provided, and the exposure time is defined by clearing unnecessary charges of the CCDs 44 and 45 and reading out signal charges.

In this still video camera, a subject image is continuously photographed. For example, every 1/60 second (or 1/30 second), one field (or one frame) from the CCDs 44 and 45 is used. Video signal is being output.

The exposure amount is adjusted so that the relatively dark area SL of the subject image formed on the CCD 44 is properly exposed, and the division ratio of the beam splitter 43 is set to 5: 1. From this, it can be expected that the exposure amount is relatively appropriate even for the relatively bright area SH formed on the CCD 45. When the image of the relatively dark region SL and the image of the bright region SH are combined, the images are appropriately matched (for example, the image of the relatively bright region SH is compared with the image of the combined image in the combined image). An AGC 48 is provided to prevent the occurrence of a situation where the image becomes darker than the image of the dark area SL). The exposure control circuit 49 detects the peak level of the video signal of the previous field (or previous frame) given from the amplifier 47, and this peak level is kept constant also in the video signal of the next field (or frame). The gain of the AGC 48 is adjusted as described above. In this way, every field (or one frame) (every 1/60 second,
The gain of the AGC 48 is adjusted (every 1/30 second), and the brightness of the brightest part of the image in the relatively bright area SH is always kept almost constant.

The output of the amplifier 46, which is a video signal with a large amount of exposure, passes through the LPF 51, and only the low-frequency component is supplied to the comparator 52. The comparator 52 has a threshold voltage V
_TH is set. The comparator 52 generates an output when the level of the input video signal exceeds the threshold voltage _VTH . The output of the comparator 52 is input to a pulse width detection circuit 53. The pulse width detection circuit 53 includes a monostable multivibrator or the like as described above. When the pulse width of the output signal of the comparator 52 exceeds the reference width W, the pulse width detection circuit 53 converts the output signal to a time corresponding to the reference width W. Output after delay. The output signal of the pulse width detection circuit 53 is supplied to the multiplexer 56 as a control signal thereof.

A delay time equal to the time corresponding to the reference width W (or the time obtained by adding the delay time due to the operation of the LPF 51) to the delay circuits 54 and 55 is set. The output video signal of the amplifier 46 and the output video signal of the AGC 48 are delayed by these delay circuits 54 and 55 by the delay time and input to the multiplexer 56.

The multiplexer 56 normally selects and outputs the output video signal of the delay circuit 54, and when the output signal is supplied from the pulse width detection circuit 53, selects and outputs the output video signal of the delay circuit 55. As a result, inlay synthesis of images based on the above-described principle is performed. The video signal output from the multiplexer 56 is subjected to γ correction and the like in a video signal processing circuit 57.

There are various other modifications relating to the inset synthesizing process, which are described in detail in the specification of the above-mentioned inset synthesizing patent application.

FIG. 11 shows an embodiment of the solid-state electronic imaging device according to the present invention and is a schematic view of a CCD. Even if exposure is not performed twice, two types of image signals having different exposure amounts are obtained from the CCD shown in FIG. 11 and are used in the above-described inset synthesizing process.

A large number of photodiodes 60A and 60B are arranged in the vertical and horizontal directions. A filter for limiting the amount of incident light is provided on the photodiode 60A, and this filter is indicated by hatching. Filters are photodiodes every other row in the vertical direction
It is provided at 60A. Preferably, the transmittance of the filter is 5: 1 with respect to the light incident on the photodiode 60B having no filter.

A vertical transfer path 62 is arranged via a P-well 61 in the horizontal direction of the photodiodes 60A and 60B.

The vertical transfer path 12 is formed immediately below the electrodes V1 to V6 which are periodically arranged. An electrode V1 is formed adjacent to the photodiode 60A in the first row from above, and an electrode V2 is formed in a direction perpendicular to the electrode V1. An electrode V3 is formed adjacent to the photodiode 60B in the second row, and is connected to the electrode V2 in the vertical direction. Further, the electrode V3 is connected to the electrode V4 in the vertical direction.

An electrode V5 is formed adjacent to the photodiode 60A in the third row, and is connected to the electrode V4 in the vertical direction. Further, the electrode V5 is connected to the electrode V2 in the vertical direction.

The electrode V6 is formed adjacent to the photodiode 60B in the fourth row, and is connected to the electrode V2. Further, the electrode V6 is connected to the electrode V4 in the vertical direction. The electrode V4 is connected in the vertical direction to the electrode V1 adjacent to the photodiode 10A in the fifth row.

In the vertical direction, the electrodes V1, V2, V3,
V4, V5, V2 and V6 are formed by repeating periodically. These electrodes V1 to V6 are supplied with vertical transfer pulses φV1 to φV6, respectively.

The lowermost electrode V4 of the vertical transfer path 12
Are connected. First horizontal transfer path 63
An amplifier 64 is connected to the output of the first and second amplifiers via the amplifier 64 in accordance with the applied horizontal transfer pulses φH1 to φH4.
Is output.

In the CCD shown in FIG. 11, a first horizontal transfer path 63 and a second horizontal transfer path 65 are connected via a shift gate 67 whose gate is opened by a shift gate pulse. An amplifier 66 is also connected to the output of the second horizontal transfer path 65, and the second signal charge is output via the amplifier 64 in accordance with the horizontal transfer pulses φH1 to φH4.

FIGS. 12 and 13 are time charts showing the transfer of the signal charges stored in the photodiodes 60A and 60B in the first field. FIG. 13 shows in detail the portion indicated by reference numeral c in FIG. ing. FIG. 14 is a time chart showing the transfer of signal charges stored in photodiodes 60A and 60B in the second field. Figure
Reference numeral 15 is a schematic diagram showing how signal charges are transferred in the vertical transfer path 12, where the signal charges are indicated by hatching.

The output of the first signal charge stored in the photodiodes 60A and 60B will be described with reference to FIGS. Photodiodes 60A and 60B
The accumulation of the electric charges into the photodiodes 60A and 60B is started by discharging unnecessary electric charges accumulated in the respective photodiodes 60A and 60B.

The vertical transfer pulses φV1 to φV6 are applied to the electrode V1.
To V6.

[0098] electrode V1 at time t _1, V3, V4,
V [V] is applied to V5 and V6. Then given a voltage V [V] at time t ₂ to the electrodes V4 and V6, given a voltage of 2V [V] to the electrode V3.
As a result, the second signal charge stored in the photodiode 60B of the first field is transferred from the photodiode 60B to a potential well generated below the electrodes V3 and V4.

At time t ₃ , the signal charge is transferred to the vertical transfer path 12 by applying the voltage V [V] to the electrode V 5.

At time t ₄ , 2 V [V] is applied to electrode V1.
Is applied. As a result, the first signal charge stored in the photodiode 60A in the first field is transferred from the photodiode 60A to a potential well generated below the electrodes V4 and V1.

[0102] In the following, the electrodes V1, V3 at time t _5,
V4, V5 and V6 voltage V [V] is applied to the voltage of V [V] to the electrodes V1, V4 and V5 are applied at time t _6, the electrodes V1 at time t _7, V2, V
A voltage of V [V] is applied to 4 and V5.

[0102] The voltage V [V] at time t ₈ to the electrodes V1, V2 and V5 are applied, given a voltage of V [V] to the electrodes V1, V2, V3, V5 and V6 at time t ₇ . Electrode V at time t ₁₀ further
2, V3 and V6 voltage V [V] is applied to the electrode V2 at time t _11, V3, V4 and V to V6
[V] is applied. As the combination of the electrodes to which the voltage is applied changes, the potential well generated under the electrode changes and moves, so that the signal charges accumulated in the potential well are also transferred with the movement of the voltage well.

As the signal charges are transferred along the vertical transfer path 62, they are given to the horizontal transfer path 63. Photodiode 60A
The first signal charge accumulated in the first horizontal transfer path 63
, Are transferred according to the horizontal transfer pulses φH1 to φH4 applied to the first horizontal transfer path 63, and applied to the amplifier 64. In the amplifier 64, the first signal charge accumulated in the photodiode 60A is amplified and output.

When the second signal charge stored in the photodiode 60B is supplied from the vertical transfer path 62 to the first horizontal transfer path 63, a shift gate pulse is supplied to the shift gate 67. By applying the shift gate pulse to the shift gate 67, the second signal charge input to the first horizontal transfer path 63 is further transferred to the second horizontal transfer path 65. The second signal charges input to the second horizontal transfer path 65 are horizontally transferred according to horizontal transfer pulses φH1 to φH4. The second signal charge output from the second horizontal transfer path 65 is provided to the amplifier 66, amplified and output.

When the reading of the signal charges stored in the photodiodes 60A and 60B in the first field is completed, the second
The signal charges stored in the photodiodes 60A and 60B in the field are read. In reading the signal charges stored in the photodiodes 60A and 60B in the second field, vertical transfer pulses φV1 to φV6 are applied to the vertical transfer path 62 according to the time chart shown in FIG. This is performed by applying a pulse. The reading of the signal charges in the second field is almost the same as the reading of the signal charges in the first field, and therefore the description is omitted.

A first signal charge generated by light whose incident light quantity is limited by a filter is output from a first horizontal transfer path 63 included in a single CCD without exposure twice, and the second horizontal transfer path is provided. From 15, a second signal charge generated by light whose incident light amount is not limited is output.

FIG. 16 shows a signal processing circuit for an analog video signal represented by signal charges obtained from two types of horizontal transfer paths as described above. FIGS. 17A and 17B show an arrangement of RGB color filters arranged on a CCD. FIG. 17A shows a stripe filter having different arrangements of R, G, and B for each column, and FIG. ) Indicate mosaic filters, respectively. FIGS. 18A to 18C are graphs showing each output signal.

On the CCD 80, RGB color filters having different densities are arranged periodically as shown in FIG. 17 (A) or (B). As a result, color image signals having different exposure amounts are output from the CCD 80.

The first signal charge representing the color image signal output from the first horizontal transfer path 63 is amplified by the amplifier 64. The output of the amplifier 64 is an output signal a. The second signal charge representing the color image signal output from the second horizontal transfer path 65 is amplified by the amplifier 66. The output of the amplifier 66 is an output signal b.

The output signal a is supplied to one input terminal of the comparator 71 and the terminal a _{1 of the} changeover switch. Output signal b is provided to amplifier 72. The amplifier 72 amplifies the output signal b by n times when the incident light quantity at which the output signal a is saturated is 1 and the incident light quantity at which the output signal b is saturated is n, as shown in FIG. Accordingly, if the output signal output from the amplifier 72 is c, the output signal a and the output signal c
Is represented by the same gradient, and represents one image obtained by the inset synthesis. The output signal c of the amplifier 72 is applied to c ₁ terminal of the changeover switch 74.

The other terminal of the comparator 91 has a predetermined voltage E
Is added to the variable resistor 73 to which the threshold value is added. As the change-over switch 74 if the signal to be input to one terminal of the comparator 71 does not exceed the thread hold value is a ₁ terminal becomes conductive, the change-over switch 74 exceeds the thread hold values conduction c ₁ terminal The state is controlled by an output signal from the comparator 71 to be in the state. As a result, the output of the changeover switch 74 becomes
As shown in (C), the output signal a is below the threshold value.
_1, and the output signal c ₁ is a thread hold value or more. As a result, the two image signals are synthesized, and the inset synthesizing process is performed. Selector switch 74
Is supplied to the color separation circuit 75.

The color separation circuit 75 generates a luminance signal Y and color signals R, G, and B from the input color image signal, and separates and outputs the luminance signal Y. The luminance signal Y is supplied to the knee circuit 76 by the color signals R, G, and B is applied to each gamma correction circuit.

The kn of the luminance signal Y in the knee circuit 76
The ee process is performed and the result is supplied to the gamma correction circuit 77.
The gamma correction circuit 77 performs gamma correction on the input luminance signal and outputs the result.

The color signals R and G input to the gamma correction circuit 78
And B are also subjected to gamma correction and applied to the matrix circuit 79. The matrix circuit 79 receives the input color signals R, G
And a circuit for generating color difference signals RY and BY from B and B, and outputs the generated color difference signals RY and BY.

The luminance signal Y output from the gamma correction circuit 77 and the color difference signal R output from the matrix circuit 79
From -Y and BY, an inset composite image is obtained.

FIG. 19 shows an example of a circuit for moving the exposure position of the CCD in the vertical direction. FIG. 20 (A) shows the exposure position of the CCD in the first field, and FIG. 20 (B) shows the exposure position in the second field. , Respectively.

Referring to FIG. 19, a piezoelectric element 84 is
Has been fixed. The switching element 81 is included in the piezoelectric element 84.
A DC voltage is supplied from a DC voltage source 82 in accordance with the on / off operation. A CCD 86 protected by a CCD package 85 is fixed to the piezoelectric element 84. As a result, the position of the CCD 86 moves according to whether or not a voltage is applied to the piezoelectric element 84.

As shown in FIGS. 20A and 20B, a filter 87 is provided on the photodiodes 60 in odd rows. Referring to FIG. 20A, in the first field, no voltage is applied to piezoelectric element 84, and exposure is performed at a predetermined exposure position. When the exposure in the first field is completed and the reading of the signal charges stored in the photodiode 60 is completed, the exposure in the second field is performed. During exposure in the second field, a DC voltage is applied to the piezoelectric element 84. When a DC voltage is applied to the piezoelectric element 84, stress is generated in the piezoelectric element 84. As a result, as shown in FIG.
The position 86 moves, and exposure in the second field is performed. When the exposure in the second field is completed, the signal charges stored in the photodiode 60 are read.

Since the number of pixels in the vertical direction is considered to be doubled by moving the CCD shown in FIG. 11 in the vertical direction using the piezoelectric element as described above, the vertical resolution of the analog video signal obtained from the CCD 86 is obtained. Will be higher.

[Brief description of the drawings]

FIG. 1 shows a scene in the field of view of a camera.

FIG. 2 is a graph showing a histogram of photoelectric conversion characteristics and luminance of an image sensor.

FIG. 3A shows an image taken with a relatively large exposure, and FIG. 3B shows an image taken with a small exposure.

FIG. 4 is a graph showing a luminance histogram of a synthesized image.

FIG. 5 is a block diagram showing an embodiment applied to a still video camera.

FIG. 6 shows sample points of image data.

FIG. 7A is a waveform diagram showing an example of a video signal, and FIG.
It is a waveform diagram which shows the video signal after passing through PF.

FIGS. 8A to 8C are waveform diagrams showing a process for detecting a bright area having a certain width or more.

FIG. 9 is a block diagram showing an example of a circuit for calculating a weighted average of image data at a boundary between a bright region and a dark region.

FIG. 10 is a block diagram showing an embodiment for performing inlay synthesis of a video signal in real time.

FIG. 11 shows an embodiment of the present invention and is a schematic view of a CCD.

FIG. 12 is a time chart of transfer of signal charges.

FIG. 13 is a partially enlarged view of the time chart shown in FIG. 12;

FIG. 14 is a time chart of transfer of signal charges.

FIG. 15 shows how signal charges are transferred.

16 is a signal processing circuit for an image signal obtained by the CCD shown in FIG.

FIG. 17A shows an example of an arrangement of a stripe filter.
(B) shows an example of the arrangement of the mosaic filters.

FIGS. 18A to 18C show changes in signals of the circuit shown in FIG.

FIG. 19 shows a moving circuit of the CCD.

FIGS. 20A and 20B show the movement of the position of the CCD.

[Explanation of symbols]

 60, 60A, 60B Photodiode 62 Vertical transfer path 63, 67 Horizontal transfer path

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Kazuyuki Masuki 1-6-6 Matsuzakadaira, Yamato-cho, Kurokawa-gun, Miyagi Prefecture Fujifilm Micro Devices Co., Ltd. In-house (56) References JP-A-3-117281 (JP, A) JP-A-60-52172 (JP, A) JP-A-1-220986 (JP, A) JP-A-2-1344991 (JP, A) JP-A-3-3577 (JP, A) JP-A-3-94586 (JP, A) (58) Field surveyed (Int. Cl. ⁷ , DB name) H04N 5/30-5/335

Claims (2)

(57) [Claims] 1. A large number of photoelectric conversion elements which are arranged in a vertical direction and a horizontal direction and accumulate signal charges of an amount corresponding to the amount of incident light, and are provided in said photoelectric conversion elements in a horizontal direction at predetermined intervals in a vertical direction. A filter for limiting the amount of light incident on the photoelectric conversion element, and a filter provided adjacent to each column in the vertical direction of the photoelectric conversion element and stored in the photoelectric conversion element provided with the filter according to a transfer signal. A vertical transfer unit for distinguishing and transferring each of the first signal charge and the second signal charge stored in the photoelectric conversion element not provided with the filter, and horizontally transferring the first signal charge input from the vertical transfer unit. the first horizontal transfer section for transferring in the direction, the second horizontal transfer unit for transferring the second signal charges input from the vertical transfer unit in a horizontal direction, and the all
The photoelectric conversion element or the filter in the vertical direction
A solid-state electronic imaging device comprising: a moving unit that moves by one row of the electric conversion elements . 2. A large number of photoelectric conversion elements which are arranged in a vertical direction and a horizontal direction and accumulate signal charges of an amount corresponding to the amount of incident light, and are provided in said horizontal photoelectric conversion elements at predetermined intervals in the vertical direction. A first filter for limiting the amount of light incident on the photoelectric conversion element according to a first transmittance; a first filter provided for the photoelectric conversion element not provided with the first filter; A second filter for limiting the amount of light incident on the element, the second filter being provided adjacent to each column in the vertical direction of the photoelectric conversion element, and stored in the photoelectric conversion element provided with the first filter in accordance with a transfer signal; A vertical transfer unit that distinguishes and transfers the first signal charge and the second signal charge stored in the photoelectric conversion element provided with the second filter, respectively, and the second signal input from the vertical transfer unit. The first horizontal transfer unit for transferring the signal charges in the horizontal direction, the second horizontal transfer unit for transferring the second signal charges in a horizontal direction to enter from the vertical transfer section, and the all
Photoelectric conversion element or the first filter and the second filter
2 filters vertically for one row of the photoelectric conversion element
A solid-state electronic imaging device comprising: moving means for moving .