JP3988760B2 - Solid-state imaging device - Google Patents

Description

  The present invention relates to a solid-state imaging device capable of expanding a dynamic range of a captured image.

  Conventionally, as a solid-state imaging device for synthesizing two image signals having different exposure amounts to obtain a video signal having a wide dynamic range, for example, there are those disclosed in Patent Document 1 and Patent Document 2.

  In Patent Document 1, a digital still that can obtain an image having a wide dynamic range by level-shifting two consecutive field images taken with different exposure times and then combining them with one frame image. A camera is disclosed.

  Further, in Patent Document 2, it is possible to obtain an image with a wide dynamic range by level-shifting a plurality of frame images obtained from a plurality of CCDs with different exposure times and then combining them with one frame image. A digital still camera is disclosed.

In addition, an example of a video camera in which the dynamic range is expanded using a special CCD that can read out a long exposure signal and a short exposure signal within one field period is known (Non-Patent Document 1).
JP-A-9-214829 JP-A-9-275527 ITE Technical Report Vol.22, No.3, pp1-6 (1998) “Development of single-panel Hyper-D color camera signal processing method”

  However, in the digital still camera disclosed in Patent Document 1, for example, since two continuous field images shot with different exposure times are combined, the combined image has an image resolution of one field, that is, Only half the number of pixels of the CCD can be obtained, and there is a concern that the resolution of the captured image is insufficient.

  On the other hand, in the digital still camera disclosed in Patent Document 2, in order to synthesize image signals with different exposure times taken by a plurality of CCDs, the synthesized image has an image resolution of one frame, That is, although the resolution corresponding to the number of pixels of the CCD can be obtained, a plurality of CCDs are required, which is disadvantageous in terms of the size and cost of the imaging device.

In addition, in the case of the imaging apparatus already reported in Non-Patent Document 1, a special CCD is required to expand the dynamic range of the captured image.
The present invention has been made in view of the above problems, and is inexpensive by using one solid-state imaging device generally used in a consumer solid-state imaging device, and has a dynamic range with an image resolution comparable to the number of pixels of a CCD. An object of the present invention is to provide a solid-state imaging device capable of capturing an image with an enlarged image.

The invention of the first aspect, a plurality of photodiodes arranged in a matrix, and the solid-state image sensor having a transfer means for outputting a charge accumulated on the photodiode to the outside, incident on the photodiode Light shielding means for shielding the light to be transmitted, and signal synthesis means for synthesizing the image signal output from the solid-state imaging device, wherein the solid-state imaging device is configured to store the charge accumulated on the photodiode as the first exposure. Only a part is output via the transfer means after application of the first readout control pulse, and after the application of the first readout control pulse, on the photodiode in the second exposure that is completed upon completion of exposure by the light shielding means. The charge accumulated in the first readout control pulse is output via the transfer means after the application of the second readout control pulse, and the signal synthesis means Captured by the serial second exposure, it is characterized in synthesizing the image signals output respectively through said transfer means.

The invention according to claim 2 of the present application includes a plurality of photodiodes arranged in a matrix, a solid-state imaging device having transfer means for outputting charges accumulated on the photodiodes to the outside, and incident on the photodiodes Light shielding means for shielding the light to be transmitted, and signal synthesis means for synthesizing the image signal output from the solid-state imaging device, and the solid-state imaging device uses the charge accumulated on the photodiode as the first exposure. After application of the first readout control pulse, output is performed via the transfer means by field readout. Further, after application of the first readout control pulse, the second exposure is completed upon completion of exposure by the light shielding means on the photodiode. The accumulated charge is output via the transfer means after application of the second readout control pulse, and the signal synthesis means The image signals photographed by the first exposure and the second exposure are respectively combined with the image signals output via the transfer means, and the image signals photographed by the first exposure are obtained by the second exposure. The present invention is characterized in that the image has fewer pixels than the photographed image signal.

According to the present invention, the exposure and signal readout mode of the solid-state imaging device is controlled, and a short-time exposure signal for one field and a long-time exposure signal for one frame are photographed and synthesized. It is possible to shoot an image with an increased dynamic range while having a resolution equivalent to the number of pixels. Furthermore, as the solid-state image pickup device used in the solid-state image pickup device, an IT-CCD generally used in a consumer solid-state image pickup device can be used. Therefore, it is necessary to use a plurality of solid-state image pickup devices or special solid-state image pickup devices. Therefore, the apparatus can be configured at low cost.

(Embodiment 1)
FIG. 1 is a block diagram of the solid-state imaging device according to Embodiment 1 of the present invention. In the figure, 1 is an optical lens, 2 is a mechanical shutter that also serves as an optical diaphragm, and 3 is a solid-state image sensor. In the first embodiment, an interline transfer CCD (commonly used in a consumer solid-state image pickup device) IT-CCD). 4 is an analog signal processing means comprising a correlated double sampling circuit and an automatic gain control (AGC) circuit, 5 is an A / D conversion means, and 6 is an image signal converted into a digital signal by the A / D conversion means 5. An image memory to be stored. Reference numeral 7 denotes a signal synthesis means for synthesizing two systems of image signals read from the image memory 6.

  The signal obtained by the signal synthesizing unit 7 is subjected to processing such as separation of the luminance signal and color signal, noise removal, edge enhancement, matrix calculation, and encoding to a specific format in the digital signal processing unit 8. The mechanical shutter drive control means 9 is a means for controlling the opening and closing of the mechanical shutter 2, and the solid-state image sensor drive control means 10 is a means for controlling exposure control, signal readout mode, timing, etc. of the solid-state image sensor 3. . It is assumed that the operation mode and operation timing of all the components including these are controlled by the system control means 11 in an integrated manner.

  2A, 2 </ b> B, 2 </ b> C, and 2 </ b> D are schematic diagrams for explaining the operation and configuration of the solid-state imaging device 3. In the first embodiment of the present invention, the solid-state imaging device 3 is an interline transfer CCD (IT-CCD) that can read out signals in two readout modes, a field readout mode and a frame readout mode. The description will be made with a so-called 4 × 2 pixel configuration of 4 vertical pixels and 2 horizontal pixels as shown in FIG.

  FIGS. 2A and 2B are diagrams for explaining the field readout mode in the IT-CCD. In FIG. 2A, a photodiode is a portion in which signal charges corresponding to the intensity of light are accumulated by photoelectric conversion. After a predetermined time, the accumulated charges are applied to a vertical transfer CCD by an applied control pulse. Moving. At this time, charges of two adjacent upper and lower photodiodes are mixed on the vertical transfer CCD and output to the outside via the horizontal transfer CCD. The above is the read operation of the first field.

  In the second field, as shown in FIG. 2B, the pair of photodiodes mixed on the vertical transfer CCD is shifted by one pixel in the vertical direction compared to the case of the first field. Thus, an image signal corresponding to one frame of the interlace method can be read out by reading out signals for two fields.

  Next, the frame reading mode will be described with reference to FIGS. In the frame readout mode, first, in the first field (FIG. 2 (c)), the charge accumulated in the photodiode is transferred to the vertical transfer CCD by skipping one pixel in the vertical direction, and this is output to the outside through the horizontal transfer CCD. . In the second field (FIG. 2 (d)), the charge of the photodiode that has not been transferred to the vertical transfer CCD in the first field is transferred to the vertical transfer CCD, and this is output to the outside via the horizontal transfer CCD. Thus, in the frame readout mode, the charges on the photodiode are output to the outside without being mixed by the vertical transfer CCD. Thus, an image signal corresponding to one frame of the interlace method can be read out by reading out signals for two fields.

  FIG. 3 is a complementary color checkered type color filter array diagram formed on the solid-state imaging device 3. In FIG. 3, Mg represents magenta, G represents green, Ye represents yellow, and Cy represents cyan. As shown in FIG. 3, one color filter corresponds to one pixel of the photodiode.

  FIG. 4 is a block diagram illustrating a configuration example of the signal synthesis unit 7. In the figure, reference numeral 701 denotes a two-line adding means for adding image signals for two horizontal scanning lines of the image signal output from the image memory 6 (hereinafter, the image signal corresponding to the horizontal scanning line is simply referred to as a horizontal line or a horizontal line). Called line signal). Reference numeral 702 denotes an interpolation unit that performs an interpolation process in the vertical direction on the image signal output from the image memory 6. The weighting addition means 703 is a means for weighting and adding the outputs of the two horizontal line addition means 701 and the interpolation means 702.

  FIG. 5 is a block diagram showing a configuration example of the two horizontal line addition means 701. In the drawing, reference numeral 70101 denotes a one-line memory, which is a means for delaying one line of the image signal output from the image memory 6 by one horizontal synchronization period. Reference numeral 70102 denotes an adder, and the horizontal line signal delayed in the 1-line memory 70101 and the horizontal line signal input to the 2-horizontal line adding means 701 are added in the adder 70102 so that two adjacent upper and lower lines are added. Is added.

  FIG. 6 is a block diagram illustrating a configuration of the interpolation unit 702. In the figure, reference numerals 70201 and 70202 denote one-line memories, which are means for delaying one line of the image signal output from the image memory 6 by one horizontal synchronization period. Reference numerals 70203 and 70204 denote amplifier means for multiplying the input signal from the image memory 6 and the output signal of the one-line memory 70202 by a certain gain. Reference numeral 70205 denotes an adder that adds the signals multiplied by the gains by the amplifier means 70203 and 70204.

  FIG. 7 is a block diagram showing the configuration of the weighted addition means 703. In the figure, reference numeral 70301 denotes synthesis coefficient generation means, which generates a coefficient k (1 ≧ k ≧ 0) according to the signal level for each pixel of the signal that has passed through the two horizontal line addition means 701, and k and 1− The value k is supplied to multipliers 70302 and 70303. Multipliers 70302 and 70303 multiply k and 1-k by the signal that has passed through interpolation means 702 and the signal that has passed through 2 horizontal line addition means 701, and the result is added by adder 70304 and output.

The operation of the solid-state imaging device according to the first embodiment of the present invention configured as described above will be described below.
In Embodiment 1 of the present invention, two images of a short exposure signal (hereinafter referred to as a short signal) and a long exposure signal (hereinafter referred to as a long signal) are taken and synthesized. It is characterized by taking an image with an expanded dynamic range. The principle of such dynamic range expansion will be described with reference to FIG. FIGS. 8A and 8B show the relationship between the brightness of the subject at the time of exposure (the amount of light incident on the solid-state image sensor) and the signal amount output from the solid-state image sensor. As shown in FIG. 8A, during long exposure, the amount of charge generated on the photodiode of the solid-state image sensor by the incident light is large, and naturally the amount of signal output is also large. However, there is an upper limit on the amount of charge accumulated in the photodiode, and if this upper limit is exceeded, saturation occurs, that is, a signal is destroyed, and the subject image cannot be accurately reproduced. Conversely, if the exposure time is set short as shown in FIG. 8B, saturation can be avoided, but this time the S / N of the low-luminance portion in the subject deteriorates. Therefore, if a signal obtained by long exposure (long signal) and a signal obtained by short exposure (short signal) are used to synthesize an image consisting of a long signal for the low luminance part and a short signal for the high luminance part. Thus, it is possible to reproduce from the low-luminance portion to the high-luminance portion of the subject, and it is possible to expand the dynamic range of the imaging apparatus. At this time, if the short signal is multiplied by a gain corresponding to the ratio of exposure amount to the long signal (exposure time ratio) (FIG. 8 (c)), then synthesis is performed as shown in FIG. 8 (d). In addition, the dynamic range can be expanded according to the exposure amount ratio. For example, when the exposure amount ratio (exposure time ratio) between the long signal and the short signal is 1: D, the dynamic range can be expanded to D times.
Hereinafter, a specific example of an imaging apparatus capable of expanding the dynamic range of a captured image according to the above principle will be described.
First, a method of photographing short signals and long signals will be described with reference to FIG. FIG. 9 is a timing chart relating to exposure of a subject image and reading of an exposed signal in the solid-state imaging device 3. In the figure, (a) is a vertical synchronization signal, (b) is a readout control pulse for controlling signal charge readout from a photodiode of the solid-state image sensor 3, (c) is an open / close state of the mechanical shutter 2, and (d) is an open / close state. An exposure signal on the photodiode of the solid-state image sensor 3, (e) shows a signal output from the solid-state image sensor 3.
At the time of short signal exposure, the mechanical shutter 2 is opened, and exposure is performed for a necessary exposure time, for example, 1/1000 second using the electronic shutter function. After the exposure for 1/1000 second is completed, the accumulated charge on the photodiode is moved to the vertical transfer CCD by the read control pulse. At this time, the solid-state imaging device 3 is assumed to be driven in the field readout mode. As described with reference to FIG. 2A, the accumulated charges on the photodiode are mixed on the vertical transfer CCD and read out to the outside. At this time, only the image signal for the first field is read out. FIG. 10 shows a short signal read in the field read mode. Note that the number of photodiodes in the vertical direction of the solid-state imaging device 3 is N (for convenience of explanation, N is an even number, but is not limited thereto). As shown in FIG. 10, the read short signal is four signals of Ye + Mg, Cy + G, Ye + G, and Cy + Mg obtained by adding signals of four colors Ye, Cy, G, and Mg, respectively. It becomes. The number of lines in the vertical direction is 1/2 of the number N of photodiodes in the vertical direction.
Next, the long signal is exposed while the short signal is being read out. The exposure period of the long signal is, for example, 1/100 second. The exposure time of the long signal is controlled by opening / closing the mechanical shutter 2, and the mechanical shutter 2 is closed 1/100 seconds after the start of exposure of the long signal to complete the exposure. By closing the mechanical shutter 2 in this way, the signal exposed for a long time is not exposed excessively during reading.
When the exposure of the long signal is completed, the accumulated charge on the photodiode is transferred to the vertical transfer CCD by the read control pulse. At this time, the solid-state imaging device 3 is assumed to be driven in the frame readout mode, and as described with reference to FIG. 2C, the photodiode charge corresponding to the odd-numbered lines in the vertical direction is read out for the first field. After completing the signal reading of the first field, this time, the charge of the photodiode corresponding to the even-numbered line in the vertical direction is read (second field), and thereby the long signal reads the signal corresponding to one frame from the solid-state imaging device 3. Note that the period of the vertical synchronization signal shown in FIG. 9A is, for example, 1/100 second, and signal reading for one field from the solid-state imaging device 3 is completed within one period of the vertical synchronization signal. . FIG. 11 shows a long signal read in the frame reading mode. As shown in FIG. 11, in the long signal read out, the first field is a two-color signal of Ye and Cy, and the second field is a two-color signal of G and Mg. The number of lines in the vertical direction is ½ of the number N of photodiodes in the vertical direction in each field. When the two fields are combined, an N-line signal corresponding to one frame is obtained.
By performing the exposure and signal readout as described above, it is possible to obtain two signals having different exposure times, that is, a short signal that is one field image and a long signal that is one frame image. In addition, since the number of horizontal lines of the short signal is ½ that of the long signal, the short signal has a smaller number of pixels than the long signal.
Next, the two signals having different exposure times obtained by the solid-state imaging device 3 are converted into digital signals by the A / D conversion means 5 through the analog signal processing means 4 and temporarily stored in the image memory 6.
A long signal and a short signal are read from the image memory 6. When the long signal is read from the image memory 6, the long signal is the first line when viewed as a one-frame image such as the first line of the first field, the first line of the second field, the second line of the first field, and so on. It is assumed that data are read sequentially from the line. The long signal read from the image memory 6 is sent to the two horizontal line adding means 701. In the two horizontal line addition means 701, when viewed as a frame signal, long signals of two adjacent upper and lower lines are added and mixed. This is because when a long signal and a short signal are combined, if the signal formats of the two signals are different from each other, the combination cannot be performed. The same processing as the pixel mixing on the transfer CCD is performed, and for the short signal, one field image is converted into one frame image by the interpolation means 702.
FIG. 12 (a) shows a long signal after the signals of two adjacent upper and lower lines added and mixed in the two horizontal line addition means 701, FIG. 12 (b) shows a short signal before interpolation processing, and FIG. 12 (c). Shows the short signal after interpolation processing. As shown in FIGS. 12A and 12C, the signal formats of the long signal and the short signal are matched by the two horizontal line addition processing for the long signal and the interpolation processing for the short signal.
The interpolating means 702 converts the field image shown in FIG. 12 (b) into the frame image shown in FIG. 12 (c) by interpolation processing. The method will be described below.
For example, when the horizontal line signal between the second line and the third line in FIG. 12B is obtained by interpolation processing, it is necessary to create a horizontal line signal composed of Ye + G and Cy + Mg signals. At this time, since the nearest lines consisting of Ye + G and Cy + Mg signals are the second line and the fourth line, a line between the second line and the third line is obtained by interpolation processing from both. However, since the spatial distance between the position for obtaining the horizontal line signal by the interpolation process and the second line and the fourth line is not equal, weighting is required according to the distance. Therefore, the interpolation unit 702 has a configuration in which the upper and lower end lines except for the center of the three consecutively input horizontal line signals are input to the multipliers 70203 and 70204. The numbers to be multiplied may be weighted as 1/4 and 3/4, respectively, and the multiplication results may be added by an adder 70205.
Note that the number multiplied by the multipliers 70203 and 70204 is determined because the ratio of the spatial distance between the position where the horizontal line signal is obtained by interpolation processing and the second line and the fourth line is 1: 3. .
Similarly, when the horizontal line signal between the third line and the fourth line is obtained by interpolation processing, it is necessary to create a horizontal line signal composed of Ye + Mg and Cy + G signals. At this time, the nearest Ye + Mg Since the lines composed of Cy + G signals are the third line and the fifth line, weighting is performed according to the ratio of the distance between them, and the line between the third line and the fourth line is obtained by interpolation processing. Can be sought.

  Through the above processing, a signal corresponding to one frame obtained by performing interpolation processing from a long signal for one frame and a short signal for one field is generated.

  Means for synthesizing the long signal and the short signal and synthesizing a signal with an expanded dynamic range is a weighted addition means 703. In the weighting and adding means 703, a synthesis coefficient k corresponding to the signal level of each pixel of the long signal is obtained by the synthesis coefficient generating means 70301 shown in FIG. 7, and the long signal and the screen are displayed in units of one pixel according to the synthesis coefficient k. The short signal that has become one frame image is synthesized by the interpolation processing existing at the same spatial position.

  FIG. 13 shows an example of a method for obtaining the synthesis coefficient k for each pixel from the signal level of the long signal in the synthesis coefficient generating means 70301. As shown in FIG. 13, when two threshold values Th_min and Th_max are set for the long signal level and the long signal level is (Expression 1), that is, when the signal level of the long signal is equal to or less than Th_min and there is no possibility of saturation. The combination coefficient k is 0, and when the long level is (Expression 2), that is, when the long signal level is equal to or higher than Th_max and the output of the solid-state imaging device is close to the saturation level, the combination coefficient k is set to 1. Note that the threshold values Th_max and Th_min are appropriately determined according to the saturation characteristics and S / N of the solid-state imaging device to be used.

  Further, when the long signal level is (Expression 3), that is, when the long signal level is intermediate, the synthesis coefficient k is determined by the linear expression of (Expression 4) as shown in FIG.

  Using the synthesis coefficient k determined as described above, the long signal and the short signal are synthesized by (Equation 5) for each pixel. A signal obtained by combining the long signal and the short signal is defined as a combined signal.

  For example, the combined signal (Ye + Mg) M11 is obtained from the long signal (Ye + Mg) L11 shown in FIG. 14 and the short signal (Ye + Mg) S11 having the same spatial position as this (Ye + Mg) L11. In this case, if the synthesis coefficient determined from the long signal is k11, the synthesis is performed according to (Equation 6).

  Other pixels of the synthesized signal are also obtained from the long signal and the short signal existing at the same spatial position as in (Equation 6).

  The constant D multiplied by the short signal in (Equation 5) and (Equation 6) is the ratio of the exposure amount of the long signal to the short signal (exposure time ratio), for example, the exposure amount of the long signal (exposure time). ) Is TL, and the exposure amount (exposure time) of the short signal is TS, D is obtained by (Expression 7).

  In this way, using the long signal and the short signal, the portion where the signal level of the long signal is equal to or less than the threshold Th_min is a long signal, and the portion where the signal level is equal to or greater than the threshold Th_max, that is, the output of the solid-state imaging device 3 is close to saturation. The portion where the signal is high and the signal is crushed normally) is a short signal, and the intermediate brightness portion is a synthesized signal composed of a signal obtained by weighted addition of the long signal and the short signal. It is possible to expand the dynamic range of the signal.

  However, of the synthesized signal whose dynamic range has been expanded, the portion consisting of the long signal is originally an image signal of one frame, so that the image resolution is high. On the other hand, since the portion consisting of the short signal is synthesized from the image signal of one field, the image resolution is lower than that of the portion consisting of the long signal. However, in general, shooting conditions that cause the signal level of the entire screen to approach saturation are rare, and even under such conditions, the signal level of the entire screen is saturated to limit the amount of incident light by narrowing the optical aperture. However, it is unlikely in practical use that the portion of the short signal occupies most of the photographed image. In addition, when an image is expressed with a limited gradation, a high gradation portion, that is, a portion with a high signal level, is often assigned a smaller gradation than a low / medium luminance portion. For this reason, the resolution degradation of the portion consisting of the short signal is not so conspicuous, and it is considered that even if the long signal and the short signal are synthesized by the method as described above, a synthesized image having a resolution equivalent to the number of pixels of the CCD can be obtained.

  As described above, the synthesized signal synthesized by the signal synthesizing unit 7 is processed by the digital signal processing unit 8 such as separation of luminance and color signals, noise removal, edge enhancement, gamma correction, matrix calculation, encoding to a specific format, and the like. Is given. Since the signal processing in the digital signal processing means 8 is not directly related to the object of the present invention, detailed description thereof is omitted.

  As described above, in the solid-state imaging device according to Embodiment 1 of the present invention, the exposure and signal readout modes of the solid-state imaging device 3 are controlled, and the short-time exposure signal for one field and the long-time exposure signal for one frame. By capturing these images and synthesizing them, it is possible to capture an image having a resolution equivalent to the number of pixels of the CCD and an expanded dynamic range. Furthermore, as the solid-state image pickup device used in the solid-state image pickup device, an IT-CCD generally used in a consumer solid-state image pickup device can be used. Therefore, it is necessary to use a plurality of solid-state image pickup devices or special solid-state image pickup devices. Therefore, the apparatus can be configured at low cost.

  (Embodiment 2) The solid-state imaging device according to Embodiment 2 of the present invention is different from that of Embodiment 1 of the present invention shown in FIG. And the processing performed by the same means is different. Hereinafter, the description of the processing content portion similar to that of the first embodiment of the present invention will be omitted, and only the portion different from the first embodiment of the present invention will be described.

  FIG. 15 is a block diagram of weighted addition means 704 in Embodiment 2 of the present invention. In the figure, reference numeral 70401 denotes a luminance signal extracting means for extracting a luminance signal component from the long signal that has passed through the two horizontal line adding means 701. 70402 is a synthesis coefficient generating unit, which generates a coefficient k (1 ≧ k ≧ 0) according to the luminance signal level of the luminance component of the long signal that has passed through the luminance signal extracting unit 70401, and becomes k and 1−k. The value is supplied to multipliers 70403 and 70404. Multipliers 70403 and 70404 multiply k and 1-k by the short signal that has passed through the interpolation means 702 and the long signal that has passed through the two horizontal line addition means 701, and the result is added and output by the adder 70405.

  FIG. 16 is a block diagram illustrating a configuration example of the luminance signal extraction unit 70401. In the figure, 704011 is a means for delaying the input signal by a period of one pixel. 704012 is an adder, and the adder 704012 adds the pixel signal delayed by the one-pixel delay unit 704011, and the pixel signal input to the luminance signal extraction unit 70401, thereby adding two pixels adjacent in the horizontal direction. And only the low frequency component of the signal is extracted. The low frequency component of the signal extracted by the luminance signal extraction means 70401 corresponds to the luminance signal of the image signal.

The operation of the solid-state imaging device according to the second embodiment of the present invention configured as described above will be described below.
Unlike the first embodiment of the present invention, in the second embodiment of the present invention, the synthesis coefficient used when synthesizing the long signal and the short signal is determined based on the signal level of the luminance signal extracted from the long signal. .
Therefore, the weighted addition means 704 has luminance signal extraction means 70401 which is means for extracting the luminance signal from the long signal.

  The luminance signal extraction means 70401 sequentially adds the signals of the two pixels adjacent in the horizontal direction out of the outputs of the two horizontal line addition means 701, so that the luminance component of the long signal (hereinafter referred to as this) long luminance signal).

For example, when obtaining the long luminance signal YL11 from the long signal (Ye + Mg) L11 and the long signal (Cy + G) L12 shown in FIG. 17, add (Ye + Mg) L11 and (Cy + G) L12. become. Similarly, when obtaining the long luminance signal YL12, (Cy + G) L12 and (Ye + Mg) L13 are added.
A method for determining the synthesis coefficient based on the luminance signal (long luminance signal) extracted from the long signal will be described below.

  FIG. 18 shows an example of a method for obtaining the synthesis coefficient k for each pixel from the signal level of the long luminance signal in the synthesis coefficient generating means 70402. As shown in FIG. 18, when two threshold values Th_min ′ and Th_max ′ are set for the long luminance signal level and the long luminance signal level is (Equation 9), that is, the luminance level of the subject is lower than Th_min ′. In this case, the synthesis coefficient k is set to 0, and the synthesis coefficient k is set to 1 when the long luminance signal level is (Equation 10), that is, when the luminance level of the subject is higher than Th_max ′. Note that the thresholds Th_max ′ and Th_min ′ are appropriately determined according to the saturation characteristics and S / N of the solid-state imaging device to be used.

  When the long luminance signal level is (Equation 11), that is, when the luminance is intermediate between the low luminance and the high luminance, the synthesis coefficient k is determined by the linear expression of (Equation 12) as shown in FIG. To do.

  Using the synthesis coefficient k determined as described above, the long signal and the short signal are synthesized by (Equation 5) for each pixel. A signal obtained by combining the long signal and the short signal is defined as a combined signal.

  For example, the combined signal (Ye + Mg) M11 is obtained from the long signal (Ye + Mg) L11 shown in FIG. 19 and the short signal (Ye + Mg) S11 having the same spatial position as this (Ye + Mg) L11. In this case, the synthesis is performed by (Equation 13) based on the synthesis coefficient (this is referred to as ky11) determined from the long luminance signal YL11 having the same spatial position as these two signals.

  Other pixels of the synthesized signal are also obtained from the long signal and the short signal existing at the same spatial position as in (Equation 13).

  Note that the constant D multiplied by the short signal in (Equation 13) is the ratio of the exposure amount of the long signal and the short signal (exposure time ratio) as in the first embodiment of the present invention. Is required.

In this way, using the long signal and short signal, the low luminance part is a long signal, the high luminance part is a short signal, and the intermediate luminance part between the low luminance part and the high luminance part is a signal obtained by weighting and adding the long signal and the short signal. It is possible to expand the dynamic range of the captured image signal by synthesizing the synthesized signal consisting of
In addition, since the luminance signal can be said to be a low-frequency component extracted from the long signal, when determining the synthesis coefficient based on this luminance signal, the influence of the noise component in the long signal on the synthesis coefficient determination can be reduced. it can.

  As described above, also in the solid-state imaging device according to the second embodiment of the present invention, the exposure and signal readout modes of the solid-state imaging device 3 are controlled, and the short-time exposure signal for one field and the long-time exposure signal for one frame. By capturing these images and synthesizing them, it is possible to capture an image with an increased dynamic range while having a resolution comparable to the number of CCD pixels. Furthermore, as the solid-state image pickup device used in the solid-state image pickup device, an IT-CCD generally used in a consumer solid-state image pickup device can be used. Therefore, it is necessary to use a plurality of solid-state image pickup devices or special solid-state image pickup devices. Therefore, the apparatus can be configured at low cost.

(Embodiment 3)
FIG. 20 is a block diagram of the solid-state imaging device according to Embodiment 3 of the present invention. In the figure, an optical lens 1, a mechanical shutter 2 that also serves as an optical aperture, a solid-state imaging device 3, an analog signal processing unit 4, an A / D conversion unit 5, an image memory 6, a shutter driving unit 9, and a solid-state imaging device driving unit 10. The functions and operations of the two horizontal line adder 701, the luminance signal extractor 70401, and the interpolator 702 are the same as those in the first and second embodiments of the present invention. A description thereof will be omitted.

  In the block diagram shown in FIG. 20, components other than those described above will be described. Reference numeral 12 denotes a luminance signal interpolation unit that performs vertical interpolation processing on the output of the luminance signal extraction unit 70401, and the luminance signal synthesis unit 13 This is a means for synthesizing the outputs of the signal extraction means 70401 and the luminance signal interpolation means 12. Note that the luminance signal input to the luminance signal interpolation unit 12 is a luminance signal extracted from the short signal, and hence is referred to as a short luminance signal, and the luminance signal extracted from the long signal is referred to as a long luminance signal. Therefore, a signal directly inputted from the luminance signal extracting unit 70401 to the luminance signal synthesizing unit 13 is a long luminance signal, and a signal inputted from the luminance signal interpolating unit 12 to the luminance signal synthesizing unit 13 is a signal after interpolation processing of the short luminance signal. It becomes.

  The signal synthesis means 14 is a means for synthesizing the outputs of the 2-line addition means 701 and the interpolation means 702. The 1-line memories 15, 16, 17, and 18 are delay means for one horizontal synchronization period required when synchronizing the outputs of the signal synthesis means 14, and the outputs and signals of the 1-line memories 15, 16, 17, and 18 A synchronizer 19 obtains a signal having a red (R) component and a signal having a blue (B) component at the same spatial position from a total of five horizontal line signals output from the combiner 14.

  The luminance signal obtained by the luminance signal synthesizing unit 13, the signal having the red (R) component and the signal having the blue (B) component obtained by the synchronizing unit 19 are subjected to noise removal and edge processing in the digital signal processing unit 20. Processing such as emphasis, matrix operation, and encoding to a specific format is performed. It is assumed that the operation modes and operation timings of all the components including these are controlled by the system control means 21 in an integrated manner.

  FIG. 21 is a block diagram showing the configuration of the luminance signal interpolation means 12. In the figure, reference numeral 1201 denotes a one-line memory, which is a means for delaying one line of the image signal output from the luminance signal extracting means 70401 by one horizontal synchronization period. Reference numerals 1202 and 1203 denote amplifier means, which respectively multiply the signal input through 1201 and the signal input to the luminance signal interpolation means 12 through the luminance signal extraction means 70401 by a certain gain. An adder 1204 adds the signals multiplied by the gains by the amplifier units 1202 and 1203.

  FIG. 22 is a block diagram showing the configuration of the luminance signal synthesis means 13. In the figure, reference numeral 1301 denotes synthesis coefficient generation means, which generates a coefficient k (1 ≧ k ≧ 0) corresponding to the signal level for each pixel of the long luminance signal passed through the luminance signal extraction means 70401, and k and 1 A value of −k is supplied to the multipliers 1302 and 1303. Multipliers 1302 and 1303 multiply k and 1-k by the short luminance signal and the long luminance signal, and the result is added by an adder 1304 and output.

  FIG. 23 is a block diagram showing the configuration of the signal synthesis means 14. In the figure, reference numerals 1401 and 1402 denote multipliers which respectively multiply the short signal and the long signal after addition of two horizontal lines by the coefficients k and 1-k supplied from the luminance signal synthesis means 13. The multiplication results are added by an adder 1403 and output.

  FIG. 24 is a block diagram showing the configuration of the synchronization means 19. In the figure, reference numeral 1901 selects three signals from input signals, selectors that output to output A, output B, and output C, and 1902 and 1903 multiply constants in signals output from output B and output C. An amplifier means, and the multiplied signal is added by an adder 1904. A selector 1905 distributes the output A of the selector 1901 and the output of the adder 1904 to an output D and an output E, and outputs them. Note that the selection of the signal output destination by the selectors 1901 and 1905 is made according to the color component of the signal, as will be described later.

The operation of the solid-state imaging device according to Embodiment 3 of the present invention configured as described above will be described below.
Also in the third embodiment of the present invention, two images of a short-time exposure signal (short signal) and a long-time exposure signal (long signal) are captured, and an image with an expanded dynamic range is captured by combining these images. The point is the same as in the first and second embodiments of the present invention. However, in the third embodiment of the present invention, the short-time exposure signal (short signal) and the long-time exposure signal (long signal) are individually synthesized by the luminance signal and the signal that is processed as the color signal later. Features.
Therefore, in the third embodiment of the present invention, as in the case of the first embodiment of the present invention, the long signal read from the image memory 6 is viewed as a frame signal in the two horizontal line adding means 701. The long signals of the upper and lower two lines adjacent to are added and mixed. This is a measure for matching the long signal to the short signal because the pixel is mixed on the vertical transfer CCD of the solid-state image pickup device 3.

  In the luminance signal extraction means 70401, as in the second embodiment of the present invention, the signals of two pixels adjacent in the horizontal direction among the outputs of the two horizontal line addition means 701 are sequentially added, and long based on (Equation 8). A luminance component of the signal (hereinafter referred to as a long luminance signal) is extracted.

  For example, when obtaining the long luminance signal YL11 from the long signal (Ye + Mg) L11 and the long signal (Cy + G) L12 shown in FIG. 17, add (Ye + Mg) L11 and (Cy + G) L12. become. Similarly, when obtaining the long luminance signal YL12, (Cy + G) L12 and (Ye + Mg) L13 are added.

  Next, as for the short signal read out from the image memory 6, the luminance signal extraction means 70401 first obtains the luminance signal as in the case of the long signal.

  FIG. 25 shows a long luminance signal, and FIG. 26 shows a short luminance signal.

  Since the short signal is a signal of one field as shown in FIG. 26, the short luminance signal is naturally a luminance signal of one field. Therefore, the means for converting the short luminance signal of one field into a signal of one frame and having the same signal format as the long luminance signal is the luminance signal interpolating means 12.

  Specifically, the luminance signal interpolating means 12 obtains the addition average value of two continuous lines by setting the gain multiplied by the amplifier means 1202 and 1203 shown in FIG. 21 to 0.5, and uses this as an interpolation signal. FIG. 27 shows the short luminance signal after the interpolation processing.

  With the above processing, a luminance signal (long luminance signal) obtained from a long signal for one frame and a luminance signal (short luminance signal) corresponding to one frame obtained through interpolation processing from a short signal for one field. Is generated. The reason for synthesizing the short luminance signal of one frame from the short signal of one field in this way is that when the short signal and the long signal are combined to increase the dynamic range, an image is generated if the short signal remains as a signal of one field. This is because there are not enough horizontal lines to be configured and it cannot be combined with a long signal which is a signal of one frame.

  A means for synthesizing the long luminance signal and the short luminance signal to synthesize a luminance signal with an expanded dynamic range is a luminance signal synthesizing means 13. In the luminance signal synthesizing unit 13, a synthesis coefficient k corresponding to the signal level of each pixel of the long luminance signal is obtained by the synthesis coefficient generating unit 1301 shown in FIG. 22, and the long luminance signal in units of one pixel according to the synthesis coefficient k. Are combined with the short luminance signal existing at the same spatial position on the screen.

  As an example of a method for obtaining the synthesis coefficient k for each pixel from the signal level of the long luminance signal, a method similar to that of Embodiment 2 of the present invention can be considered, and the description thereof is omitted.

  The long luminance signal and the short luminance signal are synthesized by (Equation 14) for each pixel by using the obtained synthesis coefficient k. A signal obtained by combining the long luminance signal and the short luminance signal is defined as a combined luminance signal.

  For example, when the combined luminance signal YM11 is obtained from the long luminance signal YL11 shown in FIG. 28 and the short luminance signal YS11 whose spatial position is the same as YL11, the combination coefficient determined from the long luminance signal (YL11) is k11. Then, synthesis is performed according to (Equation 15).

  Other pixels of the synthesized luminance signal are also obtained from the long luminance signal and the short luminance signal existing at the same spatial position as in (Equation 15).

  The constant D multiplied by the short luminance signal in (Equation 14) and (Equation 15) is the ratio of the exposure amount of the long signal and the short signal (exposure time ratio) and is obtained by (Equation 7).

  In this way, using the long luminance signal and the short luminance signal, the low luminance portion is the long luminance signal, the high luminance portion is the short luminance signal, and the intermediate luminance portion between the low luminance portion and the high luminance portion is the long luminance signal and the short luminance signal. By synthesizing a synthesized luminance signal composed of signals obtained by weighted addition of signals, the dynamic range of the luminance signal of the captured image can be expanded.

  However, among the luminance signals whose dynamic range has been expanded, the portion consisting of the long luminance signal is originally an image signal of one frame, so that the image resolution is high. On the other hand, since the portion composed of the short luminance signal is synthesized from the image signal of one field, the image resolution is lower than that of the portion composed of the long luminance signal. However, in general, shooting conditions that cause the entire screen to have high brightness are rare, and even under such conditions, the entire screen has high brightness because the amount of incident light is limited by narrowing the optical aperture. In fact, it is unlikely that the majority of the photographed image is occupied by the portion composed of the short luminance signal in practical use. In addition, when an image is expressed with a limited gradation, the high luminance part is often assigned a smaller gradation than the low / medium luminance part. For this reason, the resolution degradation of the portion consisting of the short luminance signal is not so conspicuous, and it is considered that even if the long luminance signal and the short luminance signal are synthesized by the above-described method, a synthesized image having a resolution equivalent to the number of pixels of the CCD can be obtained. .

  The above is the processing content related to the dynamic range expansion by the synthesis of the luminance signal. Next, processing relating to color signals will be described.

  The short signal read from the image memory 6 and the long signal obtained by adding the two upper and lower adjacent lines in the two horizontal line adding means 701 are combined in the signal combining means 14 for expanding the dynamic range of the color signal. Is done.

Since the short signal is a single field signal, the signal format is different from the long signal which is a single frame signal. Therefore, similarly to the first embodiment of the present invention, the interpolating means 702 converts one field image into one frame image.
The long signal after the signals of the upper and lower two lines adjacent to each other in the two horizontal line addition means 701 are added and mixed and the short signal subjected to the interpolation processing in the interpolation means 702 are as shown in FIGS. As in the first embodiment of the present invention, the signal formats of the long signal and the short signal are matched by the two horizontal line addition processing for the long signal and the interpolation processing for the short signal.

  As in the second embodiment of the present invention, the long signal whose position is spatially coincident with the long signal and the short signal input to the signal combining unit 14 is combined with the long signal and the short signal in the signal combining unit 14. This is performed for each pixel by the combination coefficient k used when the signal and the short luminance signal are combined and D obtained by (Equation 7). The signal synthesized by the signal synthesizing unit 14 is referred to as a synthesized signal.

  The above is the synthesis process for expanding the dynamic range of the color signal.

  The synthesized signal obtained by the signal synthesizing means 14 is such that a line in which Ye + Mg and Cy + G pixels are arranged in a horizontal direction and a line in which Ye + G and Cy + Mg pixels are arranged in a horizontal direction are in a vertical direction. Since the three primary colors of red, green, and blue are R, G, and B, respectively, because they are repeated in a two-line cycle, from the line where pixels of Ye + Mg and Cy + G are arranged, R The 2R-G color signal having the component is obtained from the line in which Ye + G and Cy + Mg are arranged, and the 2B-G color signal having the B component is obtained from (Equation 17).

  This is so-called color difference line sequential, and for one horizontal line signal, only one of the color signals 2R-G having an R component or 2B-G having a B component can be obtained. Therefore, in order to obtain a signal having both the R component and the B component for one horizontal line signal, the line memories 15, 16, 17, 18 and the synchronization means 19 perform a synchronization process.

  The specific contents of the synchronization processing by the line memories 15, 16, 17, 18 and the synchronization means 19 will be described below. The synchronizing means 19 is inputted with five continuous horizontal line signals from the signal synthesizing means 14 and the line memories 15, 16, 17 and 18. The signal synthesized by the signal synthesizing unit 14 is shown as a synthesized signal in FIG. 29 (a), and the five lines of signals inputted to the synchronizing unit 19 are the signals of the third to seventh lines shown in FIG. 29 (b). Suppose that At this time, it is assumed that the target of the synchronization processing is a horizontal line signal positioned at the center of the input five lines, and if the synchronization processing is performed on the horizontal line signal of the fifth line in FIG. Since 5 lines are signals corresponding to 2R-G having an R component, 2B-G having a B component may be generated by interpolation processing from surrounding horizontal line signals. Therefore, in the synchronization means 19 shown in FIG. 24, the selector 1901 outputs the signal of the fifth line to the output A and the signals corresponding to 2B-G of the third line and the seventh line to the output B and the output C. The gain multiplied by the amplifier means 1902 and 1903 is 0.5, and if the multiplication results are added by the adder 1904, the addition average result of the third line and the seventh line is obtained. The result of the averaging and the signal of the fifth line which is the output of the output A of the selector 1901 are input to the selector 1905, where the output destination is selected, and the horizontal line signal of the fifth line corresponding to 2R-G is the output D. In addition, the addition average result of the third line and the seventh line corresponding to 2B-G is output to the output E. By such an operation, a signal corresponding to 2R-G having an R component and a signal corresponding to 2B-G having a B component can be obtained at a spatial position where the fifth line exists. Similarly, for example, when the signals of the fifth line to the ninth line are input to the synchronization means 19 and the synchronization process is performed on the horizontal line signal of the seventh line, the seventh line has the 2B− with B component. Since it is a signal corresponding to G, 2R-G having an R component may be generated by interpolation processing from surrounding horizontal line signals. Therefore, in the synchronizing means 19 shown in FIG. 24, the selector 1901 outputs the signal of the seventh line to the output A and the signals corresponding to 2R-G of the fifth line and the ninth line to the output B and the output C. The gain multiplied by the amplifier means 1902 and 1903 is 0.5, and if the multiplication results are added by the adder 1904, the addition average result of the fifth line and the ninth line is obtained. The result of the averaging and the signal of the seventh line which is the output A of the selector 1901 are input to the selector 1905, where the output destination is selected and the horizontal line signal of the seventh line corresponding to 2B-G is output to the output E. The addition average result of the fifth line and the ninth line corresponding to 2R−G is output to the output D. By such an operation, a signal corresponding to 2R-G having an R component and a signal corresponding to 2B-G having a B component can be obtained at a spatial position where the seventh line exists. It is assumed that the synchronizer 19 performs selection of input / output signals or the like automatically or under the control of the system controller 21 so that the above-described processing is performed according to the input signal.

  As described above, the synthesized luminance signal synthesized by the luminance signal synthesizing unit 13 and the signal corresponding to 2R-G having the R component obtained by the synchronizing unit 19 and the signal corresponding to 2B-G having the B component. The digital signal processing means 20 performs processing such as noise removal, edge enhancement, gamma correction, matrix calculation, and encoding to a specific format. Since the signal processing in the digital signal means 20 is not directly related to the object of the present invention, a detailed description is omitted.

  As described above, in the solid-state imaging device according to the third embodiment of the present invention, the exposure and signal readout modes of the solid-state imaging device 3 are controlled, and the short-time exposure signal for one field and the long-time exposure signal for one frame. By capturing these images and synthesizing them, it is possible to capture an image with an expanded dynamic range while having a resolution comparable to the number of pixels of the solid-state imaging device. Furthermore, as the solid-state image pickup device used in the solid-state image pickup device, an IT-CCD generally used in a consumer solid-state image pickup device can be used. Therefore, it is necessary to use a plurality of solid-state image pickup devices or special solid-state image pickup devices. Therefore, the apparatus can be configured at low cost.

(Embodiment 4)
In the solid-state imaging device according to the fourth embodiment of the present invention, the thinning means 22 for the output of the two horizontal line adding means 70401 is added to the third embodiment of the present invention shown in FIG. 1 line memories 17 and 18 are deleted, and the configuration and function of the signal synthesis means, synchronization means, digital signal processing means, and system control means are different (in the fourth embodiment of the present invention, the signal synthesis means 23, synchronization). The main difference is that they are numbered and distinguished from the means 24, the digital signal processing means 25, and the system control means 26), and hence the description of the processing contents similar to those of the third embodiment of the present invention will be omitted. Only the differences from the third embodiment of the present invention will be described. FIG. 30 is a block diagram of a solid-state imaging device according to Embodiment 4 of the present invention. In the figure, the thinning means 22 is a means for thinning out the horizontal line signal from the output of the two horizontal line adding means 701 and converting one frame image into one field image. The signal synthesis unit 23 is a unit that synthesizes the outputs of the thinning unit 22 and the image memory 6 based on the synthesis coefficient k obtained by the luminance signal synthesis unit 13. The synchronization unit 24 is a unit that performs a process of synchronizing the output of the signal synthesis unit 23.

  The luminance signal obtained by the luminance signal synthesizing unit 13, the signal having the red (R) component and the signal having the blue (B) component obtained by the synchronizing unit 24 are subjected to noise removal and edge processing in the digital signal processing unit 25. Processing such as emphasis, matrix operation, and encoding to a specific format is performed. It is assumed that the operation modes and operation timings of all the components including these are controlled by the system control means 26 in an integrated manner.

FIG. 31 is a block diagram showing the configuration of the synchronization means 24. As shown in FIG. Reference numerals 2401 and 2402 denote amplifier means for multiplying a signal that has passed through the signal synthesizing means 23 and the one-line memory 16, and the multiplied signals are added by an adder 2403. Reference numeral 2404 denotes a selector that distributes the output of the one-line memory 15 and the output of the adder 2403 to output D and output E, and outputs them. Note that the selection of the signal output destination by the selector 2404 is made according to the color component of the signal, as will be described later.
The operation of the solid-state imaging device according to Embodiment 4 of the present invention configured as described above will be described below.

As described in the third embodiment of the present invention, the output of the two horizontal line adding means 701 is a long signal that is a one-frame image. However, since the short signal stored in the image memory 6 is a one-field image, the long signal and the short signal cannot be combined in the signal combining unit 23 as it is. Therefore, in Embodiment 3 of the present invention, the short signal is converted into a signal of one frame by interpolation processing.

  In Embodiment 4 of the present invention, the color signal does not have a problem in image quality even if it does not have the same amount of information as the luminance signal, and contrary to Embodiment 3 of the present invention, The long signal is converted into a one-field image by performing a thinning process in the vertical direction on the long signal which is one frame image, and is combined with the short signal by the color signal combining unit 24. Specifically, the long signal input to the signal synthesis unit 23 is converted into a one-field image by thinning out even lines of the long signal after addition of two lines as shown in FIG. . The long signal after the thinning has the same format as the short signal as shown in FIG.

  In the signal synthesizing unit 23, the long signal and the short signal which are input one-field images are the same as the third embodiment of the present invention, and the long luminance signal and the short signal whose positions are spatially coincident with these signals. The pixel is synthesized for each pixel by the synthesis coefficient k used when the luminance signal is synthesized and D obtained by (Equation 7). The signal synthesized by the signal synthesizing means 23 is referred to as a synthesized signal.

  Next, the synthesizing signal is subjected to synchronization processing in the synchronizer 24, but unlike the third embodiment of the present invention, the synthesized signal is a one-field signal. As shown in FIG. 32 (b), three lines from the second line to the fourth line are sufficient. Similar to the third embodiment of the present invention, a signal corresponding to 2R-G having an R component and a signal corresponding to 2B-G having a B component can be obtained from these three lines of signals. For example, to obtain a signal corresponding to 2R-G having an R component at the position of the third line and a signal corresponding to 2B-G having a B component, the signals of the second line and the fourth line are added and averaged. Then, a signal corresponding to 2B-G may be synthesized.

  The two signals obtained by the synchronization means 24 are processed in the digital signal processing means 25 in the same manner as in the third embodiment of the present invention, but in the fourth embodiment of the present invention, the synthesis synthesized by the signal synthesizing means 23. Since the signal is a one-field signal, it goes without saying that the digital signal processing means 25 converts it into a frame image if necessary.

As described above, also in the solid-state imaging device according to the fourth embodiment of the present invention, the exposure and signal readout mode of the solid-state imaging device 3 are controlled as in the third embodiment of the present invention to perform short-time exposure for one field. By photographing the signal and the long-time exposure signal for one frame and combining them, it is possible to photograph an image with an expanded dynamic range while having a resolution comparable to the number of pixels of the solid-state imaging device. Furthermore, in Embodiment 4 of the present invention, since the color signal is processed as a field signal, the required number of one-line memories can be reduced, and the apparatus can be configured at a lower cost.
In the first embodiment of the present invention, the short signal is a one-field image read in the field read mode. However, the present invention is not limited to this. For example, a configuration in which the horizontal line signal is thinned out in the vertical direction can be considered. As an example, as shown in FIG. 33, when a short signal is read out from the solid-state imaging device 3, a configuration in which one line signal is read out every three lines in the vertical direction can be considered. In this case, since the short signal is read without mixing the charges accumulated in the two upper and lower photodiodes on the solid-state imaging device, the two horizontal line addition processing for the long signal becomes unnecessary. Further, in the interpolation processing by the interpolation means 702 shown in FIG. 4, it is necessary to perform the interpolation processing so that the number of horizontal lines of the short signal matches the long signal. That is, in the interpolation means 702, horizontal line signals for two lines are created by interpolation processing between the horizontal line signals of the short signal. As a result, the short signal and the long signal have the same signal format and can be combined by the weighted addition means 703 shown in FIG. In this case, the synthesis coefficient k may be obtained by a method as shown in FIG. 13 from the signal level of each pixel of the long signal that is not subjected to addition of the upper and lower two horizontal lines. In addition, in the case of reading out the short signal by thinning out in this way, it is described that the two horizontal line addition processing for the long signal is not necessary, but this is not limited to this, and after performing the two horizontal line addition processing for both the long signal and the short signal, A configuration for performing synthesis processing is also conceivable.

  In the first embodiment of the present invention, the two signals having different exposure amounts are a short signal that is a 1-field image and a long signal that is a 1-frame image, but the present invention is not limited to this. May be a long signal that is one field image and a short signal that is one frame image. In this case, as shown in FIG. 34, the long signal is interpolated by the interpolation means 702 in the vertical direction, and the short signal is added by the two horizontal line adding means 701 to add two adjacent upper and lower lines. What is necessary is just to obtain | require the synthetic | combination coefficient used by the weighting addition means 703 from the long signal after an interpolation process. Further, a configuration is also conceivable in which a synthesis coefficient is obtained from a long signal before interpolation processing. In this case, the corresponding long signal does not exist at the position of the even line of the short signal shown in FIG. Since it cannot be determined, the synthesis coefficient at the even line position of the short signal is determined from the synthesis coefficient obtained from the horizontal line signal of the long signal existing at the same position as the upper and lower lines of the even line of the short signal. That's fine. In this manner, by obtaining a composite signal from a long signal that is a 1-field image and a short signal that is a 1-frame image, a dynamic range expanded image with a high resolution in a high-luminance part can be taken.

  In Embodiment 1 and Embodiment 2 of the present invention, the interpolation unit 702 uses two 1-line memories and performs interpolation processing from signals for two horizontal lines. For example, a configuration is also possible in which interpolation processing is performed from a larger number of horizontal line signals by higher-order interpolation processing using a larger number of one-line memories. Further, as shown in FIG. 35, a configuration is also conceivable in which so-called pre-value interpolation is performed to double the number of horizontal lines by repeatedly outputting one horizontal line input twice.

  In the first embodiment of the present invention, the signal synthesis means 7 obtains the synthesis coefficient k for synthesizing the long signal and the short signal for each pixel of the long signal. However, the present invention is not limited to this. A configuration for obtaining a synthesis coefficient k for each pixel from an average value, minimum value, maximum value, or intermediate value of signal levels of a plurality of pixels, and an average value of k obtained from a plurality of k values obtained for each pixel Alternatively, a configuration in which the minimum value, the maximum value, or the intermediate value is used as a synthesis coefficient for each pixel is also conceivable.

  In the first embodiment of the present invention, the signal synthesizing unit 7 may be configured to perform synthesis by obtaining a synthesis coefficient for a block composed of a plurality of pixels. For example, in FIG. 36, each of the long signals (Ye + Mg) L11 and (Cy + G) L12, (Ye + Mg) L13 and (Cy + G) L14 is one block, and the short signal ( If Ye + Mg) S11 and (Cy + G) S12, (Ye + Mg) S13 and (Cy + G) S14 are each one block, the synthesis coefficient is obtained for each block of 2 pixel units, and synthesis is performed. In this case, for example, a block consisting of a long signal (Ye + Mg) L11 and (Cy + G) L12 and a short signal (Ye + Mg) S11 and (Cy + G) S12 can be combined. When the coefficient is kb11, the calculation is performed as shown in (Equation 18). ((Ye + Mg) M11, (Cy + G) M12 are combined signals)

In this case, the synthesis coefficient kb11 is either a signal level of a long signal (for example, (Ye + Mg) L11 and (Cy + G) L12) included in the block, or a long signal (for example, (Ye + Mg)) included in the block. The k obtained by the method shown in FIG. 13 from at least one of the average value, the maximum value, the minimum value, and the intermediate value of L11 and (Cy + G) L12) may be used as the block synthesis coefficient kb11. Further, an average value, maximum value, minimum value, and intermediate value of k values (for example, k1 and k2 in FIG. 36) for each pixel obtained by the method shown in FIG. 13 from each signal level of the long signal included in the block. A configuration in which any one of these is used as a block synthesis coefficient kb11 is also conceivable. Needless to say, the number of pixels in the block is not limited to two.

Further, in the first embodiment of the present invention, the signal synthesizing means 7 is provided with a block composed of a plurality of pixels, and a long signal level existing at a specific position in the block, for example, the center position of the block, is shown in FIG. A configuration is also conceivable in which the synthesis coefficient obtained by the above method is used for the synthesis processing of each pixel in the block. In this case, it is not necessary to obtain a synthesis coefficient for each pixel, and the processing can be simplified. Note that the position of the pixel used when obtaining the synthesis coefficient need not be limited to the center position of the block.
In the first embodiment of the present invention, the signal synthesizing unit 7 may be configured to obtain the synthesis coefficient k from a short signal converted into a frame image instead of a long signal.
Further, a configuration is also conceivable in which the synthesis coefficient k is obtained from a short signal that is a field image instead of a short signal converted into a frame image. In this case, as can be seen from FIG. 12, since there is no short signal corresponding to the even lines of the long signal, the synthesis coefficient cannot be determined as it is. In this case, the synthesis coefficient at the position corresponding to the even line of the long signal may be obtained from the peripheral short signal or the peripheral synthesis coefficient.

  Moreover, although the example of the method of calculating | requiring the synthetic | combination coefficient k from a signal level in Embodiment 1 of this invention was shown in FIG. 13, the determination method of the synthetic | combination coefficient k is not restricted to this, For example, as shown in FIG. A method of determining k non-linearly according to the luminance level is also conceivable.

  In the second embodiment of the present invention, the short signal is a one-field image read in the field read mode. However, the present invention is not limited to this. For example, as shown in FIG. A configuration is also conceivable in which thinning out and reading out is performed. In this case, since the short signal is read without mixing the charges accumulated in the two upper and lower photodiodes on the solid-state imaging device, the two horizontal line addition processing for the long signal becomes unnecessary. Further, in the interpolation processing by the interpolation means 702 shown in FIG. 4, it is necessary to perform the interpolation processing so that the number of horizontal lines of the short signal matches the long signal. That is, in the interpolation means 702, horizontal line signals for two lines are created by interpolation processing between the horizontal line signals of the short signal. As a result, the short signal and the long signal have the same signal format and can be combined by the weighted addition means 703 shown in FIG. However, since the luminance signal extraction means 70401 shown in FIG. 15 is supplied with a long signal that has not been subjected to addition of two horizontal lines, it is necessary to newly add two horizontal line addition processing for extracting the luminance signal to the means. There is. Alternatively, it is necessary to provide means similar to the two horizontal line addition means 701 in the preceding stage of the luminance signal extraction means 70401 so that the signal obtained by adding the two horizontal lines is supplied to the luminance signal extraction means 70401. In addition, in the case of reading out the short signal by thinning out in this way, it is described that the two horizontal line addition processing for the long signal is not necessary, but this is not limited to this, and after performing the two horizontal line addition processing for both the long signal and the short signal, A configuration for performing synthesis processing is also conceivable.

  In the second embodiment of the present invention, the two signals having different exposure amounts are a short signal that is a 1-field image and a long signal that is a 1-frame image, but the present invention is not limited to this. May be a long signal that is one field image and a short signal that is one frame image. In this case, as shown in FIG. 34, the long signal is interpolated by the interpolation means 702 in the vertical direction, and the short signal is added by the two horizontal line adding means 701 to add two adjacent upper and lower lines. What is necessary is just to obtain | require the synthetic | combination coefficient used by the weighting addition means 703 from the luminance signal extracted from the long signal after interpolation processing. In addition, a configuration is also conceivable in which a synthesis coefficient is obtained from a luminance signal extracted from a long signal before interpolation processing. In this case, a corresponding long signal exists at the position of the even line of the short signal shown in FIG. Therefore, the even number of the short signal is calculated from the combination coefficient obtained from the luminance signal extracted from the horizontal signal of the long signal existing at the same position as the upper and lower lines of the even line of the short signal. What is necessary is just to determine the synthetic | combination coefficient of the position of a line. In this manner, by obtaining a composite signal from a long signal that is a 1-field image and a short signal that is a 1-frame image, a dynamic range expanded image with a high resolution in a high-luminance part can be taken.

  Further, in the second embodiment of the present invention, the signal synthesis means 7 obtains the synthesis coefficient k for synthesizing the long signal and the short signal for each pixel of the long luminance signal. However, the present invention is not limited to this. For example, a configuration for obtaining a synthesis coefficient k for each pixel from an average value, minimum value, maximum value, or intermediate value of luminance signal levels of a plurality of pixels, or k obtained from a plurality of k values obtained for each pixel A configuration in which an average value, a minimum value, a maximum value, or an intermediate value is used as a synthesis coefficient for each pixel is also conceivable.

  In the second embodiment of the present invention, the signal synthesizing unit 7 may be configured to obtain a synthesis coefficient for a block composed of a plurality of pixels and perform synthesis. For example, in FIG. 38, each of the long signals (Ye + Mg) L11 and (Cy + G) L12, (Ye + Mg) L13 and (Cy + G) L14 is one block, and the short signal ( If Ye + Mg) S11 and (Cy + G) S12, (Ye + Mg) S13 and (Cy + G) S14 are each one block, the synthesis coefficient is obtained for each block of 2 pixel units, and synthesis is performed. In this case, for example, a block consisting of a long signal (Ye + Mg) L11 and (Cy + G) L12 and a short signal (Ye + Mg) S11 and (Cy + G) S12 can be combined. When the coefficient is kb11, the calculation is performed as shown in (Equation 18). In this case, the synthesis coefficient kb11 is either the signal level of the long luminance signal corresponding to the block (for example, YL11 and YL12 in FIG. 38), or the average value, maximum value, minimum value, and intermediate value of the long luminance signal corresponding to the block. K determined by the method shown in FIG. 18 from at least one of the values may be used as the block synthesis coefficient kb11. Further, the average value, maximum value, minimum value, and intermediate value of k values (for example, k1 and k2 in FIG. 38) for each pixel obtained by the method shown in FIG. 18 from each signal level of the long luminance signal corresponding to the block. A configuration in which one of the values is the block synthesis coefficient kb11 is also conceivable. Needless to say, the number of pixels in the block is not limited to two.

Further, in the second embodiment of the present invention, the signal synthesizing means 7 is provided with a block composed of a plurality of pixels, and a long luminance signal level corresponding to a specific position in the block, for example, the center position of the block, is shown in FIG. A configuration is also conceivable in which the synthesis coefficient obtained by the method shown is used for the synthesis processing of each pixel in the block. In this case, it is not necessary to obtain a synthesis coefficient for each pixel, and the processing can be simplified. Note that the position of the pixel used when obtaining the synthesis coefficient need not be limited to the center position of the block.
In the second embodiment of the present invention, the signal synthesizing unit 7 may be configured to obtain the synthesis coefficient k from a luminance signal (short luminance signal) extracted from a short signal converted into a frame image instead of a long luminance signal. . Further, a configuration is also conceivable in which the synthesis coefficient k is obtained from the luminance signal extracted from the short signal that is a field image, instead of the luminance signal extracted from the short signal converted into the frame image. In this case, as can be seen from FIG. 12, since there is no short signal corresponding to the even lines of the long signal, the synthesis coefficient cannot be determined as it is. In this case, the synthesis coefficient at the position corresponding to the even line of the long signal may be obtained from the peripheral short luminance signal or the peripheral synthesis coefficient.

Further, although an example of a method for obtaining the synthesis coefficient k from the signal level in Embodiment 2 of the present invention is shown in FIG. 18, the method for determining the synthesis coefficient k is not limited to this. For example, as shown in FIG. A method of determining k non-linearly according to the luminance level is also conceivable.
In the third embodiment of the present invention, the short signal is a one-field image read in the field read mode. However, the present invention is not limited to this. For example, as shown in FIG. A configuration is also conceivable in which thinning out and reading out is performed. In this case, since the short signal is read out without mixing the charges accumulated in the upper and lower photodiodes on the solid-state imaging device, the configuration shown in FIG. In the configuration shown in FIG. 40, the upper and lower two-pixel mixing of the short signal is performed by the two horizontal line addition means 27 (same means as the two horizontal line addition means 701, but the number is 27 for distinction). As a result, functions and effects similar to those of the configuration shown in FIG. 20 can be realized. However, it goes without saying that in the luminance signal interpolating means 12, the content of the interpolation processing changes depending on how the short signal is thinned out. For example, when the short signal is a signal thinned out as shown in FIG. 30, for example, as shown in FIG. 41, two horizontal interpolated horizontal line signals are provided between each horizontal line of the short luminance signal (FIG. 41 (c)). May be created by interpolation. Needless to say, the interpolation unit 702 can also generate the necessary horizontal line signal according to how the short signal is thinned out. In the case of the configuration in which the horizontal line signal is thinned and read out in the vertical direction as shown in FIG. 33, FIG. 40 shows a configuration having two 2-horizontal line addition means of 2-horizontal line addition means 701 and 27. The present invention is not limited to this, and a configuration without the two horizontal line addition means 701 and 27 is also conceivable. In this case, it is possible to extract the luminance from the long signal and the short signal by including a unit having the same effect as the two horizontal line adding units 701 and 27 in the luminance signal extracting unit 70401. Further, in such a configuration without the two horizontal line adding means 701 and 27, the color filter formed on the solid-state image pickup device 3 is composed of primary colors of red (R), green (G), and blue (B), for example. In general, it is also effective in an imaging system that obtains a luminance signal and a color signal without mixing charges on the photodiode of the solid-state imaging device 3.
In the third embodiment and the fourth embodiment of the present invention, the two signals having different exposure amounts are a short signal that is a one-field image and a long signal that is a one-frame image. Alternatively, depending on the use of the solid-state imaging device, a long signal that is one field image and a short signal that is one frame image may be used. In this case, the luminance signal interpolating means 12 performs vertical interpolation processing on the long luminance signal obtained from the long signal, and the luminance signal synthesizing means 13 synthesizes the long luminance and the short luminance signal after the interpolation processing. What is necessary is just to obtain | require the synthetic | combination coefficient used in the case from the long luminance signal after an interpolation process. Further, as in the first embodiment and the second embodiment of the present invention, a configuration is also conceivable in which the synthesis coefficient is obtained from the luminance signal extracted from the long luminance signal before the interpolation processing. In this manner, by obtaining a composite signal from a long signal that is a 1-field image and a short signal that is a 1-frame image, a dynamic range expanded image with a high resolution in a high-luminance part can be taken.
In the third embodiment of the present invention, the interpolation unit 702 uses two 1-line memories and performs interpolation processing from signals for two horizontal lines. However, the present invention is not limited to this. A configuration is also conceivable in which interpolation processing is performed from a larger number of horizontal line signals by high-order interpolation processing using a one-line memory. In addition, a configuration in which so-called pre-value interpolation is performed to double the number of horizontal lines by repeatedly outputting each input horizontal line twice.

  In the third embodiment and the fourth embodiment of the present invention, the luminance signal interpolation means 12 uses the addition average value of the two horizontal line signals as an interpolation signal, but the present invention is not limited to this. A configuration in which interpolation processing is performed from a horizontal line signal by high-order interpolation processing and a configuration in which an interpolation signal is obtained by previous value interpolation are also conceivable.

  In Embodiment 3 and Embodiment 4 of the present invention, the synthesis coefficient k for synthesizing the long luminance signal and the short luminance signal in the luminance signal synthesizing unit 13 is obtained for each pixel of the long luminance signal. However, the present invention is not limited to this. For example, a composition for obtaining the synthesis coefficient k for each pixel from the average value, minimum value, maximum value, or intermediate value of the long luminance signal levels of a plurality of pixels, or the value obtained for each pixel. A configuration in which an average value, a minimum value, a maximum value, or an intermediate value of a plurality of k values among k values is used as a synthesis coefficient for each pixel is also conceivable.

  Further, in Embodiment 3 and Embodiment 4 of the present invention, the luminance signal combining means 13 may be configured to perform combining by obtaining a combination coefficient for a block composed of a plurality of pixels. For example, in FIG. 43, assuming that the long luminance signals YL11 and YL12, YL13 and YL14 are each one block, and the short luminance signals YS11 and YS12 and YS13 and YS14 existing at the same position are one block, this two-pixel unit. It is also possible to obtain a synthesis coefficient for each block and perform synthesis.In this case, for example, a block consisting of long luminance signals YL11 and YL12 and a short luminance signal YS11 and YS12 are synthesized, assuming that the synthesis coefficient of this block is kb11. (Equation 19) (YM is the synthesized luminance signal).

In this case, the synthesis coefficient kb11 is either the signal level of the long luminance signal corresponding to the block (for example, Y11 and Y12 in FIG. 43), or the average value, maximum value, minimum value, and intermediate value of the long luminance signal corresponding to the block. K determined by the method shown in FIG. 18 from at least one of the values may be used as the block synthesis coefficient kb11. Further, the average value, maximum value, minimum value, and intermediate value of k values (for example, k1 and k2 in FIG. 43) for each pixel obtained by the method shown in FIG. 18 from each signal level of the long luminance signal corresponding to the block. A configuration in which one of the values is the block synthesis coefficient kb11 is also conceivable. Needless to say, the number of pixels in the block is not limited to two.

  In Embodiment 3 and Embodiment 4 of the present invention, the luminance signal synthesis means 13 provides a block composed of a plurality of pixels and corresponds to a specific position in this block, for example, the center position of the block. A configuration in which the synthesis coefficient obtained by the method shown in FIG. 18 from the long luminance signal level is used for the synthesis processing of each pixel in the block is also conceivable. In this case, it is not necessary to obtain a synthesis coefficient for each pixel, and the processing can be simplified. Note that the position of the pixel used when obtaining the synthesis coefficient need not be limited to the center position of the block.

  In the third embodiment and the fourth embodiment of the present invention, in the signal synthesis means 14 and 23, the synthesis coefficient k for synthesizing the long signal and the short signal is a synthesis coefficient generation means 1301 from the long luminance signal. However, the present invention is not limited to this. For example, as shown in FIG. 42, a synthesis coefficient generation unit 1404 is uniquely provided inside the signal synthesis units 14 and 23, and a plurality of pixels is obtained. The composition of obtaining the synthesis coefficient k for each pixel from the average value, minimum value, maximum value or intermediate value of the long luminance signal level, or the average value of k obtained from a plurality of k values obtained for each pixel or A configuration in which the minimum value, the maximum value, or the intermediate value is used as a synthesis coefficient for each pixel is also conceivable. Here, the function of the synthesis coefficient generation unit 1404 is the same as that of the synthesis coefficient generation unit 1301.

  In the third embodiment and the fourth embodiment of the present invention, the signal synthesis means 14 and 23 may be configured to perform synthesis by obtaining a synthesis coefficient for a block composed of a plurality of pixels. For example, in FIG. 38, each of the long signals (Ye + Mg) L11 and (Cy + G) L12, (Ye + Mg) L13 and (Cy + G) L14 is one block, and the short signal ( If Ye + Mg) S11 and (Cy + G) S12, (Ye + Mg) S13 and (Cy + G) S14 are each one block, the synthesis coefficient is obtained for each block of 2 pixel units, and synthesis is performed. In this case, for example, a block consisting of a long signal (Ye + Mg) L11 and (Cy + G) L12 and a short signal (Ye + Mg) S11 and (Cy + G) S12 can be combined. When the coefficient is kb11, the calculation is performed as shown in (Expression 18). In this case, the synthesis coefficient kb11 is either a signal level of a long luminance signal (for example, YL11 and YL12 in FIG. 38) that exists at the same spatial position as each block, or a long that exists at the same spatial position as the block. K obtained by the method shown in FIG. 18 from at least one of the average value, the maximum value, the minimum value, and the intermediate value of the luminance signal may be used as the block synthesis coefficient kb11. Further, the average value and the maximum value of k values (for example, k1 and k2 in FIG. 38) for each pixel obtained by the method shown in FIG. 18 from each signal level of the long luminance signal existing spatially at the same position as the block. A configuration in which any one of the value, the minimum value, and the intermediate value is used as the block synthesis coefficient kb11 is also conceivable. Needless to say, the number of pixels in the block is not limited to two.

  In Embodiment 3 and Embodiment 4 of the present invention, the signal synthesizing means 14 and 23 provide a block composed of a plurality of pixels, and a specific position in this block, for example, the center position of the block A configuration is also conceivable in which the synthesis coefficient obtained by the method shown in FIG. 18 from the long luminance signal level existing at the same spatial position is used for the synthesis processing of each pixel in the block. In this case, it is not necessary to obtain a synthesis coefficient for each pixel, and the processing can be simplified. Note that the position of the pixel used when obtaining the synthesis coefficient need not be limited to the center position of the block.

  In the third embodiment and the fourth embodiment of the present invention, the synthesis coefficient k used in the signal synthesis means 14 and 23 is obtained by multiplying the value obtained from the long luminance signal by the above method by a certain coefficient. Alternatively, a configuration in which a constant coefficient is added or subtracted is also conceivable.

  In the third embodiment and the fourth embodiment of the present invention, in the luminance signal synthesis means 13 and the signal synthesis means 14 and 23, the synthesis coefficient k is not a long luminance signal but a short signal converted into a frame image. A configuration obtained from the extracted luminance signal (short luminance signal) is also conceivable. Further, a configuration is also conceivable in which the synthesis coefficient k is obtained from the luminance signal extracted from the short signal that is a field image, instead of the luminance signal extracted from the short signal converted into the frame image. In this case, as can be seen from FIG. 12, since there is no short signal corresponding to the even lines of the long signal, the synthesis coefficient cannot be determined as it is. In this case, the composite coefficient at the position corresponding to the even line of the long signal uses the peripheral short luminance signal or the peripheral composite coefficient as it is, or is obtained from the average value, maximum value, minimum value or intermediate value of the peripheral composite coefficient. Possible methods are considered. In addition, a method is also conceivable in which interpolation processing is used to determine the peripheral composite coefficients and their positional relationships.

Moreover, although the example of the method of calculating | requiring the synthesis coefficient k from the luminance signal level in Embodiment 3 of this invention and Embodiment 4 of this invention was shown in FIG. 18, the determination method of the synthesis coefficient k is not restricted to this. For example, as shown in FIG. 39, a method of determining k non-linearly according to the luminance level is also conceivable.
In the fourth embodiment of the present invention, the short signal is a one-field image read in the field read mode. However, the present invention is not limited to this. For example, as shown in FIG. A configuration is also conceivable in which thinning out and reading out is performed. In this case, since the short signal is read without mixing the charges accumulated in the two upper and lower photodiodes on the solid-state image sensor, the upper and lower two pixels are mixed by the two horizontal line addition means as in FIG. As a result, the same functions and effects as the configuration shown in FIG. 30 can be realized. However, as in the third embodiment of the present invention, it goes without saying that the luminance signal interpolating means 12 changes the content of the interpolation processing depending on how the short signal is thinned out. It goes without saying that the thinning means 22 may also perform thinning so that the long signal has the same signal format as the short signal according to how the short signal is thinned.

Further, in the fourth embodiment of the present invention, the configuration in which the thinning means 22 performs the thinning process of the long signal in the vertical direction has been described. However, as shown in FIG. A configuration is also conceivable in which the direction thinning means 27 is provided so that the horizontal pixels of both the long signal and the short signal that have passed through the two horizontal line addition means 701 are thinned to, for example, 1/2. In this case, if the number of pixels in the horizontal direction is reduced to 1/2 as described above, the 1-line memories 15 and 16 for the synchronization process can be replaced with the half-line 0.5-line memories 28 and 29. Is possible. Thus, by thinning out pixels in the horizontal direction, the configuration of the solid-state imaging device of the present invention can be further simplified and made inexpensive.
In this case, if the horizontal band limitation of the long signal and the short signal is performed before the thinning process in the horizontal direction, unnecessary aliasing does not occur due to the thinning process. Similarly, if band limitation is applied also in the vertical direction, it goes without saying that unnecessary aliasing can be avoided even in thinning processing in the vertical direction.

  In all the embodiments of the present invention, the long signal and the short signal are temporarily stored in the image memory 6. However, the present invention is not limited to this. For example, only one of the long signal and the short signal is stored in the image memory 6. A method of storing and synthesizing the remaining one of the signals from the solid-state imaging device 3 and the signal reading from the image memory 6 may be considered. In this case, the capacity of the image memory 6 can be reduced, and a solid-state imaging device can be configured at a lower cost.

  In all the embodiments of the present invention described above, the color filter array formed on the solid-state imaging device 3 is described using a complementary color checkered type consisting of four colors of magenta, green, yellow, and cyan as shown in FIG. However, the present invention is not limited to this. For example, magenta (Mg) and green (G) as shown in FIG. 45 are arranged so that their positions are not reversed for each line, or green (G) as shown in FIG. In addition, a configuration in which two complementary color filters of cyan (Cy) and yellow (Ye) are arranged in a stripe shape is also conceivable.

  In all the embodiments of the present invention described above, the color filter array formed on the solid-state image pickup device 3 has been described using a configuration comprising four colors of magenta, green, yellow, and cyan as shown in FIG. The configuration using primary color filters consisting of green (G), blue (B), and red (R) is not limited to this, and the Bayer method shown in FIG. 47 is an example of the filter arrangement. 48, the G stripe RB complete checkered method shown in FIG. 49, the stripe method shown in FIG. 50, the diagonal stripe method shown in FIG. 51, and the G stripe RB line sequential method shown in FIG. The G stripe RB point sequential method shown in FIG. 53 can be considered. Needless to say, when the primary color filter is used in this way, the luminance signal is obtained according to (Equation 20).

  In all the embodiments of the present invention described above, the color filter array formed on the solid-state image pickup device 3 is a complementary color checkered type consisting of four colors of magenta, green, yellow, and cyan as shown in FIG. Since the readout of the signal is described as the field readout, the configuration including the upper and lower two horizontal line addition processing of the long signal by the two horizontal line addition means 701 is shown in order to match the signal format of the long signal and the short signal. Instead, when other filter arrangements such as those shown in FIGS. 45 to 53 are employed, or when thinning readout as shown in FIG. 33 is performed instead of field readout, the two horizontal line addition processing is performed. Needless to say, this is not always necessary.

  In all the embodiments of the present invention, the thresholds Th_max, Th_min, Th_max ′, and Th_min ′ for obtaining the synthesis coefficient are set as shown in (Equation 21), and the long exposure signal and the short exposure signal are set. Instead of weighted addition, a configuration in which a certain signal level is switched to a boundary is also conceivable.

1 is a block diagram showing a solid-state imaging device according to Embodiment 1 of the present invention. Explanatory drawing of the signal reading mode from the solid-state image sensor 3 in Embodiment 1 of this invention. The figure which shows the example of the color filter arrangement | positioning formed on the solid-state image sensor 3 in Embodiment 1 of this invention. The block diagram which shows the structure of the signal synthetic | combination means 7 in Embodiment 1 of this invention. The block diagram which shows the structure of the 2 horizontal line addition means 701 in Embodiment 1 of this invention. The block diagram which shows the structure of the interpolation means 702 in Embodiment 1 of this invention. FIG. 2 is a block diagram showing the configuration of weighted addition means 703 in Embodiment 1 of the present invention. Explanatory drawing explaining the principle of dynamic range expansion in Embodiment 1 of this invention Explanatory drawing for demonstrating the exposure and read-out timing of the long signal and short signal in Embodiment 1 of this invention Explanatory drawing for demonstrating the short signal in Embodiment 1 of this invention. Explanatory drawing for demonstrating the long signal in Embodiment 1 of this invention Explanatory drawing for demonstrating 2 horizontal line addition processing and interpolation processing in Embodiment 1 of this invention Graph for explaining a synthesis coefficient determination method in Embodiment 1 of the present invention Explanatory drawing for demonstrating the method of the signal synthesis process in Embodiment 1 of this invention The block diagram which shows the structure of the weighting addition means 704 in Embodiment 2 of this invention. The block diagram which shows the structure of the luminance signal extraction means 70401 in Embodiment 2 of this invention. Explanatory drawing for demonstrating the production method of the long luminance signal in Embodiment 2 of this invention Graph for explaining a synthesis coefficient determination method in Embodiment 2 of the present invention Explanatory drawing for demonstrating the method of the signal synthesis process in Embodiment 2 of this invention c Block diagram showing a solid-state imaging device The block diagram which shows the structure of the luminance signal interpolation means 12 in Embodiment 3 of this invention. The block diagram which shows the structure of the luminance signal synthetic | combination means 13 in Embodiment 3 of this invention. The block diagram which shows the structure of the signal synthetic | combination means 14 in Embodiment 3 of this invention. The block diagram which shows the structure of the synchronization means 19 in Embodiment 3 of this invention. Explanatory drawing for demonstrating the long luminance signal in Embodiment 3 of this invention Explanatory drawing for demonstrating the short luminance signal in Embodiment 3 of this invention. Explanatory drawing for demonstrating the interpolation process of the luminance signal in Embodiment 3 of this invention Explanatory drawing for demonstrating the synthetic | combination method of the luminance signal in Embodiment 3 of this invention. Explanatory drawing for demonstrating the synchronization process by the synchronization means 19 in Embodiment 3 of this invention Block diagram showing a solid-state imaging device according to Embodiment 4 of the present invention The block diagram which shows the structure of the synchronization means 24 in Embodiment 4 of this invention. Explanatory drawing for demonstrating the synchronization process by the synchronization means 24 in Embodiment 3 of this invention Explanatory drawing which shows another example of the image signal read-out method from the solid-state image sensor 3. Explanatory drawing for explaining 2-horizontal line addition processing and interpolation processing when a long signal is a field image and a short signal is a frame image in the first embodiment of the present invention. Explanatory drawing for explaining the previous value interpolation processing Explanatory drawing which shows another example of the synthetic | combination method of the long signal and short signal in Embodiment 1 of this invention. The graph which shows another example of the method of determining a synthetic | combination coefficient from the long signal level in Embodiment 1 of this invention. Explanatory drawing which shows another example of the synthetic | combination method of the long signal and short signal in Embodiment 2 of this invention. The graph which shows another example of the method of determining a synthetic | combination coefficient from the long luminance signal level in Embodiment 2 of this invention. Block diagram of a solid-state imaging device when the short signal readout method is changed in Embodiment 3 of the present invention Explanatory drawing for demonstrating the content of the luminance signal interpolation process at the time of changing the reading method of a short signal in Embodiment 3 of this invention The block diagram which shows another structure of the signal synthetic | combination means 14 in Embodiment 3 of this invention. Explanatory drawing which shows another example of the synthetic | combination method of the long luminance signal and short luminance signal in Embodiment 3 of this invention and Embodiment 4 of this invention. Block diagram showing another example of a solid-state imaging device according to Embodiment 4 of the present invention The figure which shows another example of the color filter arrangement | positioning formed on the solid-state image sensor 3. The figure which shows another example of the color filter arrangement | positioning (CyYeG stripe system) formed on the solid-state image sensor 3 The figure which shows another example of the color filter arrangement | positioning (Bayer system) formed on the solid-state image sensor 3. The figure which shows another example of the color filter arrangement | positioning (interline system) formed on the solid-state image sensor 3. The figure which shows another example of the color filter arrangement | positioning (G stripe RB perfect checkered system) formed on the solid-state image sensor 3 The figure which shows another example of the color filter arrangement | positioning (stripe system) formed on the solid-state image sensor 3. The figure which shows another example of the color filter arrangement | positioning (diagonal stripe system) formed on the solid-state image sensor 3. The figure which shows another example of the color filter arrangement | positioning (G stripe RB line sequential system) formed on the solid-state image sensor 3 The figure which shows another example of the color filter arrangement | positioning (G stripe RB point sequential system) formed on the solid-state image sensor 3

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Optical lens 2 Mechanical shutter 3 Solid-state image sensor 4 Analog signal processing means 5 A / D conversion means 6 Image memory 7 Signal synthesis means 8 Digital signal processing means 9 Shutter drive control means 10 Solid-state image sensor drive control means 11 System control means

Claims (12)

A plurality of photodiodes arranged in a matrix; a solid-state imaging device having a transfer means for outputting the charge accumulated on the photodiodes to the outside; a light shielding means for shielding light incident on the photodiode; Signal synthesizing means for synthesizing image signals output from the solid-state imaging device, and the solid-state imaging device uses a first readout control pulse for only a part of the electric charge accumulated on the photodiode as the first exposure. After the application of the first readout control pulse, the second readout of the charge accumulated on the photodiode in the second exposure completed upon completion of the exposure by the light shielding means. After the control pulse is applied, the signal is output through the transfer unit, and the signal synthesis unit is photographed by the first exposure and the second exposure, A solid-state imaging apparatus characterized by synthesizing the image signal outputted through respectively said transfer means. A plurality of photodiodes arranged in a matrix; a solid-state imaging device having a transfer means for outputting the charge accumulated on the photodiodes to the outside; a light shielding means for shielding light incident on the photodiode; Signal synthesizing means for synthesizing image signals output from the solid-state imaging device, and the solid-state imaging device uses the charge accumulated on the photodiode as the first exposure after applying the first readout control pulse. A second read control is performed to output the charge accumulated on the photodiode in the second exposure which is output through the transfer means by reading and is completed after the exposure by the light shielding means is completed after the application of the first read control pulse. After the pulse is applied, the signal is output through the transfer unit, and the signal synthesis unit performs the first exposure and the second exposure. The image signals captured by the first exposure are combined, and the image signals captured by the first exposure are compared with the image signals captured by the second exposure. A solid-state imaging device characterized in that the number of images is small. Exposure time for the first exposure is a solid-state imaging device according to claim 1 or claim 2, characterized in that controlled by the electronic shutter. In the first and second exposures, the solid-state imaging device according to any one of claims 1 to 3, characterized in that the exposure by varying the exposure amount. 3. The solid-state imaging device according to claim 2 , wherein after the second readout control pulse is applied, the electric charge accumulated in the photodiode in the second exposure is output by frame readout. Image read out after the application of the first read control pulse is compared with an image read out after the application a second read control pulse, claim 1, characterized in that the image having less number of pixels, 3,4 either The solid-state imaging device described in 1. The solid-state imaging device according to claim 6 mechanical shielding means from the claim 1, characterized in that also serves the optical diaphragm. Color filters formed on the solid-state imaging device magenta, green, yellow, solid-state imaging device according to any one of claims 1 to 7, characterized in that the four colors of cyan. A color filter array formed on the solid-state imaging device magenta, green, yellow, solid-state imaging device according to any one of claims 1 to 8, characterized in that the complementary color checkered type consisting of four colors of cyan . Solid color filters red is formed on the imaging device, the green, the solid-state imaging device according to any one of claims 1 to 7, characterized in that the three colors of blue. Solid color filter array formed on the imaging device red, green, solid-state imaging device according to any one of claims 1 to 7, characterized in that the three-color stripe type consisting of three colors of blue. Solid-state image sensor solid-state imaging device according to any one of claims 1 to 11, characterized in that the interline transfer CCD (IT-CCD).

Images