JP2008277894A - Imaging device, imaging method, and integrated circuit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To widen the dynamic range that an imaging device needs to have without lowering resolution and deteriorating the S/N ratio. <P>SOLUTION: In an imaging device 100, a wide-dynamic-range video signal having good S/N ratio and no deterioration in resolution can be obtained by substituting a short signal having inferior S/N ratio in a flat part by a long signal having good S/N ratio, and when the long signal is saturated, substituting by a corrected short signal obtained by adjusting the gain of the short signal. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus, an image pickup method, and an integrated circuit capable of dealing with a moving image and expanding a dynamic range without degradation in resolution by a combination of an image pickup element driving method and signal processing.

2. Description of the Related Art In recent years, imaging devices have been used for various purposes, and there are methods for increasing the S / N ratio of a captured image acquired by the imaging device, increasing the resolution, and increasing the dynamic range depending on the used application. Proposed.
A conventional imaging apparatus having a function of expanding the dynamic range will be described below.
2. Description of the Related Art Conventionally, an image pickup apparatus for expanding a dynamic range is known as disclosed in Japanese Patent Application Laid-Open No. 9-116815. FIG. 4 shows a schematic configuration diagram of an imaging element unit 900 (using a CCD) of this conventional imaging apparatus. The image sensor unit 900 includes a high-sensitivity pixel 23 arranged in the vertical direction, a low-sensitivity pixel 24 arranged in the vertical direction, a vertical CCD 25 that transfers charges vertically, a horizontal CCD 26 that transfers charges horizontally, The limiter 27 clips a predetermined charge amount or more and the charge detection unit 28 having a floating diffusion amplifier configuration. In the image sensor unit 900, the high-sensitivity pixels 23 and the low-sensitivity pixels 24 are alternately arranged in the horizontal direction (lateral direction in FIG. 4). That is, the high sensitivity pixels 23 and the low sensitivity pixels 24 are alternately arranged so that the columns of the high sensitivity pixels 23 and the columns of the low sensitivity pixels 24 form a stripe.

Next, the operation of the conventional imaging device having the imaging element unit 900 will be described. First, the charge obtained by photoelectric conversion by the high sensitivity pixel 23 and the charge obtained by photoelectric conversion by the low sensitivity pixel 24 are obtained. All pixels are transferred to the vertical CCD 25 in synchronization with the vertical synchronization signal. The charges transferred to the vertical CCD 25 are further transferred to the horizontal CCD 26 in synchronization with the horizontal synchronization signal. The charge transferred to the horizontal CCD 26 is clipped by the limiter 27 before being transferred to the charge detector 28 when the charge is higher than a predetermined level. The charge transferred to the horizontal CCD 26 is clipped by the limiter 27 and then transferred to the charge detection unit 28. At this time, the pulse applied to the reset gate RG is generated by adding the charge from the adjacent high-sensitivity pixel and the charge from the low-sensitivity pixel by driving at half the frequency of the drive pulse of the horizontal CCD 26. Output from the image sensor unit 900.

  As shown in FIG. 5, the signal amount (charge amount) obtained (output) from the high-sensitivity pixel 23 and the low-sensitivity pixel 24 increases in direct proportion as the amount of light increases, but is set by the limiter 27. When the predetermined amount TH1 is exceeded, the signal amount becomes a constant value D1, and does not increase any further. On the other hand, in the low sensitivity pixel 24, even if the light amount exceeds the predetermined amount TH1, the signal amount (charge amount) output from the low sensitivity pixel 24 does not exceed the fixed value D1. Therefore, the signal amount (charge amount) output from the low-sensitivity pixel 24 increases in proportion to the light amount even if the light amount increases to a predetermined amount TH1 or more. The charge detection unit 28 adds and mixes two pixel signals, that is, the signal based on the charge acquired by the high sensitivity pixel 23 and the signal based on the charge acquired by the low sensitivity pixel 24. The signal mixed by the charge detection unit 28 (mixed signal) has the characteristics indicated by L1 in FIG. As can be seen from the light quantity-signal quantity characteristic shown in L1 of FIG. 5, the mixed signal is obtained as a signal having a wide dynamic range without saturation of the output signal quantity even when the light quantity is large. .

FIG. 6 is a timing chart for explaining the operation of the image pickup apparatus disclosed in Japanese Patent Laid-Open No. 9-200261.
FIG. 6 shows drive pulses for the image pickup device (image pickup device using CCD) of the image pickup apparatus.
In this image pickup apparatus, the image pickup device is operated with the readout gate pulse VG, the vertical register transfer pulse VS, and the horizontal register transfer pulse VH being doubled with respect to the vertical synchronization pulse VD.
As shown in FIG. 6, in this imaging apparatus, the frame is divided into two front and rear, the signal is taken out with a long charge accumulation time (corresponding to the normal accumulation time in FIG. 6) in the first half frame, and the short charge is taken in the second half frame. A signal is extracted at the accumulation time (corresponding to the shutter accumulation time in FIG. 6), and then both signals are combined. According to this method, when the amount of light increases in the first half of the frame, the output of the signal due to the charge acquired by the image sensor is saturated, but in the second half, it is not saturated, so there is no saturation by combining these two types of signals. A signal can be obtained. Therefore, by using this method, the dynamic range of the imaging apparatus can be expanded.
JP-A-9-116815 Japanese Patent Laid-Open No. 9-200261

However, the imaging apparatus disclosed in Japanese Patent Laid-Open No. 9-116815 has a problem that the resolution deteriorates because two pixels are added to form a signal of one pixel.
In the imaging device disclosed in Japanese Patent Laid-Open No. 9-200261, an image signal for one screen is acquired twice in one frame period while changing the charge accumulation time, and the two acquired image signals are synthesized. Therefore, it is difficult to increase the speed. That is, in this imaging apparatus, only one screen image signal can be acquired within one frame period, and N screen image signals (N is a natural number of N> 1) are acquired within one frame period. It is difficult to realize such processing. In addition, in this imaging apparatus, when a moving moving image is processed within one frame period, the timings of two charge accumulation times for accumulating charges in the imaging element are different, so that the two acquired image signals are misaligned. When an image formed by combining the two acquired image signals is displayed on a display device, there is a problem that an image (video) on the display screen is displaced.

  The present invention solves the above-described problems, and can output a video signal with no deterioration in resolution, and it can be applied to a moving image that changes drastically within a predetermined period (for example, a frame period or a field period). However, an object of the present invention is to provide an imaging device, an imaging method, and an integrated circuit that can suppress the occurrence of video (image) blurring.

  The first invention includes an imaging unit, a charge accumulation time setting unit, a drive unit, an L / S separation unit, a saturation detection unit, a correction value calculation unit, a multiplication unit, a first interpolation unit, The imaging apparatus includes a first selection unit, a second interpolation unit, a difference level detection unit, a first selection unit, a third interpolation unit, a second selection unit, and an L / S synthesis unit. The imaging unit is an imaging device including a plurality of pixels that can set different charge accumulation times for each pixel, and includes a first pixel group that is configured of pixels and accumulates charges during the first charge accumulation time. And an image sensor including a second pixel group that is configured of pixels and accumulates charges in a second charge accumulation time that is longer than the first charge accumulation time, and converts light from the subject into an electrical signal. Get video signal. The charge accumulation time setting unit sets a first charge accumulation time and a second charge accumulation time. The drive unit drives the imaging unit based on the first charge accumulation time and the second charge accumulation time. The L / S separation unit outputs a video signal output from the imaging unit to a short signal that is a video signal acquired by the first charge accumulation time and a long signal that is a video signal acquired by the second charge accumulation time. To separate. The saturation detector detects the signal level of the long signal. The correction value calculation unit calculates a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time. The multiplication unit obtains a corrected short signal by multiplying the short signal by the correction value calculated by the correction value calculation unit. The first interpolation unit interpolates the corrected short signal to obtain an interpolated long signal that is a signal having the same timing as the long signal. The first selection unit selects the long signal when the saturation detection unit determines that the signal level of the long signal is equal to or lower than a predetermined value. When the signal level of the long signal exceeds the predetermined value by the saturation detection unit, If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. The second interpolation unit interpolates the long signal to obtain an interpolated short signal that is a signal having the same timing as the corrected short signal. The difference level detection unit detects a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal. The first selection unit selects the long signal when the saturation detection unit determines that the signal level of the long signal is equal to or lower than a predetermined value. When the signal level of the long signal exceeds the predetermined value by the saturation detection unit, If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. The third interpolation unit interpolates the output of the saturation detection unit and obtains the saturation detection short signal at the same timing as the short signal. The second selection unit selects the corrected short signal when it is determined by the saturation detection short signal that the long signal exceeds the predetermined level, and the long signal does not exceed the predetermined level by the saturation detection short signal. If it is determined that the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is selected, When it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level, the difference level detection unit causes the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal to exceed the predetermined value. If it is determined that the interpolation short signal is not selected, the interpolation short signal is selected, and the selected short signal is set as the final short signal. To output. The L / S synthesis unit generates an output video signal by sequentially switching the final long signal output from the first selection unit and the final short signal output from the second selection unit.

In this imaging apparatus, a video signal output from an imaging unit having an imaging element including a first pixel group that accumulates charges during a first charge accumulation time and a second pixel group that accumulates charges during a second charge accumulation time. From this, a long signal and a short signal are acquired by the L / S separator. Further, the saturation detection unit detects whether or not the long signal is saturated. For the portion where the long signal is saturated, the short signal is replaced with an interpolated long signal interpolated to the signal level of the long signal to obtain the final long. The signal is output by the first selection unit as a signal. Further, when the second selection unit determines that the long signal exceeds a predetermined level based on the saturation detection short signal, the modified short signal is selected, and the long signal is predetermined based on the saturation detection short signal. If it is determined that the level does not exceed the level, and the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds the predetermined value, the correction is performed. When the short signal is selected and it is determined that the long signal does not exceed the predetermined level based on the saturation detection short signal, the signal level of the corrected short signal and the signal of the interpolated short signal are detected by the difference level detection unit. When it is determined that the level and the difference do not exceed the predetermined value, the interpolation short signal is selected and the selected No. is output as a final short signal. Further, the L / S synthesis unit sequentially outputs the modified long signal and the modified short signal output from the multiplication unit, and outputs the result as an output video signal.
That is, in this imaging device, a long signal having a long charge accumulation time and a good S / N ratio is used as much as possible, and if a long signal is used, only a portion where the quality of the output signal from the imaging device may be deteriorated is short. The process of using a signal is performed. Therefore, this image pickup apparatus can output a video signal that does not deteriorate in resolution, and even a moving image that changes drastically within a predetermined period (for example, a frame period or a field period). ) It is possible to suppress the occurrence of blurring and to output an image with a good S / N ratio.

2nd invention is 1st invention, Comprising: An image pick-up element is comprised from the pixel of several rows and several columns, a 1st pixel group is comprised from the pixel of an odd number column, and a 2nd pixel group is an even number Composed of columns of pixels,
Thereby, the video signal acquired from the accumulated charges in the odd-numbered columns of pixels can be acquired as a short signal, and the video signal acquired from the accumulated charges of the even-numbered pixels can be acquired as a long signal.

3rd invention is 1st invention, Comprising: An image pick-up element is comprised from the pixel of several rows and several columns, a 1st pixel group is comprised from the pixel of an even number column, and a 2nd pixel group is an odd number Consists of columns of pixels.
As a result, the video signal acquired from the accumulated charge in the even-numbered column pixels can be acquired as a short signal, and the video signal acquired from the accumulated charge in the odd-numbered column pixels can be acquired as the long signal.

4th invention is 1st invention, Comprising: An image pick-up element is comprised from the pixel of multiple rows and multiple columns, a 1st pixel group is comprised from the pixel of odd-numbered rows, and a 2nd pixel group is an even number Consists of rows of pixels.
As a result, the video signal acquired from the accumulated charges in the odd-numbered pixels can be acquired as a short signal, and the video signal acquired from the accumulated charges in the even-numbered pixels can be acquired as a long signal.

5th invention is 1st invention, Comprising: An image pick-up element is comprised from the pixel of multiple rows and multiple columns, a 1st pixel group is comprised from the pixel of an even-numbered row, and a 2nd pixel group is an odd number Consists of rows of pixels.
As a result, the video signal acquired from the accumulated charges in the even-numbered pixels can be acquired as a short signal, and the video signal acquired from the accumulated charges in the odd-numbered pixels can be acquired as a long signal.
A sixth invention is any one of the first to fifth inventions, and the imaging device is a CMOS image sensor.
Thereby, an imaging apparatus is realizable by using a CMOS type image sensor as an image pick-up element of an image pick-up part.

A seventh aspect of the invention is an image pickup device that includes a plurality of pixels that can set different charge accumulation times for each pixel, and is a first pixel that is composed of pixels and accumulates charges during the first charge accumulation time. And an image sensor including a pixel group and a second pixel group configured to store charges in a second charge storage time that is longer than the first charge storage time, and converts light from the subject into an electrical signal. An imaging method used in an imaging apparatus including an imaging unit that acquires a video signal in this manner, a charge accumulation time setting step, a driving step, an L / S separation step, a saturation detection step, and a correction value calculation step, , Multiplication step, first interpolation step, first selection step, second interpolation step, difference level detection step, first selection step, third interpolation step, second selection step, L We include a S synthesis step. In the charge accumulation time setting step, a first charge accumulation time and a second charge accumulation time are set. In the driving step, the imaging unit is driven based on the first charge accumulation time and the second charge accumulation time. In the L / S separation step, the video signal output from the imaging unit includes a short signal that is a video signal acquired by the first charge accumulation time, and a long signal that is a video signal acquired by the second charge accumulation time. To separate. In the saturation detection step, the signal level of the long signal is detected. In the correction value calculation step, a correction value for correcting the signal level of the short signal to the signal level of the long signal is calculated based on the first charge accumulation time and the second charge accumulation time. In the multiplication step, the corrected short signal is obtained by multiplying the short signal by the correction value calculated in the correction value calculating step. In the first interpolation step, an interpolated long signal that is a signal having the same timing as the long signal is obtained by interpolating the corrected short signal. In the first selection step, when it is determined by the saturation detection step that the signal level of the long signal is below a predetermined value, the long signal is selected, and when the signal level of the long signal exceeds the predetermined value by the saturation detection unit If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. In the second interpolation step, a long signal is interpolated to obtain an interpolated short signal that is a signal having the same timing as the corrected short signal. In the difference level detection step, a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal is detected. In the first selection step, when it is determined by the saturation detection step that the signal level of the long signal is below a predetermined value, the long signal is selected, and when the signal level of the long signal exceeds the predetermined value by the saturation detection unit If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. In the third interpolation step, an output signal indicating the detection result in the saturation detection step is interpolated to obtain a saturation detection short signal having the same timing as the short signal. In the second selection step, when it is determined by the saturation detection short signal that the long signal exceeds the predetermined level, the corrected short signal is selected, and the long signal does not exceed the predetermined level by the saturation detection short signal. If it is determined that the difference level detection step determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is selected, When it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level, the difference level detection step causes the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal to exceed the predetermined value. If it is determined that it is not, the interpolation short signal is selected, and the selected signal is And outputs it as a short signal. In the L / S synthesis step, an output video signal is generated by sequentially switching the final long signal output from the first selection unit and the final short signal output from the second selection unit.
Thereby, it is possible to realize an imaging method having the same effect as that of the first invention.

An eighth aspect of the invention is an image pickup device that includes a plurality of pixels that can set different charge accumulation times for each pixel, and is a first pixel that is composed of pixels and accumulates charges during the first charge accumulation time. And an image sensor including a pixel group and a second pixel group configured to store charges in a second charge storage time that is longer than the first charge storage time, and converts light from the subject into an electrical signal. An integrated circuit used together with an imaging unit that acquires a video signal, a charge accumulation time setting unit, a drive unit, an L / S separation unit, a saturation detection unit, a correction value calculation unit, a multiplication unit, A first interpolation unit, a first selection unit, a second interpolation unit, a difference level detection unit, a first selection unit, a third interpolation unit, a second selection unit, an L / S synthesis unit, Is provided. The charge accumulation time setting unit sets a first charge accumulation time and a second charge accumulation time. The drive unit drives the imaging unit based on the first charge accumulation time and the second charge accumulation time. The L / S separation unit outputs a video signal output from the imaging unit to a short signal that is a video signal acquired by the first charge accumulation time and a long signal that is a video signal acquired by the second charge accumulation time. To separate. The saturation detector detects the signal level of the long signal. The correction value calculation unit calculates a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time. The multiplication unit obtains a corrected short signal by multiplying the short signal by the correction value calculated by the correction value calculation unit. The first interpolation unit interpolates the corrected short signal to obtain an interpolated long signal that is a signal having the same timing as the long signal. The first selection unit selects the long signal when the saturation detection unit determines that the signal level of the long signal is equal to or lower than a predetermined value. When the signal level of the long signal exceeds the predetermined value by the saturation detection unit, If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. The second interpolation unit interpolates the long signal to obtain an interpolated short signal that is a signal having the same timing as the corrected short signal. The difference level detection unit detects a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal. The first selection unit selects the long signal when the saturation detection unit determines that the signal level of the long signal is equal to or lower than a predetermined value. When the signal level of the long signal exceeds the predetermined value by the saturation detection unit, If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. The third interpolation unit interpolates the output of the saturation detection unit and obtains the saturation detection short signal at the same timing as the short signal. The second selection unit selects the corrected short signal when it is determined by the saturation detection short signal that the long signal exceeds the predetermined level, and the long signal does not exceed the predetermined level by the saturation detection short signal. If it is determined that the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is selected, When it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level, the difference level detection unit causes the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal to exceed the predetermined value. If it is determined that the interpolation short signal is not selected, the interpolation short signal is selected, and the selected short signal is set as the final short signal. To output. The L / S synthesis unit generates an output video signal by sequentially switching the final long signal output from the first selection unit and the final short signal output from the second selection unit.
By using this integrated circuit together with the imaging unit, an integrated circuit that exhibits the same effect as that of the first invention can be realized.

The ninth invention includes an imaging unit, a charge accumulation time setting unit, a drive unit, an L / S separation unit, a saturation detection unit, a correction value calculation unit, a multiplication unit, a first interpolation unit, 1 is an integrated circuit including a first selection unit, a second interpolation unit, a difference level detection unit, a first selection unit, a third interpolation unit, a second selection unit, and an L / S synthesis unit. The imaging unit is an imaging device including a plurality of pixels that can set different charge accumulation times for each pixel, and includes a first pixel group that is configured of pixels and accumulates charges during the first charge accumulation time. And an image sensor including a second pixel group that is configured of pixels and accumulates charges in a second charge accumulation time that is longer than the first charge accumulation time, and converts light from the subject into an electrical signal. Get video signal. The charge accumulation time setting unit sets a first charge accumulation time and a second charge accumulation time. The drive unit drives the imaging unit based on the first charge accumulation time and the second charge accumulation time. The L / S separation unit outputs a video signal output from the imaging unit to a short signal that is a video signal acquired by the first charge accumulation time and a long signal that is a video signal acquired by the second charge accumulation time. To separate. The saturation detector detects the signal level of the long signal. The correction value calculation unit calculates a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time. The multiplication unit obtains a corrected short signal by multiplying the short signal by the correction value calculated by the correction value calculation unit. The first interpolation unit interpolates the corrected short signal to obtain an interpolated long signal that is a signal having the same timing as the long signal. The first selection unit selects the long signal when the saturation detection unit determines that the signal level of the long signal is equal to or lower than a predetermined value. When the signal level of the long signal exceeds the predetermined value by the saturation detection unit, If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. The second interpolation unit interpolates the long signal to obtain an interpolated short signal that is a signal having the same timing as the corrected short signal. The difference level detection unit detects a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal. The first selection unit selects the long signal when the saturation detection unit determines that the signal level of the long signal is equal to or lower than a predetermined value. When the signal level of the long signal exceeds the predetermined value by the saturation detection unit, If it is determined, the interpolated long signal is selected, and the selected signal is acquired as the final long signal. The third interpolation unit interpolates the output of the saturation detection unit and obtains the saturation detection short signal at the same timing as the short signal. The second selection unit selects the corrected short signal when it is determined by the saturation detection short signal that the long signal exceeds the predetermined level, and the long signal does not exceed the predetermined level by the saturation detection short signal. If it is determined that the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is selected, When it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level, the difference level detection unit causes the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal to exceed the predetermined value. If it is determined that the interpolation short signal is not selected, the interpolation short signal is selected, and the selected short signal is set as the final short signal. To output. The L / S synthesis unit generates an output video signal by sequentially switching the final long signal output from the first selection unit and the final short signal output from the second selection unit.
Thus, an integrated circuit that exhibits the same effect as that of the first invention can be realized.

  According to the present invention, a video (image) can be output even for a moving image that can output a video signal with no degradation in resolution and changes drastically within a predetermined period (for example, a frame period or a field period). It is possible to realize an imaging device, an imaging method, and an integrated circuit that can suppress the occurrence of blurring and can output an image with a good S / N ratio.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
<Configuration of imaging device>
FIG. 1 shows a configuration diagram of an imaging apparatus 100 according to the first embodiment of the present invention.
The imaging apparatus 100 includes an imaging unit 1, a driving unit 2 for driving the imaging unit 1, an L / S separation unit 3 that separates a video signal acquired by the imaging unit 1 according to the length of charge accumulation time, and charge accumulation. A saturation detection unit 4 that detects whether the level of a long signal obtained from a pixel having a long time exceeds a predetermined level, a correction value calculation unit 5 that corrects a pixel output level that varies depending on the charge accumulation time, and a short signal Is used to correct the signal drop due to the charge accumulation time to obtain a corrected short signal, and using the corrected short signal, an intermediate point of the corrected short signal, that is, an interpolated long signal having the same timing as the long signal A first interpolation unit 7 for obtaining Furthermore, the imaging apparatus 100 selects either the long signal or the interpolated long signal based on the detection result in the saturation detection unit 4, and uses the long signal and the first selection unit 8 that outputs the final long signal. A second interpolation unit 9 for obtaining an intermediate point between the corrected long signals, that is, an interpolated short signal having the same timing as the short signal, and the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal is detected. The difference level detection unit 10 that detects whether or not the predetermined value is exceeded, the third interpolation unit 11 that adjusts the timing of the output signal from the saturation detection unit 4, the output signal from the difference level detection unit 10, and the third Based on the output signal from the interpolating unit 11, either the interpolated short signal or the modified short signal is selected and output as the final short signal. A selector 12, an L / S synthesis unit 13 that obtains a video signal by sequentially switching a final long signal and a final short signal, and a process unit that performs camera signal processing such as gamma correction and DTL (detail) addition on the video signal 14 and a timing generator unit 15 for generating a driving pulse signal and the like for determining a synchronization signal of the camera and a driving timing of the imaging unit 1.

The imaging unit 1 acquires light from the subject as an electrical signal by photoelectric conversion, and outputs the acquired electrical signal to the L / S separation unit 3.
FIG. 2 shows a configuration diagram of the imaging unit 1 (in the case of a CMOS image sensor) and the driving unit 2 of the imaging apparatus of the present invention. In FIG. 2, the imaging unit 1 includes 6 pixels horizontally and 4 pixels vertically, and each pixel 16 includes a photodiode. Needless to say, the number of pixels is for convenience of explanation and is not limited to this number of pixels.
As shown in FIG. 2, the imaging unit 1 includes a plurality of pixels 16 that are arranged vertically and horizontally, a switch transistor 20, and an output amplifier 21. The pixel 16 is an element that accumulates charges proportional to the amount of incident light and outputs an electrical signal corresponding to the amount of accumulated charges. As the imaging unit 1, a CMOS type image sensor is preferably used.

  The driving unit 2 drives the imaging unit 1 based on a control signal output from the timing generator unit 15 (such as a camera synchronization signal or a driving pulse signal for determining the driving timing of the imaging unit 1). As shown in FIG. 2, the drive unit 2 includes a first vertical register / shutter 17, a second vertical register / shutter 18, and a horizontal register 19. The first vertical register shutter 17 is connected to the pixels 16 in the odd-numbered columns, selects a horizontal line address, operates the pixel 16 at the selected horizontal address for a specified time, and accumulates charges. (So-called electronic shutter function is realized). The second vertical register shutter 18 is connected to the even number of pixels 16 and selects a horizontal line address, operates the pixel 16 at the selected horizontal address for a specified time, and accumulates charges. (So-called electronic shutter function is realized).

  As the first vertical register / shutter 17 and the second vertical register / shutter 18, for example, a shift register circuit can be used. The horizontal register 19 is connected to the switch transistors 20 provided for the number of columns formed by the pixels 16 as shown in FIG. 2, and selects a vertical line address (performs horizontal scanning). The horizontal register 19 selects the vertical line address by turning on the switch transistor 20 corresponding to the vertical line address. The output amplifier 21 is connected to the switch transistor 20 as shown in FIG. When an electrical signal corresponding to the amount of charge accumulated in the pixel 16 at the address selected by the first vertical register / shutter 17, the second vertical register / shutter 18, and the horizontal register 19 is input to the output amplifier 21, an output is made. The amplifier 21 amplifies the input signal and outputs it to the L / S separator 3 via the output terminal 22.

Since the imaging unit 1 and the driving unit 2 configured in this manner can individually control the charge accumulation time of the pixel 16 corresponding to each pixel, the charge accumulation when acquiring an image signal for one screen. Time can be adjusted easily. For example, it is possible to easily shorten the charge accumulation time of the pixels 16 arranged in the odd columns and lengthen the charge accumulation time of the pixels 16 arranged in the even columns.
Hereinafter, for convenience of explanation, a case where the charge accumulation time of the pixels 16 arranged in the odd columns is shortened and the charge accumulation time of the pixels 16 arranged in the even columns is lengthened will be described.
The L / S separation unit 3 separates the signal output from the imaging unit 1 according to the length of the charge accumulation time of the pixel 16. When the output signal from the imaging unit 1 when the charge accumulation time of the pixel 16 is long is a long signal, and the output signal from the imaging unit 1 when the charge accumulation time of the pixel 16 is short is a short signal, the L / S separation unit 3 outputs the long signal to the saturation detection unit 4, the second interpolation unit 9, and the first selection unit 8, and outputs the short signal to the multiplication unit 6. Further, the L / S separator 3 can receive a control signal from the timing generator 15, and the L / S separator 3 outputs a long signal and a short signal based on the control signal from the timing generator 15. To do.

The saturation detection unit 4 receives the long signal output from the L / S separation unit 3 and detects whether or not the level of the long signal obtained from the pixel 16 having a long charge accumulation time exceeds a predetermined level. The result is output to the first selection unit 8.
The correction value calculation unit 5 calculates a correction value for correcting the pixel output level (the output level of the electric signal output from the pixel 16) that varies depending on the charge accumulation time. The correction value is calculated based on the charge accumulation time of the pixel 16 for generating the long signal and the charge accumulation time of the pixel 16 for generating the short signal. The correction value calculation unit 5 acquires information on the charge accumulation time of the pixel 16 for generating the long signal and the charge accumulation time of the pixel 16 for generating the short signal from the timing generator unit 15, and includes the information in the information. Based on this, a correction value is calculated, and the calculated correction value is output to the multiplication unit 6.

The multiplication unit 6 multiplies the correction value output from the correction value calculation unit 5 by the short signal output from the L / S separation unit 3, and uses the signal obtained by the multiplication as a modified short signal. 1 is output to the interpolation unit 7, the difference level detection unit 10, and the second selection unit 12.
The first interpolation unit 7 generates an interpolation long signal having the same timing as the intermediate point of the corrected short signal, that is, the long signal, using the corrected short signal output from the multiplication unit 6, and the generated interpolation long signal is first selected. Output to unit 8. For example, the first interpolation unit 7 performs an averaging process (arithmetic averaging, geometric averaging, LPF process) on the two corrected short signals before and after and generates an interpolated long signal.
The first selection unit 8 receives the detection result from the saturation detection unit 4, the long signal output from the L / S separation unit 3, and the interpolation long signal output from the first interpolation unit 7, and receives the saturation detection unit 4. When the long signal is determined to be equal to or lower than the predetermined level, the long signal is output as the final long signal to the L / S synthesis unit 9 and the saturation detection unit 4 determines that the long signal exceeds the predetermined level. In this case, the interpolated long signal is output as a final long signal to the L / S synthesis unit 9.

The second interpolation unit 9 uses the long signal output from the L / S separation unit 3 to generate an interpolated short signal having the same timing as the intermediate point of the long signal, that is, the corrected short signal, and the generated interpolated short signal. 2 is output to the selection unit 12. For example, the second interpolation unit 9 performs an averaging process (arithmetic averaging, geometric averaging or LPF process) on the two long signals before and after, and generates an interpolation short signal.
The difference level detection unit 10 detects the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal, detects whether the difference exceeds a predetermined value, and sends the detection result to the second selection unit 12. Output.
The third interpolation unit 11 matches the output signal from the saturation detection unit 4 with the timings of the interpolation short signal and the modified short signal, and outputs them to the second selection unit 12.

The second selection unit 12 receives the output signal from the difference level detection unit 10 and the output signal from the third interpolation unit 11 and outputs the final short signal to the L / S synthesis unit 13.
That is, when the third interpolation unit determines that the long signal exceeds the predetermined level, the second selection unit 12 selects the corrected short signal and outputs the corrected short signal as the final short signal. The second selection unit 12 is a case where the third interpolation unit determines that the long signal does not exceed the predetermined level, and the difference level detection unit 10 determines the signal level of the corrected short signal and the interpolation short signal. When it is determined that the difference between the signal level and the signal level exceeds a predetermined value, the corrected short signal is selected, and the corrected short signal is output as the final short signal. When the third interpolation unit determines that the long signal does not exceed the predetermined level, the difference level detection unit 10 determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal is predetermined. When it is determined that the value does not exceed the value, the interpolation short signal is selected, and the interpolation short signal is output as the final short signal.

The L / S synthesis unit 13 receives the final long signal output from the first selection unit 8, the final short signal output from the second selection unit 12, and the control signal from the timing generator unit 15 as inputs. Based on the control signal from the timing generator unit 15, the final long signal and the final short signal are sequentially switched and output to the process unit 14 as a video signal.
The process unit 14 performs signal processing for the camera such as gamma correction and addition of DTL (detail) on the video signal output from the L / S synthesis unit 13.
The timing generator unit 15 generates a synchronization signal of the camera, a driving pulse signal for determining the driving timing of the imaging unit 1, and the like, and outputs them to the driving unit 2. Further, the timing generator unit 15 outputs a control signal for adjusting the timing of the output signal to the L / S separation unit 3 and the L / S synthesis unit 13. Further, the timing generator unit 15 outputs information regarding the charge accumulation time of the pixel 16 for generating the long signal and the charge accumulation time of the pixel 16 for generating the short signal to the correction value calculation unit 5. The function of the charge accumulation time setting unit is realized by the timing generator unit 15.

<Operation of imaging device>
The operation of the imaging apparatus 100 configured as described above will be described with reference to FIGS.
FIG. 3 is a timing chart showing signal waveforms at points (points a to l) of the imaging apparatus 100 shown in FIG. The vertical axis in FIG. 3 is the signal level, and the horizontal axis is time.
First, in the imaging unit 1 (CMOS type image sensor) of FIG. 2, the charge accumulation time for odd-numbered columns of pixels is set to be long, and the charge accumulation time for even-numbered columns of pixels is set to be short. This setting is performed by setting the first vertical register / shutter 17 and the second vertical register / shutter 18. In the imaging unit 1, the first vertical register / shutter 17 and the second vertical register / shutter 18 can easily perform different driving for each pixel, so that the shutter speed can also be set for each pixel. Such pixel-by-pixel control is difficult when a CCD is used as the image sensor.

The horizontal register 19 turns on the switch transistors 20 arranged in the horizontal direction for a signal with a long charge accumulation time acquired from the pixels in the odd columns and a signal with a short charge accumulation time acquired from the pixels in the even columns. As a result, the signal is output to the output terminal 22 via the output amplifier 21. An example of an output signal of the imaging unit 1 (CMOS type image sensor) is shown in FIG.
In FIG. 3, the signal of the portion with the odd number shown in the uppermost row indicates a long signal with a long charge accumulation time, and the signal with the even number indicates a short signal with a short charge accumulation time. . An output signal from the imaging unit 1 is separated into a long signal and a short signal by the L / S separation unit 3. FIG. 3B shows a long signal separated by the L / S separator 3, and FIG. 3C shows a short signal separated by the L / S separator 3.

The short signal output from the L / S separation unit 3 is the correction coefficient calculated by the correction value calculation unit 5 at the multiplication unit 6 (= (longer charge accumulation time) / (shorter charge accumulation time)). Is converted into a corrected short signal. FIG. 3E shows a corrected short signal.
Next, the corrected short signal is interpolated by the first interpolation unit 7 and converted into a signal having the same timing as the long signal output from the L / S separation unit 3. This converted signal is an interpolated long signal. FIG. 3F shows the interpolated long signal. Here, the first interpolation unit 7 generates an interpolated long signal by, for example, generating a signal obtained by averaging two corrected short signals before and after, and matching the timing of the generated signal with the long signal. Specifically, a signal obtained by averaging the corrected short signal of the part (2) in FIG. 3 and the corrected short signal of the part (4) is generated, and the timing of the signal is set to the long of the part (3). By making it coincide with the signal, an interpolated long signal is created. In addition, a signal generated by simply delaying the modified short signal and matching the timing with the long signal may be used as the interpolated long signal.

  The long signal output from the L / S separation unit 3 and the interpolated long signal output from the first interpolation unit 7 by the first selection unit 8 are detected by the detection result of the saturation detection unit 4 (the signal shown in FIG. 3 (d3)). Is equivalent to this), the final long signal is generated. This will be described with reference to FIG. In FIG. 3, the signal level indicated by the dotted line in FIG. 3 will be described as a predetermined value. In the portions (1), (3), (5), (7), and (9) where the long signal does not exceed the predetermined value, the long signal is selected by the first selection unit 8 and the long signal is final. It is output as a long signal. In the portions (11), (13), and (15) where the long signal exceeds a predetermined value, the interpolated long signal is selected by the first selection unit 8, and the interpolated long signal is output as a modified long signal. FIG. 3 (g) shows this final long signal.

  Next, the long signal is interpolated by the second interpolation unit 9 and converted into a signal having the same timing as the modified short signal output from the multiplication unit 6. This converted signal is an interpolation short signal. FIG. 3 (h) shows this interpolation short signal. Here, for example, the second interpolation unit 9 generates a signal obtained by averaging the two front and rear long signals, and generates an interpolation short signal by matching the timing of the generated signal with the modified short signal. Specifically, a signal obtained by averaging the long signal of the part (1) in FIG. 3 and the long signal of the part (3) is generated, and the timing of the signal is changed to the modified short signal of the part (2). To create an interpolated short signal. In addition, a signal generated by simply delaying (delaying) the long signal and matching the timing with the corrected short signal may be used as the interpolated short signal.

The corrected short signal and the interpolation short are input to the difference level detection unit 10 and the second selection unit 12.
The difference level detector 10 detects the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal, and detects whether the difference exceeds a predetermined value. Then, a signal indicating the detection result in the difference level detection unit 10 is output to the second selection unit 12.
Further, the output signal from the saturation detection unit 4 is adjusted by the third interpolation unit so as to coincide with the timings of the interpolation short signal and the modified short signal, and is output to the second selection unit 12. The output signal from the third interpolation unit is shown in FIG.
When the third interpolation unit determines that the long signal exceeds the predetermined level, the second selection unit 12 selects the corrected short signal and outputs the corrected short signal as the final short signal. In addition, when the third interpolation unit determines that the long signal does not exceed the predetermined level, the difference level detection unit 10 determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal is predetermined. When it is determined that the value exceeds the value, the second selection unit 12 selects the corrected short signal, and the corrected short signal is output as the final short signal. When the third interpolation unit determines that the long signal does not exceed the predetermined level, the difference level detection unit 10 determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal is predetermined. When it is determined that the value does not exceed the value, the second selection unit 12 selects the interpolation short signal and outputs the interpolation short signal as the final short signal. FIG. 3 (i) shows the output signal of the difference level detector 10, and FIG. 3 (k) shows the final short signal.

Finally, a signal synthesized by sequentially taking out the final short signal and the final long signal by the L / S synthesis unit 13 is output from the L / S synthesis unit 13 as a video signal. FIG. 3 (l) shows this video signal. As can be seen from FIG. 3 (l), in the output signal from the imaging unit 1, the signals of the portions (11), (13) and (15) where the signal level is saturated are reproduced without being saturated. Wide dynamic range has been realized. That is, in the imaging apparatus 100, since the signals of all the pixels of the imaging unit 1 are reproduced without being saturated, a video signal with high resolution can be obtained.
Further, in the imaging apparatus 100 of the present invention, when a long signal having a long charge accumulation time and a good S / N ratio is used as much as possible, and the long signal is used, the quality of the output signal from the imaging apparatus may be degraded. Only the short signal is used. That is, the video signal acquired by the imaging apparatus 100 of the present invention has a good S / N ratio. Furthermore, even when an image with intense movement within one frame (or field) period is processed by the imaging apparatus 100 of the present invention, the occurrence of image blur is effectively suppressed while maintaining the effect of expanding the dynamic range. Can do.

In the above description, the imaging apparatus 100 configured to separate the processing channels of the long signal and the short signal has been described. However, the long signal and the short signal are processed in time series (time division processing) by digital processing. The processing channel may not be separated.
Further, in the imaging unit 1, the configuration in which pixels with long and short charge accumulation times are alternately arranged in the horizontal direction has been described. However, the same configuration may be adopted in which pixels with long and short charge accumulation times are alternately arranged in the vertical direction. It is clear that an effect can be obtained.
In the case where the present invention is applied to an imaging apparatus configured with RGB separate imaging units (CMOS type image sensors), such as a three-plate camera apparatus, the imaging apparatus 100 performs the processing of the imaging apparatus 100 shown in FIG. What is necessary is just to set it as the structure which provides a system | strain separately 3 RGB.

Further, although the imaging apparatus 100 of the present invention is configured to correct the level of the short signal, it goes without saying that the same effect can be obtained by correcting the level of the long signal.
[Other Embodiments]
In the above embodiment, the case where processing is performed based on a frame has been described. However, processing may be performed based on a field.
In the imaging device described in the above embodiment, each block may be individually made into one chip by a semiconductor device such as an LSI, or may be made into one chip so as to include a part or the whole.
Here, although LSI is used, it may be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Further, the method of circuit integration is not limited to LSI, and implementation with a dedicated circuit or a general-purpose processor is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied as a possibility.
Moreover, each process of the said embodiment may be implement | achieved by hardware, and may be implement | achieved by software. Further, it may be realized by mixed processing of software and hardware.

  The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the scope of the invention.

  The imaging apparatus, imaging method, and integrated circuit according to the present invention can increase the dynamic range without degrading the resolution and without degrading the S / N ratio, and for tunnels and security that require a high dynamic range. It is useful as an imaging device, an imaging method, and an integrated circuit.

1 is a configuration diagram of an imaging apparatus 100 according to a first embodiment of the present invention. 1 is a configuration diagram of an imaging unit 1 and a drive unit 2 according to a first embodiment of the present invention. Waveform diagram for explaining the operation of the imaging apparatus according to the first embodiment Configuration diagram of conventional imaging device Explanatory diagram of dynamic range expansion of imaging device Operation waveform diagram of conventional imaging device

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Imaging part 2 Drive part 3 L / S separation part 4 Saturation detection part 5 Correction value calculation part 6 Multiplication part 7 1st interpolation part 8 1st selection part 9 2nd interpolation part 10 Difference level detection part 11 3rd Interpolation unit 12 Second selection unit 13 L / S synthesis unit 14 Process unit 15 Timing generator unit 16 Pixel 17 First vertical register / shutter 18 Second vertical register / shutter 19 Horizontal register /
20 Switch transistor 21 Output amplifier 22 Output terminal

Claims (9)

An image sensor comprising a plurality of pixels capable of setting different charge accumulation times for each pixel, the first pixel group comprising the pixels and accumulating charges in a first charge accumulation time, and the pixels And an image sensor that includes a second pixel group that accumulates charges in a second charge accumulation time that is longer than the first charge accumulation time, and converts the light from the subject into an electrical signal to generate an image. An imaging unit for acquiring a signal;
A charge accumulation time setting unit for setting the first charge accumulation time and the second charge accumulation time;
A drive unit that drives the imaging unit based on the first charge accumulation time and the second charge accumulation time;
The video signal output from the imaging unit is converted into a short signal that is the video signal acquired by the first charge accumulation time and a long signal that is the video signal acquired by the second charge accumulation time. An L / S separator to separate;
A saturation detector for detecting a signal level of the long signal;
A correction value calculation unit that calculates a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time;
A multiplier for obtaining a modified short signal by multiplying the short signal by the correction value calculated by the correction value calculator;
A first interpolation unit that interpolates the modified short signal and obtains an interpolated long signal that is a signal having the same timing as the long signal;
When the saturation detection unit determines that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value. A first selection unit that selects the interpolated long signal and obtains the selected signal as a final long signal;
A second interpolation unit that interpolates the long signal to obtain an interpolated short signal that is a signal having the same timing as the modified short signal;
A difference level detector for detecting a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal;
When the saturation detection unit determines that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value. A first selection unit that selects the interpolated long signal and obtains the selected signal as a final long signal;
A third interpolation unit for interpolating the output of the saturation detection unit to obtain a saturation detection short signal having the same timing as the short signal;
If it is determined by the saturation detection short signal that the long signal exceeds a predetermined level, the modified short signal is selected, and it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level. If the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is And when the saturation detection short signal determines that the long signal does not exceed a predetermined level, the difference level detection unit detects the signal level of the corrected short signal and the signal of the interpolated short signal. When it is determined that the level and the difference do not exceed the predetermined value, the interpolation short signal is transmitted. Select a second selection unit for outputting the selected signal as the final short signal,
An L / S synthesis unit that generates an output video signal by sequentially switching a final long signal output from the first selection unit and a final short signal output from the second selection unit;
An imaging apparatus comprising: The image sensor is configured by the pixels in a plurality of rows and columns, the first pixel group is configured by the pixels in an odd number column, and the second pixel group is configured by the pixels in an even number column.
The imaging device according to claim 1. The imaging device is configured by a plurality of rows and a plurality of columns of the pixels, the first pixel group is configured by the even columns of the pixels, and the second pixel group is configured by an odd columns of the pixels.
The imaging device according to claim 1. The imaging device is configured by the pixels in a plurality of rows and columns, the first pixel group is configured by the pixels in odd rows, and the second pixel group is configured by the pixels in even rows.
The imaging device according to claim 1. The imaging device is configured by the pixels in a plurality of rows and columns, the first pixel group is configured by the pixels in even rows, and the second pixel group is configured by the pixels in odd rows.
The imaging device according to claim 1. The image sensor is a CMOS image sensor.
The imaging device according to claim 1. An image sensor comprising a plurality of pixels capable of setting different charge accumulation times for each pixel, the first pixel group comprising the pixels and accumulating charges in a first charge accumulation time, and the pixels And an image sensor that includes a second pixel group that accumulates charges in a second charge accumulation time that is longer than the first charge accumulation time, and converts the light from the subject into an electrical signal to generate an image. An imaging method used in an imaging device including an imaging unit that acquires a signal,
A charge accumulation time setting step for setting the first charge accumulation time and the second charge accumulation time;
A driving step of driving the imaging unit based on the first charge accumulation time and the second charge accumulation time;
The video signal output from the imaging unit is converted into a short signal that is the video signal acquired by the first charge accumulation time and a long signal that is the video signal acquired by the second charge accumulation time. An L / S separation step to separate;
A saturation detection step of detecting a signal level of the long signal;
A correction value calculating step for calculating a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time;
A multiplication step of obtaining a modified short signal by multiplying the short signal by the correction value calculated in the correction value calculating step;
A first interpolation step of interpolating the modified short signal to obtain an interpolated long signal that is a signal having the same timing as the long signal;
When it is determined by the saturation detection step that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value If so, a first selection step of selecting the interpolated long signal and obtaining the selected signal as a final long signal;
A second interpolation step of interpolating the long signal to obtain an interpolated short signal that is a signal having the same timing as the modified short signal;
A difference level detection step of detecting a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal;
When it is determined by the saturation detection step that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value If so, a first selection step of selecting the interpolated long signal and obtaining the selected signal as a final long signal;
A third interpolation step of interpolating an output signal indicating a detection result in the saturation detection step to obtain a saturation detection short signal having the same timing as the short signal;
If it is determined by the saturation detection short signal that the long signal exceeds a predetermined level, the modified short signal is selected, and it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level. When the difference level detection step determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is And when it is determined by the saturation detection short signal that the long signal does not exceed a predetermined level, the signal level of the corrected short signal and the signal of the interpolated short signal are determined by the difference level detection step. When it is determined that the level and difference do not exceed the predetermined value, the interpolation Select hort signal, a second selection step of outputting the selected signal as the final short signal,
An L / S synthesis step of generating an output video signal by sequentially switching a final long signal output from the first selection unit and a final short signal output from the second selection unit;
An imaging method comprising: An image sensor comprising a plurality of pixels capable of setting different charge accumulation times for each pixel, the first pixel group comprising the pixels and accumulating charges in a first charge accumulation time, and the pixels And an image sensor that includes a second pixel group that accumulates charges in a second charge accumulation time that is longer than the first charge accumulation time, and converts the light from the subject into an electrical signal to generate an image. An integrated circuit used together with an imaging unit for acquiring a signal,
A charge accumulation time setting unit for setting the first charge accumulation time and the second charge accumulation time;
A drive unit that drives the imaging unit based on the first charge accumulation time and the second charge accumulation time;
The video signal output from the imaging unit is converted into a short signal that is the video signal acquired by the first charge accumulation time and a long signal that is the video signal acquired by the second charge accumulation time. An L / S separator to separate;
A saturation detector for detecting a signal level of the long signal;
A correction value calculation unit that calculates a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time;
A multiplier for obtaining a modified short signal by multiplying the short signal by the correction value calculated by the correction value calculator;
A first interpolation unit that interpolates the modified short signal and obtains an interpolated long signal that is a signal having the same timing as the long signal;
When the saturation detection unit determines that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value. A first selection unit that selects the interpolated long signal and obtains the selected signal as a final long signal;
A second interpolation unit that interpolates the long signal to obtain an interpolated short signal that is a signal having the same timing as the modified short signal;
A difference level detector for detecting a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal;
When the saturation detection unit determines that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value. A first selection unit that selects the interpolated long signal and obtains the selected signal as a final long signal;
A third interpolation unit for interpolating the output of the saturation detection unit to obtain a saturation detection short signal having the same timing as the short signal;
If it is determined by the saturation detection short signal that the long signal exceeds a predetermined level, the modified short signal is selected, and it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level. If the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is And when the saturation detection short signal determines that the long signal does not exceed a predetermined level, the difference level detection unit detects the signal level of the corrected short signal and the signal of the interpolated short signal. When it is determined that the level and the difference do not exceed the predetermined value, the interpolation short signal is transmitted. Select a second selection unit for outputting the selected signal as the final short signal,
An L / S synthesis unit that generates an output video signal by sequentially switching a final long signal output from the first selection unit and a final short signal output from the second selection unit;
An integrated circuit comprising: An image sensor comprising a plurality of pixels capable of setting different charge accumulation times for each pixel, the first pixel group comprising the pixels and accumulating charges in a first charge accumulation time, and the pixels And an image sensor that includes a second pixel group that accumulates charges in a second charge accumulation time that is longer than the first charge accumulation time, and converts the light from the subject into an electrical signal to generate an image. An imaging unit for acquiring a signal;
A charge accumulation time setting unit for setting the first charge accumulation time and the second charge accumulation time;
A drive unit that drives the imaging unit based on the first charge accumulation time and the second charge accumulation time;
The video signal output from the imaging unit is converted into a short signal that is the video signal acquired by the first charge accumulation time and a long signal that is the video signal acquired by the second charge accumulation time. An L / S separator to separate;
A saturation detector for detecting a signal level of the long signal;
A correction value calculation unit that calculates a correction value for correcting the signal level of the short signal to the signal level of the long signal based on the first charge accumulation time and the second charge accumulation time;
A multiplier for obtaining a modified short signal by multiplying the short signal by the correction value calculated by the correction value calculator;
A first interpolation unit that interpolates the modified short signal and obtains an interpolated long signal that is a signal having the same timing as the long signal;
When the saturation detection unit determines that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value. A first selection unit that selects the interpolated long signal and obtains the selected signal as a final long signal;
A second interpolation unit that interpolates the long signal to obtain an interpolated short signal that is a signal having the same timing as the modified short signal;
A difference level detector for detecting a difference between the signal level of the corrected short signal and the signal level of the interpolated short signal;
When the saturation detection unit determines that the signal level of the long signal is less than or equal to a predetermined value, the long signal is selected, and the saturation detection unit determines that the signal level of the long signal exceeds a predetermined value. A first selection unit that selects the interpolated long signal and obtains the selected signal as a final long signal;
A third interpolation unit for interpolating the output of the saturation detection unit to obtain a saturation detection short signal having the same timing as the short signal;
If it is determined by the saturation detection short signal that the long signal exceeds a predetermined level, the modified short signal is selected, and it is determined by the saturation detection short signal that the long signal does not exceed the predetermined level. If the difference level detection unit determines that the difference between the signal level of the corrected short signal and the signal level of the interpolated short signal exceeds a predetermined value, the corrected short signal is And when the saturation detection short signal determines that the long signal does not exceed a predetermined level, the difference level detection unit detects the signal level of the corrected short signal and the signal of the interpolated short signal. When it is determined that the level and the difference do not exceed the predetermined value, the interpolation short signal is transmitted. Select a second selection unit for outputting the selected signal as the final short signal,
An L / S synthesis unit that generates an output video signal by sequentially switching a final long signal output from the first selection unit and a final short signal output from the second selection unit;
An integrated circuit comprising:

Images