JP2006005898A - Time correlation detection type image sensor and image analyzing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform various kinds of image analysis which have been conventionally disabled because of a limitation of a time resolution. <P>SOLUTION: A plurality of photodetection elements are provided, each photodetection element including a photodiode FD for converting an input photon into current, a transistor couple for modulating the current generated by the photodiode FD in accordance with an inputted external electric signal S and a capacitor C for time-integrating the current modulated by the transistor couple, and the plurality of photodetection elements are scanned to sequentially read integration values of the capacitors C. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a time correlation detection type sensor that performs time correlation detection between incident light intensity and an external electrical signal, and an image analysis method using the time correlation detection type sensor.

  Applications of image sensing and image analysis are diverse, such as computer vision, automatic visual inspection, scientific measurement, and remote sensing. However, as input means, there are almost no methods other than diversion of image sensors for broadcasting, and because of its limited functions, there are many restrictions on application development of image processing at present. is there.

  Particularly problematic is the poor temporal resolution of a frame rate of 1/30 seconds. Even if a high-speed scan of a MOS image sensor or a time signal output type device such as an image detector is used, the sensitivity and original space Since resolution is sacrificed, it is not an essential solution.

  In order to solve this problem, the present invention seeks to provide a solution based on considerations returned to the image sensing methodology. Originally, a request to increase both spatial resolution and temporal resolution results in an enormous amount of information, and thus has an inherent difficulty. The present invention employs a method of applying appropriate processing at the sensor stage and compressing and outputting information necessary and sufficient for subsequent image analysis. Further, it is possible to obtain a technical effect that various image analyzes are possible.

(1) The time correlation detection type image sensor according to the first aspect of the invention corresponds to the image sensor as shown in FIG. 4, and includes a photoelectric conversion element that converts input photons into current, and an input modulation signal. A plurality of light-receiving elements each having a plurality of transistors for shunting and modulating the current generated by the photoelectric conversion elements according to the above, and a plurality of capacitors for respectively time-integrating the currents modulated by the plurality of transistors;
And a scanning circuit that scans a plurality of light receiving elements at a period later than that of the modulation signal, sequentially reads and resets charges of the plurality of capacitors.
(2) The time correlation detection type image sensor according to claim 2 corresponds to the image sensor as shown in FIG. 6, and is a photoelectric conversion element that converts input photons into current, and is applied as a modulation signal. A photoelectric conversion element that diverts and modulates the generated current according to an external electric field, and a plurality of light receiving elements each having a plurality of capacitors that time-integrate the diverted current;
And a scanning circuit that scans a plurality of light receiving elements at a period later than that of the modulation signal, sequentially reads and resets charges of the plurality of capacitors.
(3) The invention according to claim 3 is the time-correlated image sensor according to claim 1 or 2, wherein the object to be measured is irradiated with modulated light and the reflected light from the object to be measured is received. The modulation signal is inputted so that the sum of the integral values of the two represents the average illuminance of the object to be measured, and the difference between the integral values of the plurality of capacitors becomes the first derivative value of the object to be measured.
(4) The image analysis apparatus according to claim 4 receives the light source that irradiates the object with time-modulated light, the delay means that delays the time-modulated signal of the light source, and the reflected light from the object. The time-correlated image sensor according to any one of claims 1 to 3, wherein each of the charges of the plurality of capacitors obtained by inputting the signal delayed by the delay means to the time-correlated image sensor as a modulation signal is provided. It is characterized in that the shape of the object is specified by calculating the sum and difference.
(5) A time correlation detection type image sensor according to claim 5 corresponds to the image sensor as shown in FIG. 5, and is a photoelectric conversion element that converts input photons into current, and according to an input modulation signal. A plurality of light-receiving elements each having a plurality of transistors for shunting and modulating current generated by the photoelectric conversion elements, and a plurality of capacitors for time-integrating currents modulated by the plurality of transistors, respectively;
And a scanning circuit that scans a plurality of light receiving elements with a period later than that of the modulation signal, sequentially reads and resets the difference in charge of the plurality of capacitors.
(6) A time correlation detection type image sensor according to claim 6 corresponds to the image sensor as shown in FIGS. 5 and 6 and is a photoelectric conversion element that converts input photons into current, and includes a modulation signal. A photoelectric conversion element that shunts and modulates the generated current according to an external electric field applied as a plurality of light receiving elements each having a plurality of capacitors that respectively time-integrate the shunt current;
And a scanning circuit that scans a plurality of light receiving elements with a period later than that of the modulation signal, sequentially reads and resets the difference in charge of the plurality of capacitors.
(7) An image analysis apparatus according to claim 7 includes a light source that irradiates an object with time-modulated light;
7. A time correlation type image sensor comprising: delay means for delaying a time modulation signal of a light source; and a time correlation type image sensor according to claim 5 for receiving reflected light from an object. The difference between the charges of the plurality of capacitors obtained by inputting the signal as a modulation signal is read as a first-order differential image of the object by a scanning circuit.

(1) According to the first and second aspects of the invention, when converting input photons into current, the generated current is shunted and modulated in accordance with the input electric signal, and the modulation-controlled current is converted. A light receiving element having a plurality of capacitors each for time integration or when converting input photons into current, the current generated according to the external electric field applied as a modulation signal is shunted and modulated, and the shunt current is respectively Since it has multiple light receiving elements with multiple capacitors for time integration and scans multiple light receiving elements with a period slower than the modulation signal, the integrated signals are read sequentially, so time correlation with high time resolution Detection can be performed.
(2) According to the invention of claim 3, the time-correlated image sensor according to claim 1 or 2, wherein the object to be measured is irradiated with modulated light and the reflected light from the object to be measured is received. The sum of the integral values of the capacitors represents the average illuminance of the object to be measured, and the modulation signal is input so that the difference between the integral values of the capacitors is the first derivative of the object to be measured. An image sensor can be used for an image analysis apparatus.
(3) According to the image analyzing apparatus of claim 4, the light source that irradiates the object with time-modulated light, the delay means that delays the time-modulated signal of the light source, and the reflected light from the object are received. Each of the plurality of capacitors obtained by inputting the signal delayed by the delay means as a modulation signal to the time correlated image sensor. Since the sum and difference of charges are calculated, the shape of the object can be easily specified using the output signal of the image sensor.
(4) In the inventions according to claims 5 to 7, since the difference between the charges of the plurality of capacitors of the light receiving element in claims 1 and 2 is sequentially read out and reset, an image sensor that is not easily saturated can be obtained. Measurement accuracy is improved.

  Hereinafter, an embodiment of a time correlation detection type image sensor and an image analysis method according to the present invention will be described with reference to FIGS.

  A basic structure of a time correlation image sensor according to the present invention is shown in FIG. The main points of this structure are (a) a photodiode detector FD that converts incident photons into photocurrent (multiplicand), (b) an external electric signal (multiplier) S that is commonly supplied to all pixels, c) a current mode multiplier 1 for generating a current proportional to the product between the photocurrent and the electrical signal; (d) a capacitor C for integrating the product current as a correlation value by time integration; e) A scanning unit (CCD or MOS switch circuit) 2 for scanning the correlation value and outputting it as a normal video signal is provided.

と表されるものとなる。 If the output of the photodetector of the pixel (i, j) is f _{i, j} (t), the external electric signal is g (t), and the integration time corresponding to the scanning period is T, (i , J) The pixel output is

It will be represented.

FIG. 2 shows the structure (operation) of the time correlation detection type sensor according to the present invention as seen from the frequency band. With varying light such as modulated light, f _{i, j} (t) can assume any high time frequency band. In a normal image sensor, this band is severely limited by the scanning period (frame rate). In the above sensor structure, this upper limit is removed up to the limit band of the photodetector. Since the external electric signal S is common to all pixels, this band is not limited by the scanning cycle, and the upper limit band is estimated to be about the limit band of the signal distribution wiring. On the other hand, since the output signal is a correlation value of these and an integral value for the scanning period, the band is always within the band of the scanning period.

-Generation of distance images based on light propagation time-
So far, correlation measurement in image sensing has been discussed mainly with respect to spatial correlation-pattern matching-and correlation with respect to the time axis has hardly been discussed. Here, a few specific examples are used to show what kind of application development can be opened for image sensing when very fast time correlation is obtained. As a form of the spatial resolution and output signal, it is assumed that it is positively compatible with a normal broadcasting image sensor. If the complexity of the circuit is suppressed to about 10 transistors / pixel, it can be a target up to a spatial resolution of about high definition.

  First, generation of a distance image based on light propagation time will be described with reference to FIG. This application becomes possible when a band of several GHz or more can be realized for both the photodetector and the external electric signal input. It should be noted that with a time resolution of several GHz, the propagation length resolution of light is several cm or less required for three-dimensional measurement.

を満たすものである。 The light source is assumed to be a broadband (including high frequency component) time modulation such as a laser diode or LED. Here, the LED 3 is used. This LED 3 does not emit light at a constant level, but emits light by modulating the luminance of light emission to an M-sequence pattern. Let this modulated light be M (t). That is,

It satisfies.

  The modulation signal of the LED 3 is delayed by a delay time τ through the delay device 6 and input to the sensor 5 as an external electric signal. In addition, incident light captured by each light receiving element of the sensor 5 is modulated by an external electric signal, and a charge corresponding to a product of the incident light and the external electric signal is accumulated as a correlation value. By sequentially scanning each element, the accumulated electric charge of each element is read out, whereby a correlation is obtained.

ただし、光の減衰等は簡略化のため無視してある。また、対象４をセンサ５上に結像させる結像光学系の図示は省略している。ここで、ｆ_i （ｘ，ｙ）は距離ｌ_i のところにある画像、ｎ（ｘ，ｙ，ｔ）は照明光や太陽光などの観測雑音である。この強度分布Ｇ（ｘ，ｙ，ｔ）と変調信号Ｍ（ｔ）の時間相関関数は、

であり、外部信号として与えるＭ（ｔ）の時間遅れがτ＝ｌ_i ／ｃの時だけ、ｆ_i （ｘ，ｙ）の出力が得られるということである。すなわち、適当にτを選択することによって、外来光によらず所望の距離の物体４表面のみを明るく検出することができる。したがって、例えばτを段階的に変化させることにより、物体４の立体像が得られる。 Here, the sum of the distance from the LED 3 to the object 4 and the distance from the object 4 to the sensor 5 is l _i (i = 1, 2,... N), and the speed of light is c. However, it is assumed that the object 4 is stationary. The light intensity distribution on the sensor 5 is generally written as follows.

However, light attenuation and the like are ignored for simplification. An imaging optical system for imaging the object 4 on the sensor 5 is not shown. Here, f _i (x, y) is an image at a distance l _i , and n (x, y, t) is observation noise such as illumination light or sunlight. The time correlation function between the intensity distribution G (x, y, t) and the modulation signal M (t) is

The output of f _i (x, y) is obtained only when the time delay of M (t) given as an external signal is τ = l _i / c. That is, by appropriately selecting τ, it is possible to brightly detect only the surface of the object 4 at a desired distance regardless of external light. Therefore, for example, a three-dimensional image of the object 4 is obtained by changing τ stepwise.

-Calculation of first derivative of image-
As a method of giving a modulation component to the intensity of incident light, there is a method of minutely vibrating the sensor. For the application described below, it is sufficient that the time frequency band of the photodetector and the external electric signal is 100 kHz or less. Further, generating the differentiation without using the inter-pixel difference has an advantage that an error due to a difference in pixel sensitivity which is the largest noise source of the image sensor can be eliminated.

となり、テーラー展開をして１次近似すると、

である。ここで、式（６）にεｃｏｓωｔを乗じて時間平均をとると、

となる。ここで、＜ｃｏｓωｔ＞＝０、＜ｓｉｎωｔｃｏｓωｔ＞＝０、＜ｃｏｓ² ωｔ＞＝１／２を用いた。式（７）の左辺はセンサ上に相関値として生成され、εも既知であるから、センサ出力として画像のｘ微分ｆ_0x（ｘ，ｙ）が読み出されることになる。同様にして、ｆ_0y（ｘ，ｙ）も式（６）にεｓｉｎωｔを掛けて時間平均をとることにより求まる。 The light intensity distribution that will arrive on the sensor when the sensor is stationary is _denoted by f ₀ (x, y). The sensor is given a rotational shift motion with a small radius ε and a rotational angular velocity ω, that is, a parallel motion with a circle of radius ε as a locus in the light receiving surface of the sensor. We assume that the subject's motion is negligible (if any) compared to the sensor's motion speed. At this time, the light intensity distribution captured by the sensor at time t is

Then, when Taylor expansion is performed and first order approximation,

It is. Here, by multiplying equation (6) by εcosωt and taking the time average,

It becomes. Here, <cos ωt> = 0, <sin ωt cos ωt> = 0, and <cos ² ωt> = 1/2 were used. Since the left side of equation (7) is generated as a correlation value on the sensor and ε is also known, the x derivative f _0x (x, y) of the image is read as the sensor output. Similarly, f _0y (x, y) is also obtained by multiplying equation (6) by ε sin ωt and taking the time average.

This means that the x differential f _0x (x, y) of the image is read by using the x-direction component of the rotational shift amount as an external electric signal. Similarly, by using the y-direction component of the rotational shift amount as an external electric signal, the y derivative f _0y (x, y) of the image is read out. If the constant is read as an external signal, a moving average image by rotational shift can be obtained.

  Obtaining the first derivative with high accuracy leads to a significant improvement in the accuracy of various image processing (edge detection, optical flow detection, binocular stereo, etc.) using gradient vectors, not just reducing the amount of calculation processing. . Instead of vibrating the sensor itself, the imaging may be moved (vibrated) on the fixed sensor by, for example, vibrating a part of the optical system.

-Sensing cell structure of time-correlated image sensor-
Next, a specific configuration of the sensing cell of the time correlation type image sensor will be exemplified.

A variable conductance differential amplifier is known as a typical circuit configuration for realizing multiplication. FIG. 4 shows a configuration method using this. Although bipolar transistors Q ₁ and Q ₂ are used in FIG. 4, MOSFET pairs biased in the subthreshold region may be used as differential transistor pairs.

に対する相互コンダクタンスｇ_m は、トランジスタＱ₁ ，Ｑ₂ のエミッタバイアス電流を与える光電流Ｉ_PD（ｔ）に比例する。したがって、この比例係数をρとすると、両コレクタ電流は、

となり、コンデンサＣ₁ ，Ｃ₂ に蓄積される電荷には、

のように、Ｉ_PD （ｔ）とＶ（ｔ）の相関に比例した成分と、Ｉ_PD の積分値に比例する成分との和が生成される。ＭＯＳＦＥＴＱ₃ ，Ｑ₄ による電荷転送の後で、画素ごとにこれらの和と差をとることにより、

のごとく、それらが分離される。前者の和の成分は、通常のイメージセンサの出力（平均照度）に等しい。したがって、例えば上述した画像の１次微分の算出を行う場合には、式（８）の差動入力電圧Ｖ（ｔ）をセンサの外部電気信号（変調信号）として用い、２つのコンデンサＣ₁ ，Ｃ₂ の電荷をＦＥＴＱ₃ およびＱ₄ をオンさせて電圧を介して検出し、その差分を相関として求めればよい。なお、式（９）〜式（１３）の添字はコンデンサＣ₁ ，Ｃ₂ の添字に対応している。 The differential input voltage of this circuit

The mutual conductance g _{m with} respect to is proportional to the photocurrent I _PD (t) that gives the emitter bias currents of the transistors Q ₁ and Q ₂ . Therefore, if this proportionality coefficient is ρ, both collector currents are

The charge accumulated in the capacitors C ₁ and C ₂ is

Thus, a sum of a component proportional to the correlation between I _PD (t) and V (t) and a component proportional to the integral value of I _PD is generated. By taking these sums and differences for each pixel after charge transfer by MOSFETs Q ₃ and Q ₄ ,

As such, they are separated. The former sum component is equal to the output (average illuminance) of a normal image sensor. Therefore, for example, when calculating the first derivative of the image described above, the differential input voltage V (t) of the equation (8) is used as an external electric signal (modulation signal) of the sensor, and the two capacitors C ₁ , The charge of C ₂ may be detected via voltage with FETs Q ₃ and Q ₄ turned on, and the difference between them may be obtained as a correlation. Note that the subscripts in the equations (9) to (13) correspond to the subscripts of the capacitors C ₁ and C ₂ .

However, the disadvantage of this circuit configuration is that the integrated value component of _IPD , which has a very large value depending on the application, easily saturates the storage capacitors (capacitors C ₁ and C ₂ ) and the CCD transfer line. Is in the point that is expected. In order to give up the average illuminance component and generate only the difference component, a method of accumulating electric charge in one capacitor by synthesizing the current branched by adding a current mirror circuit and combining two currents as shown in FIG. A method of incorporating an FET switch circuit (Q ₅ , Q ₆ ) that extracts only the difference component of the accumulated charges of the two capacitors C ₃ and C ₄ is also conceivable. Usually with charges the two capacitors C _3, C ₄ and maintained at a low potential V _XFR in FIG. 5, by switching the V _XFR to the high potential at the time of scanning, via the FETs Q ₃ two capacitors C _3, A voltage corresponding to the difference component of the accumulated charge of C ₄ can be detected.

  In the circuits of FIGS. 4 and 5, the photocurrent generated in the detector is distributed to the two capacitors at a proportional ratio proportional to the voltage input, and can be regarded as being stored as electric charge. If such an apportioning mechanism for the generated current can be embedded in the photodetector itself, it is expected that the frequency band of multiplication will be greatly expanded compared to the case where charges are distributed by circuit means (for example, several GHz or more). ).

FIG. 6 shows an example of such a photodetector, in which the drift direction of optical carriers generated in a semiconductor is controlled by an external electric field so as to modulate the number of carriers flowing down to two pn junctions, and capacitors C ₅ , C _The current is distributed to ₆ . The electric charges of the capacitors C ₅ and C ₆ are read by turning on the MOSFETs Q ₇ and Q ₈ . In the photodetector shown in FIG. 6, the difference between V _MPY ⁺ and V _MPY ⁻ corresponds to the external electric signal (modulation signal) in the generation of the distance image based on the light propagation time described above.

  Next, a switching type half binary correlation circuit using the FET bridge shown in FIG. 7 will be described. Although this circuit configuration has a drawback that the external electrical signal is limited to a binary signal, it has the advantage of easily obtaining high-precision operation, and has common points in the semiconductor manufacturing process with the conventional CCD image sensor. There is an advantage that there are many.

The operation of this circuit is schematically illustrated in FIG. First, when light enters a pixel, the light is converted into a current by a photodiode PD for each pixel. In a normal CCD camera, electric charge is directly accumulated by a capacitor by this current, and the voltage across the capacitor is scanned. In the method proposed here, four FET before normal charge storage (Q ₁₁ ~Q ₁₄₎ performs switching by using the capacitor C at the time of scanning through the FETs Q ₁₅ in a state of being turned on FETs Q ₁₂ and Q _{13 7} voltage is detected. This corresponds to driving with an M-sequence signal, for example. That is, by repeatedly performing current switching, the charge of the central capacitor C ₇ is accumulated (released), and the correlation is calculated by measuring the voltage across the capacitor C ₇ .

The figure which shows the structure of the image sensor by this invention. The figure which shows the operation | movement band of the image sensor shown in FIG. The figure which shows the production | generation of the distance image by the light propagation time using the image sensor by this invention. The circuit diagram which shows the example which shunts a photocurrent to two capacitors using a differential transistor. The circuit diagram which shows the example which takes out only a difference by adding FET switch to the circuit shown in FIG. The figure which shows typically the structure which allocates the optical carrier generate | occur | produced in the semiconductor to two capacitors. The circuit diagram which shows the switching system half binary correlation circuit using FET bridge | bridging. FIG. 8 illustrates an operation of the circuit illustrated in FIG. 7.

Explanation of symbols

1 Current mode multiplier 3 LED
4 Target C Capacitor S Electrical signal FD Photodiode

Claims (7)

A photoelectric conversion element that converts input photons into current, a plurality of transistors that shunt and modulate the current generated by the photoelectric conversion element according to the input modulation signal, and a current that is modulated by the plurality of transistors is timed A plurality of light receiving elements having a plurality of capacitors to be integrated;
A time correlation detection type image sensor comprising: a scanning circuit that scans the plurality of light receiving elements at a period later than that of the modulation signal to sequentially read out and reset the charges of the plurality of capacitors. A photoelectric conversion element that converts input photons into a current, the photoelectric conversion element that shunts and modulates the generated current according to an external electric field applied as a modulation signal, and time-integrates each of the shunt current A plurality of light receiving elements having a plurality of capacitors;
A time correlation detection type image sensor comprising: a scanning circuit that scans the plurality of light receiving elements at a period later than that of the modulation signal to sequentially read out and reset the charges of the plurality of capacitors. The time-correlated image sensor according to claim 1 or 2, wherein the object to be measured is irradiated with modulated light and the reflected light from the object to be measured is received.
The sum of integrated values of the plurality of capacitors represents an average illuminance of the measurement target object, and the modulation signal is input so that a difference between the integration values of the plurality of capacitors becomes a first derivative value of the measurement target object. A time correlation detection type image sensor. A light source that irradiates an object with time-modulated light;
Delay means for delaying the time-modulated signal of the light source;
The time-correlated image sensor according to any one of claims 1 to 3, which receives reflected light from the object.
Calculating the sum and difference of the charges of the plurality of capacitors obtained by inputting the signal delayed by the delay means as a modulation signal to the time-correlated image sensor, and specifying the shape of the object An image analysis apparatus characterized by the above. A photoelectric conversion element that converts input photons into current, a plurality of transistors that shunt and modulate the current generated by the photoelectric conversion element according to the input modulation signal, and a current that is modulated by the plurality of transistors is timed A plurality of light receiving elements having a plurality of capacitors to be integrated;
A time correlation detection type image sensor comprising: a scanning circuit that scans the plurality of light receiving elements at a period slower than the modulation signal and sequentially reads and resets the difference in charge of the plurality of capacitors. A photoelectric conversion element that converts input photons into a current, the photoelectric conversion element that shunts and modulates the generated current according to an external electric field applied as a modulation signal, and time-integrates each of the shunt current A plurality of light receiving elements having a plurality of capacitors;
A time correlation detection type image sensor comprising: a scanning circuit that scans the plurality of light receiving elements at a period slower than the modulation signal and sequentially reads and resets the difference in charge of the plurality of capacitors. A light source that irradiates an object with time-modulated light;
Delay means for delaying the time-modulated signal of the light source;
The time-correlated image sensor according to claim 5, which receives reflected light from the object.
The scanning circuit reads out the difference in charge of the plurality of capacitors obtained by inputting the signal delayed by the delay means as a modulation signal to the time-correlated image sensor as a first-order differential image of the object. An image analysis device.

Images