JPH10281868A - Time correlation detection-type image sensor and image analysis method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform various kinds of image analyses that have been impossible so far due to the limitation in a time resolution. SOLUTION: A plurality of photodetectors with a photo diode FD converting an input photon to a current, a pair of transistors for modulating a current generated by the photo diode FD according to an inputted external electrical signal S, and capacitors C for integrating a current modulated by a pair of transistors in a time domain are provided. As a result, by scanning a plurality of photodetectors, the integration value of each capacitor C can be successively read out.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a time correlation detection type sensor for detecting a time correlation between an incident light intensity and an external electric signal, and an image analysis method using the time correlation detection type sensor.

[0002]

2. Description of the Related Art The applications of image sensing and image analysis are wide-ranging, such as computer vision, automatic visual inspection, scientific measurement, and remote sexing. However, there is almost no method other than diversion of the image sensor for broadcasting as an input means, and there are many restrictions on application development of image processing due to its limited function at present. is there.

[0003] A particular problem is that the frame rate 1 /
The time resolution of 30 seconds is poor, and even if a high-speed scanning of a MOS image sensor or a time signal output type device such as an image detector is used, the sensitivity and the original spatial resolution are sacrificed, so this is an essential solution. do not become.

[0004]

SUMMARY OF THE INVENTION In order to solve this problem, the present invention seeks to provide a solution based on a reconsideration of image sensing methodologies. Originally, a request to increase both the spatial resolution and the temporal resolution results in an enormous amount of information, and thus has an inherent difficulty. The present invention employs a method of applying appropriate processing at the sensor stage and compressing and outputting the necessary and sufficient information for the subsequent image analysis. In addition, a technical effect that various image analysis can be performed is obtained.

[0005]

FIG. 1 shows an embodiment of the present invention.
8, the invention according to claim 1 includes a conversion unit FD that converts an input photon into a current, and a current that modulates a current generated by the conversion unit FD in accordance with an input modulation signal S. A modulating means 1 (FIG. 1) and an integrating means C for integrating the current modulated by the current modulating means with time;
The above object is achieved by providing a plurality of light receiving elements having (FIG. 1) and scanning means for scanning the plurality of light receiving elements at a period slower than the modulation signal and sequentially reading out the integrated value of the integrating means. The invention described in claim 2 is the first invention.
In the time detection correlation type image sensor described in (1), the same modulation signal is input to the current modulation means 1 of each light receiving element. The invention according to claim 3 uses a light source 3 that irradiates the object 4 with time-modulated light, and a delay unit that delays the time-modulated signal of the light source 3, and the signal delayed by the delay unit is claimed. The light from the light source 3 is reflected by the object 4 by inputting it as a modulation signal to the time-correlation type image sensor described in 1 or 2, and the light of the object is determined by the light propagation time when the light is received by the image sensor. Only a specific portion is detected (FIG. 3). According to a fourth aspect of the present invention, a position of an image input to the light receiving element is synchronized with the modulation signal while applying a modulation signal to the current modulation means 1 of the time detection correlation type image sensor according to the first or second aspect. By vibrating the image, the differential value of the image is obtained.

[0006] In the section of the means for solving the above-mentioned problems, which explains the configuration of the present invention, the drawings of the embodiments of the present invention are used to make the present invention easy to understand. However, the present invention is not limited to this.

[0007]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a time correlation detection type image sensor and an image analysis method according to the present invention will be described with reference to FIGS.

FIG. 1 shows the basic structure of a time correlation type image sensor according to the present invention. The point of this structure is to provide a photodiode detector FD for converting an incident photon into a photocurrent (multiplicand), to supply an external electric signal (multiplier) S to all the pixels in common, and to compare the photocurrent with the electric signal. A current mode multiplier 1 for generating a current proportional to the product between them; a capacitor C for integrating the above product current with time and accumulating it as a correlation value; scanning the correlation value and outputting it as a normal video signal Scanning unit (CCD or M
OS switch circuit) 2.

The output of the photodetector of pixel (i, j) is represented by f _{i, j}
(T), the external electric signal is g (t), and the integration time corresponding to the scanning period is T, the output of the (i, j) pixel of the time correlation type image sensor is

(Equation 1) It will be represented as

FIG. 2 shows the structure (operation) of the time correlation detection type sensor according to the present invention viewed from the frequency band. For changing light such as modulated light, any high time frequency band can be assumed for f _{i, j} (t). In a normal image sensor, this band is strongly limited by a scanning period (frame rate), but in the above-described sensor structure, this upper limit is removed to the limit band of the photodetector. Since the external electric signal S is common to all pixels, this band is not limited by the scanning period, and the upper limit band is estimated to be about the limit band of the signal distribution wiring. On the other hand, the output signal is a correlation value thereof and is an integrated value for the scanning cycle, so that the band is always within the band of the scanning cycle.

-Generation of Range Image by Light Propagation Time- Correlation measurement in image sensing has been mainly discussed so far mainly on spatial correlation-pattern matching, and correlation with respect to a time axis has hardly been discussed. Here, using a few specific examples, what kinds of application developments can be opened to image sensing when a very high-speed time correlation can be obtained will be described. The spatial resolution and the form of the output signal are assumed to be positively compatible with a normal broadcast image sensor. If the circuit complexity is reduced to about 10 transistors / pixel, a spatial resolution of about HDTV can be targeted.

First, generation of a distance image based on light propagation time will be described with reference to FIG. This application becomes possible when a band of several GHz or more can be realized for both the photodetector and the external electric signal input. With a time resolution of several GHz, the light propagation length resolution is several cm required for three-dimensional measurement.
It should be noted that:

It is assumed that a light source that can be time-modulated over a wide band (including a high frequency component), such as a laser diode or an LED, is assumed. Here, LED3 is used. The LED 3 does not emit light at a constant level, but emits light by modulating the luminance of the emission into an M-sequence pattern. This modulated light is defined as M (t). That is,

(Equation 2) It satisfies.

The modulation signal of the LED 3 is delayed by a delay time τ via a delay device 6 and input to the sensor 5 as an external electric signal. The incident light captured by each light receiving element of the sensor 5 is modulated by an external electric signal, and an electric charge corresponding to a product of the incident light and the external electric signal is accumulated as a correlation value. By sequentially scanning each element, the accumulated electric charge of each element is read out, whereby the correlation is obtained.

Here, the sum of the distance from the LED 3 to the object 4 and the distance from the object 4 to the sensor 5 is l _i (i =
1, 2,... N), and the speed of light is c. However, it is assumed that the object 4 is stationary. The light intensity distribution on the sensor 5 is generally written as:

(Equation 3) However, light attenuation and the like are ignored for simplification. Further, an illustration of an imaging optical system for forming an image of the object 4 on the sensor 5 is omitted. Here, f _i (x, y) is an image at a distance l _i , and n (x, y, t) is observation noise such as illumination light or sunlight. This intensity distribution G (x, y, t)
And the time correlation function of the modulated signal M (t) is

(Equation 4) And the time delay of M (t) given as an external signal is τ
= L _i / c means that an output of f _i (x, y) is obtained. That is, by appropriately selecting τ, only the surface of the object 4 at a desired distance can be detected brightly regardless of external light. Thus, for example, τ
Is changed step by step, a three-dimensional image of the object 4 is obtained.

-Calculation of first derivative of image- As a method of giving a modulation component to the intensity of incident light, there is a method of causing a sensor to vibrate minutely. In the applications described below, the time frequency band of the photodetector and the external electric signal are both 100 kHz.
z or less is sufficient. Generating the derivative without using the pixel-to-pixel difference has the advantage that errors due to the pixel sensitivity difference, which is the largest noise source of the image sensor, can be eliminated.

The light intensity distribution that will arrive on the sensor when the sensor is stationary is _denoted by f ₀ (x, y). The sensor is given a rotational shift motion with a small radius ε and a rotational angular velocity ω, that is, a parallel motion with a circle having a radius ε as a locus in the light receiving surface of the sensor. Compared to the movement speed of the sensor,
Assume that the subject's movements (if any) are negligible. At this time, the light intensity distribution captured by the sensor at time t is:

(Equation 5) And a first-order approximation with Taylor expansion gives

(Equation 6) It is. Here, multiplying equation (6) by εcosωt and taking a time average gives

(Equation 7) Becomes Here, <cosωt> = 0, <sinωtc
osωt> = 0 and <cos ² ωt> = １ ／ were used.
The left side of equation (7) is generated as a correlation value on the sensor, and ε
Is also known, the x derivative f _{0x of the} image is used as the sensor output.
(X, y) will be read. Similarly, f
_0y (x, y) is also obtained by multiplying equation (6) by ε sinωt and taking a time average.

This is because the x derivative of the image f _0x (x,
y) is read out. Similarly, by using the y-direction component of the rotational shift amount as an external electric signal,
The y derivative f _0y (x, y) of the image is read. Also,
If a constant is read as an external signal, a moving average image due to rotational shift can be obtained.

The fact that the first derivative can be obtained with high precision is not limited to the reduction in the amount of arithmetic processing, but also the great accuracy of various image processing (edge detection, optical flow detection, binocular stereo, etc.) using gradient vectors. Leads to improvement.
Instead of vibrating the sensor itself, the imaging may be moved (vibrated) on a fixed sensor by, for example, vibrating a part of the optical system.

-Structure of Sensing Cell of Time Correlation Image Sensor- Next, a specific configuration of the sensing cell of the time correlation image sensor will be described.

As a typical circuit configuration for realizing multiplication,
Variable conductance differential amplifiers are known. FIG.
Indicates a construction method using this. Although bipolar transistors Q ₁ and Q ₂ are used in FIG. 4, a pair of MOSFETs biased in a subthreshold region may be used as a differential transistor pair.

The differential input voltage of this circuit

(Equation 8) Is the transconductance g _m of the transistor Q
_1, photocurrent I give emitter bias current Q ₂ '
_It is proportional to _PD (t). Therefore, if this proportionality factor is ρ, both collector currents are

(Equation 9) And the charges stored in the capacitors C ₁ and C ₂ include:

(Equation 10)

[Equation 11] , A sum of a component proportional to the correlation between I _PD (t) and V (t) and a component proportional to the integral value of I _PD is generated. After charge transfer by MOSFETs Q ₃ and Q _4, by taking the sum and difference of these for each pixel,

(Equation 12)

(Equation 13) Like, they are separated. The former sum component is equal to the output (average illuminance) of a normal image sensor. Therefore, for example, when calculating the first derivative of the above-described image, the differential input voltage V (t) of Expression (8) is used as an external electric signal (modulation signal) of the sensor, and the two capacitors C ₁ , and the charge of C ₂ to turn on the FETs Q ₃ and Q ₄ is detected through the voltage may be determined the difference as the correlation. Note that the suffixes of Expressions (9) to (13) are the capacitors C
Corresponds to _1, the subscript of C _2.

However, a drawback of this circuit configuration is that, depending on the application, the component of the integrated value of _IPD , which has a very large value, tends to saturate the storage capacitors (capacitors C ₁ and C ₂ ) and the CCD transfer line, so that the accuracy is low. Difficulties are expected. In order to give up the average illuminance component and generate only the difference component, a method in which currents branched by adding a current mirror circuit are combined in reverse phase to accumulate charge in one capacitor, or as shown in FIG. A method of incorporating a FET switch circuit (Q ₅ , Q ₆ ) for extracting only the difference component between the charges stored in the _three capacitors C ₃ and C ₄ may be considered.
Normally, the two capacitors C ₃ and C ₄ are charged by maintaining V _XFR in FIG. 5 at a low potential, and by switching V _XFR to a high potential during scanning, the two capacitors C ₃ and C ₃ are connected via the FET Q ₃ . voltage can be detected in accordance with the difference component of the charge accumulated in the C _4.

In the circuits of FIGS. 4 and 5, it can be considered that the photocurrent generated in the detector is distributed to the two capacitors at a proportional ratio proportional to the voltage input, and is stored as electric charge. If such a mechanism for apportioning the generated current can be embedded in the photodetector itself, it is expected that the frequency band of the multiplication will be greatly expanded (for example, several GHz or more) as compared with the case where electric charges are distributed by circuit means. ).

FIG. 6 shows an example of such a photodetector.
The direction of drift of optical carriers generated in the semiconductor is controlled by an external electric field to modulate the number of carriers flowing down to two pn junctions, and current is distributed to the capacitors C ₅ and C ₆ . The charges of the capacitors C ₅ and C ₆ are MOSFE
The data is read out by turning on TQ ₇ and Q ₈ . In the photodetector shown in FIG. 6, the difference between V _MPY ⁺ and V _MPY ⁻ corresponds to the external electric signal (modulation signal) in the generation of the distance image based on the light propagation time.

Next, a switching type half binary correlation circuit using the FET bridge shown in FIG. 7 will be described. This circuit configuration has the disadvantage that the external electric signal is limited to a binary signal, but has the advantage of being able to easily obtain high-precision operation, and also has the common features with the conventional CCD image sensor in the semiconductor manufacturing process. There is an advantage that there are many.

The operation of this circuit is shown schematically in FIG. First, when light enters a pixel, the light is converted into a current by a photodiode PD for each pixel. In an ordinary CCD camera, the electric charge is directly accumulated in the capacitor by this current, and the voltage across the capacitor is scanned.
In the method proposed here, switching is performed using four FETs (Q _{11 to} Q ₁₄ ) before normal charge accumulation, and the FE is turned on during scanning with the FETs Q ₁₂ and Q ₁₃ turned on.
Through the TQ ₁₅ detects the voltage of the capacitor C _7. This corresponds to, for example, driving with an M-sequence signal. That is, by repeating the current switching charge of the central capacitor C ₇ is accumulated (released), by measuring the voltage across the capacitor C _7, so that the correlation was calculated.

[0028]

According to the first aspect of the present invention, a conversion means for converting an input photon into a current, a current modulation means for modulating a current generated by the conversion means in accordance with an input electric signal, and a current modulation A plurality of light receiving elements having an integrating means for time-integrating the current controlled by the means, and scanning the plurality of light receiving elements to sequentially read out the integrated value of the integrating means, thereby performing time correlation detection with high time resolution. Can be. According to the third aspect of the present invention, the light source that irradiates the object with the time-modulated light and the delay unit that delays the time-modulated signal of the light source are used, and the signal delayed by the delay unit is used. Alternatively, since a modulation signal is input to the time correlation type image sensor described in 2, the distance image or the like can be generated. According to the fourth aspect of the invention, the position of the image input to the image sensor is vibrated while applying the modulation signal to the time detection correlation type image sensor according to the first or second aspect, so that the accuracy is high. The derivative of the image can be determined.

[Brief description of the drawings]

FIG. 1 is a diagram showing a structure of an image sensor according to the present invention.

FIG. 2 is a diagram showing an operation band of the image sensor shown in FIG.

FIG. 3 is a diagram showing generation of a distance image based on light propagation time using an image sensor according to the present invention.

FIG. 4 is a circuit diagram showing an example in which a photocurrent is divided into two capacitors by using a differential transistor.

FIG. 5 is a circuit diagram showing an example in which an FET switch is added to the circuit shown in FIG. 4 to extract only a difference.

FIG. 6 is a diagram schematically showing a configuration in which optical carriers generated in a semiconductor are allocated to two capacitors.

FIG. 7: Switching method half 2 using FET bridge
FIG. 3 is a circuit diagram showing a value correlation circuit.

FIG. 8 illustrates operation of the circuit illustrated in FIG. 7;

[Explanation of symbols]

 Reference Signs List 1 current mode multiplier 3 LED 4 target C capacitor S electric signal FD photodiode

Claims (4)

[Claims] 1. A converter for converting an input photon into a current, a current modulator for modulating a current generated by the converter in accordance with an input modulation signal, and a time integration of the current modulated by the current modulator. A plurality of light receiving elements each having an integrating means, and a scanning means for scanning the plurality of light receiving elements at a period slower than the modulation signal and sequentially reading out the integrated value of the integrating means. Detection type image sensor. 2. The time correlation detection type image sensor according to claim 1, wherein the same modulation signal is input to the current modulation means of each of the light receiving elements. 3. A signal source which irradiates an object with time-modulated light and delay means for delaying a time-modulated signal of the light source, wherein the signal delayed by the delay means is provided.
By inputting the modulation signal to the time-correlation type image sensor according to the above, the light from the light source is reflected by the object, and further determined by the light propagation time when received by the image sensor. An image analysis method characterized by detecting only a specific portion. 4. A position of an image input to said light receiving element in synchronization with said modulation signal while applying a modulation signal to said current modulation means of said time detection correlation type image sensor according to claim 1 or 2. An image analysis method, wherein a differential value of the image is obtained by vibrating.