JP2022135438A - Distance image capture device and distance image capture method - Google Patents

Abstract

To provide a distance image capture device that captures a correct distance image without changing an optical structure in the device and without interrupting the size reduction, the height reduction, the higher definition, and the improvement of the quantum efficiency of the device.SOLUTION: A distance image capture device includes: a light source unit that irradiates a measurement space with optical pulses; a light-reception unit including a photoelectric conversion element that generates charges according to light entering from the measurement space, a plurality of pixel circuits including a plurality of charge accumulation units for accumulating charges in a frame period, and a pixel driving circuit that allocates and accumulates charges in the charge accumulation unit by performing on/off processing of a transfer transistor at a predetermined accumulation timing synchronizing with the emission of the optical pulses; and a distance calculation unit that calculates a distance between a subject and the light-reception unit in the measurement space in accordance with the amount of charges determined based on a first charge amount of the charges accumulated in the charge accumulation unit. The distance calculation unit calculates a distance by subtracting a second charge amount by noise charges being the accumulated charges other than the charges allocated and accumulated in the on/off processing of the transfer transistor from the first charge amount.SELECTED DRAWING: Figure 1

Description

The present invention relates to a range image capturing device and a range image capturing method.

Conventionally, the time-of-flight (hereinafter referred to as "ToF") method of measuring the distance to a subject based on the flight time of light, using the fact that the speed of light is known. There is a device (see, for example, Patent Document 1).
The ToF range imaging device consists of a light source unit that emits light and an imaging unit that includes a pixel array in which a plurality of pixel circuits that detect light for measuring distance are arranged in a two-dimensional matrix (array). I have. Each of the pixel circuits has, as a component, a photoelectric conversion element (for example, a photodiode) that generates an electric charge corresponding to the intensity of light.
With this configuration, the ToF range image capturing device can acquire (capture) information about the distance between itself and the subject and an image of the subject in the measurement space (three-dimensional space).

In addition, since it is used in portable terminals such as smartphones and tablet terminals, there is a demand for miniaturization and low profile (thinness) of the housing (imaging device) of the range image imaging device.
On the other hand, there is a demand for an increase in the number of pixels in an imaging unit in order to obtain a higher definition image captured by a distance image capturing device.
Therefore, the incident angle of light incident on the pixel circuits arranged at the ends in the pixel array becomes larger than the incident angle of incident light incident on the pixel circuits arranged near the center. .

As the angle of incidence of light incident on the pixel circuit increases, the light irradiates not only the photoelectric conversion element in the pixel circuit but also circuits around the photoelectric conversion element, such as a charge storage section that stores charges.
Then, not only in the photoelectric conversion element but also in the peripheral circuit (including the charge storage section), charges corresponding to the light irradiated by the photoelectric effect (hereinafter referred to as noise charge) are generated, and these charge in the high-potential charge storage section. influx.

As a result, the noise charge from the peripheral circuit is added to the charge transferred from the photoelectric conversion element to the charge storage unit and used for distance measurement. Decrease accuracy.
Therefore, as a countermeasure for suppressing the generation of the noise charge described above, the pixel circuit is configured so that incident light is not incident on peripheral circuits other than the photoelectric conversion element.

For example, as a measure to suppress the incidence of light on peripheral circuits other than the photoelectric conversion element in the pixel circuit, a microlens is formed above each pixel circuit, and the light is collected by the microlens. A configuration is used in which emitted light is made incident on a photoelectric conversion element. As a result, light incident on the pixel circuit is condensed by the microlens so that only the photoelectric conversion element is irradiated with the light, and light irradiation to other peripheral circuits is suppressed.

In addition, as a measure to suppress the incidence of light on peripheral circuits other than photoelectric conversion elements in other pixel circuits, a light-shielding layer with a metal pattern is provided in the layer directly above the pixel circuits to shield the circuit parts other than the photoelectric conversion elements. Such a configuration is employed in an FSI (Front Side Illuminated) CMOS image sensor.

JP 2015-29054 A

However, in light-condensing by microlenses, it is necessary to reduce the thickness of the microlenses as the thickness of the distance image pickup device is reduced due to the demand for a low profile of the distance image pickup device.
For this reason, the microlens is not provided with a sufficient curvature for condensing the incident light onto the photoelectric conversion element, making it difficult to collect the light so that only the photoelectric conversion element is irradiated with the light. ing.

In addition, in the FSI method, the wiring layer of the wiring in the pixel circuit is arranged between the photoelectric conversion element and the microlens. can form a metal pattern for
On the other hand, in the BSI (Back Side Illuminated) method, light is not incident from the front surface of a semiconductor substrate on which pixel circuits are formed, but is configured to be incident from the back surface of the semiconductor substrate.

Therefore, since no circuit is formed, the wiring layer does not exist on the back surface of the semiconductor substrate. cannot be formed.
In addition, in the configuration of the BSI system, as in Patent Document 1, for the purpose of improving the sensitivity to incident light, in order to increase the quantum efficiency, a scattering/reflecting structure is provided to reduce the optical path length of light within the semiconductor substrate. A lengthening configuration may be adopted.
In this configuration, the light condensed by the microlens is scattered within the semiconductor substrate, so more light enters the peripheral circuit, causing more noise charge to occur in the peripheral circuit other than the photoelectric conversion element. becomes.

The present invention has been made in view of such circumstances, without changing the optical configuration in the device, without hindering the miniaturization, low profile, high definition, and improvement of the quantum efficiency of the device. An object of the present invention is to provide a range image capturing device and a range image capturing method for capturing more accurate range images.

In order to solve the above-described problems, the distance image pickup apparatus of the present invention includes a light source unit that irradiates a light pulse into a measurement space, a photoelectric conversion element that generates charges according to the light incident from the measurement space, and a frame. a plurality of pixel circuits each having a plurality of charge storage units for accumulating the charge in a cycle; and at predetermined accumulation timing synchronized with irradiation of the light pulse, each of the charge storage units is subjected to ON/OFF processing of the transfer transistor. A distance between the subject and the light receiving section in the measurement space is calculated based on a light receiving section having a pixel drive circuit for distributing and accumulating charges, and a first charge amount of the charges accumulated in each of the charge accumulating sections. a distance calculation unit, wherein the distance calculation unit calculates the distance by subtracting, from each of the first charge amounts, a second charge amount of noise charge, which is a charge generated by a device other than the photoelectric conversion element. characterized by

In the distance image pickup device of the present invention, any one of the frames is a noise charge amount acquisition frame for acquiring the second charge amount, and the light pulse is emitted from the light source unit in the noise charge amount acquisition frame. After that, in a state in which the pixel drive circuit turns off the transfer transistor and does not distribute the charge from the photoelectric conversion element to the charge storage unit, the charge amount stored in the charge storage unit is set as the second charge amount. and, in a frame after the noise charge amount acquisition frame, the pixel drive circuit turns on and off the transfer transistor to distribute and store the charge in each of the charge storage units to obtain the first charge amount. Characterized by

The distance image capturing apparatus of the present invention divides each frame to generate a first sub-frame and a second sub-frame, and in the first sub-frame, after the light pulse is emitted from the light source unit, The charge amount accumulated in the charge accumulation unit is defined as the second charge amount in a state where the pixel drive circuit turns off the transfer transistor and the photoelectric conversion element does not accumulate charges in the charge accumulation unit, and In the second sub-frame, the pixel driving circuit turns on and off the transfer transistor to distribute and accumulate the charge in each of the charge accumulation units to obtain the first charge amount.

The distance image pick-up device of the present invention comprises an accumulation period in which the frame distributes and accumulates the charges in each of the charge accumulation sections, and a readout period in which the amount of charge accumulated from each of the charge accumulation sections is read out. and each of the first subframe and the second subframe is configured by shortening the accumulation period.

The distance image pickup device of the present invention comprises a light source unit that irradiates a light pulse into a measurement space, a photoelectric conversion element that generates electric charges according to the light incident from the measurement space, and a plurality of light sources that accumulate the electric charges in a frame cycle. a plurality of pixel circuits each having a charge storage unit; and a pixel drive circuit that performs on/off processing of each transfer transistor in each of the charge storage units at a predetermined accumulation timing synchronized with the irradiation of the light pulse, thereby distributing and storing the charge. and a distance calculation unit configured to calculate the distance between the subject and the light receiving unit in the measurement space based on the charge amount determined by the first charge amount of the charges accumulated in each of the charge storage units. wherein the distance calculation unit subtracts, from each of the first charge amounts, a second charge amount due to noise charges, which are accumulated charges other than the charges distributed and accumulated by the ON/OFF processing of the transfer transistors, to obtain the distances. It is characterized by performing calculations.

In the distance image pickup device of the present invention, the distance calculation unit calculates the second When the second allocation number of the allocation changes in each of the frames after the noise charge amount acquisition frame, the division result obtained by dividing the second allocation number by the first allocation number is used as an adjustment coefficient, and the adjustment is performed. Multiplying the fourth charge amount by a coefficient to calculate a fifth charge amount, adding the fifth charge amount to the third charge amount, correcting the second charge amount, and using the corrected amount to calculate the distance. It is characterized by

The range image pickup apparatus of the present invention is characterized in that the light pulse emitted from the light source unit is a pulse of light in a near-infrared wavelength band with a predetermined width.

The distance image pickup device of the present invention is characterized in that the structure of the pixel circuit is a BSI (Back Side Illumination) structure.

The distance image pickup device of the present invention is characterized in that the pixel circuit has three or more charge storage units.
In the distance image pickup device of the present invention, the pixel circuit is provided with one or more charge discharging transistors for discharging charges from the photoelectric conversion elements except for a period during which the charges are distributed and accumulated in each of the charge accumulation units. It is characterized by

A distance image capturing method of the present invention controls a distance image capturing device including a plurality of pixel circuits each composed of a photoelectric conversion element and a plurality of charge storage units, a light source unit, a pixel driving circuit, and a distance calculation unit. A distance image capturing method, wherein the light source unit irradiates a light pulse into a measurement space, and the pixel circuit generates charge generated by the photoelectric conversion element according to the light incident from the measurement space in a frame period. is stored in the charge storage section, and the pixel driving circuit performs on/off processing of each transfer transistor in each of the charge storage sections at a predetermined accumulation timing synchronized with the irradiation of the light pulse to distribute the charge. and the distance calculation unit determines the distance between the subject and the light receiving unit in the measurement space based on the charge amount determined by the first charge amount of the charges accumulated in each of the charge accumulation units. a distance calculation step of calculating a distance, wherein the distance calculation unit calculates a second charge amount by noise charge, which is an accumulated charge other than the charge distributed and accumulated by the ON/OFF processing of the transfer transistor, from each of the first charge amounts. is subtracted to calculate the distance.

As described above, according to the present invention, a more accurate measurement can be performed without changing the optical configuration in the device, without hindering the miniaturization, low profile, high definition, and improvement of the quantum efficiency of the device. A distance image capturing device and a distance image capturing method for capturing a distance image can be provided.

FIG. 1 is a block diagram showing a schematic configuration of a range image pickup apparatus according to the first embodiment of the present invention. 3 is a circuit diagram showing an example of a configuration of a pixel circuit 321 arranged in a range image sensor 32 in the range image pickup device according to the first embodiment of the present invention; FIG. FIG. 4 is a timing chart for transferring charges generated by the photoelectric conversion element PD to each of the charge storage units CS; 3 is a diagram showing an example of the arrangement (layout pattern) of each transistor of the pixel circuit 321 in this embodiment. FIG. FIG. 10 is a diagram illustrating a process of acquiring the charge amount of noise charges accumulated in the charge accumulation unit CS in the second embodiment; FIG. 10 is a conceptual diagram illustrating charge accumulation processing in the charge accumulation unit CS in the noise charge amount acquisition mode; FIG. 10 is a conceptual diagram illustrating charge accumulation processing in the charge accumulation unit CS in the distance measurement charge amount acquisition mode; FIG. 10 is a diagram illustrating a process of acquiring the charge amount of noise charges accumulated in the charge accumulation unit CS in the second embodiment; FIG. 12 is a diagram illustrating a process of acquiring the charge amount of noise charges accumulated in the charge accumulation unit CS in the third embodiment; FIG. 11 is a conceptual diagram illustrating noise charges corresponding to reflected light RL in the charge accumulation unit CS in the distance measurement charge amount acquisition mode in the third embodiment; 10 is a flowchart showing an operation example of processing for calculating the distance between the distance image sensor 32 and the subject S in the distance image pickup device 1 according to the third embodiment.

<First embodiment>
A first embodiment of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to a first embodiment of the present invention. A distance image pickup device 1 configured as shown in FIG. Note that FIG. 1 also shows a subject S, which is an object whose distance is to be measured in the distance image pickup device 1 . The distance image pickup device is, for example, a distance image sensor 32 (described later) in the light receiving section 3 .

The light source unit 2 irradiates a light pulse PO to a shooting target space in which a subject S whose distance is to be measured in the range image pickup device 1 exists, under the control of the range image processing unit 4 . The light source unit 2 is, for example, a surface emitting semiconductor laser module such as a vertical cavity surface emitting laser (VCSEL: Vertical Cavity Surface Emitting Laser). The light source unit 2 includes a light source device 21 and a diffuser plate 22 .

The light source device 21 is a light source that emits a laser beam in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) as a light pulse PO to irradiate the subject S. FIG. The light source device 21 is, for example, a semiconductor laser light emitting element. The light source device 21 emits pulsed laser light under the control of the timing control section 41 .
The diffuser plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 over a surface on which the subject S is irradiated. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO, and the subject S is irradiated with the light pulse PO.

The light receiving unit 3 receives the reflected light RL of the light pulse PO reflected by the subject S whose distance is to be measured in the distance image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving section 3 includes a lens 31 and a distance image sensor 32 .
The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32 . The lens 31 emits the incident reflected light RL to the distance image sensor 32 side and causes the pixel circuits provided in the light receiving area of the distance image sensor 32 to receive the light (incident).

The distance image sensor 32 is an image pickup device used in the distance image pickup device 1 . The distance image sensor 32 includes a plurality of pixel circuits 321 and a pixel drive circuit 322 that controls each of the pixel circuits 321 in a two-dimensional light receiving area.
The pixel circuit 321 includes one photoelectric conversion element (for example, a photoelectric conversion element PD to be described later) and a plurality of charge storage units (for example, charge storage units CS1 to CS4 to be described later) corresponding to the one photoelectric conversion element. , and components for distributing charges to the respective charge storage units.

The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge storage units according to control from the timing control unit 41 . Also, the distance image sensor 32 outputs a pixel signal corresponding to the amount of charge distributed to the charge storage section. In the range image sensor 32, a plurality of pixel circuits are arranged in a two-dimensional matrix, and pixel signals for one frame corresponding to each pixel circuit are output.

The distance image processing unit 4 controls the distance image capturing device 1 and calculates the distance to the subject S. FIG.
The distance image processing section 4 includes a timing control section 41 , a distance calculation section 42 and a measurement control section 43 .
The timing control section 41 controls the timing of outputting various control signals required for distance measurement under the control of the measurement control section 43 . The various control signals here include, for example, a signal for controlling the irradiation of the light pulse PO, a signal for distributing the reflected light RL to a plurality of charge storage units, a signal for controlling the number of distributions per frame, and the like. The number of distributions is the number of repetitions of the process of distributing charges to the charge storage section CS (see FIG. 3).

The distance calculation unit 42 outputs distance information obtained by calculating the distance to the subject S based on the pixel signals output from the distance image sensor 32 under the control of the measurement control unit 43 . The distance calculation unit 42 calculates the delay time Td from the irradiation of the light pulse PO to the reception of the reflected light RL based on the charge amounts accumulated in the plurality of charge accumulation units CS. The distance calculation unit 42 calculates the distance from the distance imaging device 1 to the subject S according to the calculated delay time Td.

The measurement control unit 43 divides each mode of frames repeated in a frame cycle into a noise charge amount acquisition mode, which is a frame for acquiring charge noise, and a distance measurement charge amount acquisition mode, which is a normal frame for distance measurement. select.
Then, the measurement control unit 43 controls the timing in the timing control unit 41 and the calculation in the distance calculation unit 42 corresponding to each of the noise charge amount acquisition mode and the distance measurement charge amount acquisition mode ( later). Here, noise charges are charges generated by incident light (background light and reflected light) from the measurement space in a region other than the photoelectric conversion element and accumulated in the charge accumulation section CS. That is, the noise charges are accumulated charges other than the charges distributed and accumulated by the ON/OFF processing of the transfer transistors G. FIG.

That is, the distance image capturing apparatus according to this embodiment calculates the distance between the object and the distance image sensor 32 based on the charges accumulated in the charge accumulation unit CS. Therefore, in the calculation of the calculated distance measurement, noise charges due to background light or the like are accumulated in the charge accumulation unit CS until the time of readout in each frame, regardless of the charges read out from the photoelectric conversion element at a predetermined timing. Calculated distances are less accurate.

With such a configuration, in the distance image capturing device 1, the light receiving unit 3 receives the reflected light RL that is reflected by the subject S from the light pulse PO in the near-infrared wavelength band that the light source unit 2 irradiates the subject S, A distance image processing unit 4 outputs distance information obtained by measuring the distance between the subject S and the distance image capturing device 1 .
Although FIG. 1 shows the distance image pickup device 1 having a configuration in which the distance image processing unit 4 is provided inside, the distance image processing unit 4 is a component provided outside the distance image pickup device 1. may

Here, the configuration of the pixel circuit 321 in the distance image sensor 32 will be described. FIG. 2 is a circuit diagram showing an example of the configuration of the pixel circuit 321 arranged in the range image sensor 32 in the range image pickup device according to the first embodiment of the present invention. The pixel circuit 321 in FIG. 2 is a configuration example including four pixel signal readout units RU1 to RU4.

The pixel circuit 321 includes one photoelectric conversion element PD, charge discharge transistors GD (GD1 and GD2 described later), and four pixel signal readout units RU (RU1 to RU4) that output voltage signals from corresponding output terminals O. Prepare. Each pixel signal readout unit RU includes a transfer transistor G, a floating diffusion FD, a charge storage capacitor C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The floating diffusion FD and the charge storage capacitor C constitute a charge storage section CS.

In the pixel circuit 321 shown in FIG. 2, the pixel signal readout unit RU1 that outputs a voltage signal from the output terminal O1 includes a transfer transistor G1 (transfer MOS transistor), a floating diffusion FD1, a charge storage capacitor C1, and a reset transistor RT1. , a source follower transistor SF1, and a selection transistor SL1. In the pixel signal readout unit RU1, the charge storage unit CS1 is composed of the floating diffusion FD1 and the charge storage capacitor C1. The pixel signal readout units RU2, RU3 and RU4 also have the same configuration.

The photoelectric conversion element PD is an embedded photodiode that photoelectrically converts incident light, generates electric charges corresponding to the incident light (incident light), and accumulates the generated electric charges. In this embodiment, incident light is incident from the space to be measured.
In the pixel circuit 321, the charges generated by the photoelectric conversion element PD photoelectrically converting incident light are distributed to the four charge storage units CS (CS1 to CS4), respectively, and the respective charges corresponding to the charge amounts of the distributed charges are distributed. A voltage signal is output to the pixel signal processing circuit 325 .
Further, the configuration of the pixel circuits arranged in the distance image sensor 32 is not limited to the configuration including the four pixel signal readout units RU (RU1 to RU4) as shown in FIG. A pixel circuit having a configuration in which the readout unit RU includes one or more pixel signal readout units RU may be used.

In driving the pixel circuit 321 of the distance image pickup device 1, the light pulse PO is emitted for the irradiation time To, and the reflected light RL is received by the distance image sensor 32 after the delay time Td. Under the control of the timing control unit 41, the pixel driving circuit 322 transfers charges generated in the photoelectric conversion elements PD to the transfer transistors G1, G2, G3, and G4 in synchronization with the irradiation of the light pulse PO as accumulation drive signals. TX1 to TX4 are supplied and switched according to respective timings, and are accumulated in the order of the charge accumulation units CS1, CS2, CS3, and CS4.

Then, the pixel driving circuit 322 controls the reset transistor RT and the selection transistor SL with the drive signals RST and SEL, respectively, and converts the charge accumulated in the charge storage section CS into an electric signal with the source follower transistor SF. , and outputs the generated electric signal to the distance calculator 42 via the terminal O. FIG.
In addition, the pixel drive circuit 322 discharges the charge generated in the photoelectric conversion element PD to the power supply VDD by the drive signal RSTD under the control of the timing control unit 41 (erases the charge).

FIG. 3 is a diagram showing a timing chart for transferring charges generated by the photoelectric conversion element PD to each of the charge storage units CS.
In the timing chart of FIG. 3, the vertical axis indicates the pulse level and the horizontal axis indicates time. The relative relationship on the time axis between the light pulse PO and the reflected light RL, the timing of each of the storage drive signals TX1 to TX4 supplied to the transfer transistors G1 to G4, and the timing of the drive signal RSTD to be supplied to the charge discharge transistor GD. showing.

The timing control unit 41 causes the light source unit 2 to irradiate the measurement space with the light pulse PO. As a result, the light pulse PO is reflected by the subject and received by the light receiving section 3 as reflected light RL. Then, the photoelectric conversion element PD generates charges corresponding to each of the background light and the reflected light RL. The pixel driving circuit 322 controls on/off of each of the transfer transistors G1 to G4 in order to transfer the charge generated by the photoelectric conversion element PD to each of the charge storage units CS1 to CS4.
That is, the pixel drive circuit 322 supplies the accumulation drive signals TX1 to TX4 to the transfer transistors G1 to G4 as "H" level signals having a predetermined time width (the same width as the irradiation time To).

The pixel drive circuit 322, for example, turns on the transfer transistor G1 provided on the transfer path for transferring the charge from the photoelectric conversion element PD to the charge storage unit CS1. As a result, charges photoelectrically converted by the photoelectric conversion element PD are accumulated in the charge accumulation unit CS1 via the transfer transistor G1. After that, the pixel drive circuit 322 turns off the transfer transistor G1. As a result, transfer of charges to the charge storage section CS1 is stopped. In this manner, the vertical scanning circuit 322 accumulates charges in the charge accumulation section CS1. The same applies to the other charge storage units CS2, CS3 and CS4.

At this time, in a charge accumulation period (a period during which charges are accumulated in each of the charge accumulation sections CS in a frame) in which charges are distributed to the charge accumulation sections CS, each of the accumulation drive signals TX1, TX2, TX3, and TX4 is a transfer transistor. The accumulation cycle supplied to each of G1, G2, G3 and G4 is repeated.
Charges corresponding to the incident light are transferred from the photoelectric conversion element PD to the charge storage units CS1, CS2, CS3, and CS4 via the transfer transistors G1, G2, G3, and G4, respectively. A plurality of accumulation cycles are repeated during the charge accumulation period.
As a result, charge is accumulated in each of the charge accumulation units CS1, CS2, CS3, and CS4 in each accumulation cycle of the charge accumulation units CS1, CS2, CS3, and CS4 in the charge accumulation period.

Further, when repeating the accumulation cycle of each of the charge storage units CS1, CS2, CS3, and CS4, the pixel drive circuit 322 transfers the charge from the photoelectric conversion element PD after the transfer (transfer) of the charge to the charge storage unit CS4 is completed. The "H" level drive signal RSTD is supplied to the charge discharging transistor GD provided on the discharging path to turn it on.
As a result, the charge discharge transistor GD discards the charge generated in the photoelectric conversion element PD after the immediately preceding charge storage period of the charge storage section CS4 before the start of the storage period of the charge storage section CS1 (ie, photoelectric conversion). reset device PD). That is, one or a plurality of (one or more) charge discharge transistors GD are provided, and charge generated by incident light from the photoelectric conversion element PD is distributed to each of the charge storage units CS1, CS2, CS3, and CS4. Charges are discharged from the photoelectric conversion element PD in periods other than the accumulation period.

Then, the pixel drive circuit 322 sequentially applies voltage signals from all the pixel circuits 321 arranged in the light-receiving unit 3 in units of rows (horizontal arrangement) of the pixel circuits 321 to A/D conversion processing and the like. signal processing.
After that, the pixel driving circuit 322 sequentially outputs the voltage signals after the signal processing to the distance calculating section 42 in the order of the columns arranged in the light receiving section 3 .

The accumulation of charges in the charge accumulation unit CS by the pixel drive circuit 322 and the discarding of the charges photoelectrically converted by the photoelectric conversion elements PD as described above are repeatedly performed over one frame. As a result, charges corresponding to the amount of light received by the distance image pickup device 1 during a predetermined time interval are accumulated in each of the charge accumulation units CS. The pixel drive circuit 322 outputs to the distance calculation section 42 an electrical signal corresponding to the amount of charge for one frame accumulated in each of the charge accumulation sections CS.

Due to the relationship between the timing of irradiating the light pulse PO and the timing of accumulating electric charge in each of the charge accumulating units CS (CS1 to CS4), the charge accumulating unit CS1 receives light such as background light before irradiation of the light pulse PO. A charge amount corresponding to the external light component is held. Further, the charge amounts corresponding to the reflected light RL and the external light component are distributed and held in the charge storage units CS2, CS3, and CS4. The distribution (distribution ratio) of the amount of charge distributed to the charge storage units CS2 and CS3 or to the charge storage units CS3 and CS4 is the delay time until the light pulse PO is reflected by the object S and is incident on the distance image pickup device 1. It becomes a ratio according to Td.

Returning to FIG. 1, the distance calculator 42 uses this principle to calculate the delay time Td by the following equation (1) or (2).
Td=To×(Q3-Q1)/(Q2+Q3-2×Q1) (1)
Td = To + To x (Q4 - Q1) / (Q3 + Q4 - 2 x Q1) (2)
Here, To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge storage section CS1, Q2 is the amount of charge accumulated in the charge storage section CS2, and Q3 is the amount of charge accumulated in the charge storage section CS3. A charge amount Q4 indicates the charge amount accumulated in the charge accumulation section CS4. For example, when Q4=Q1, the distance calculation unit 42 calculates the delay time Td using equation (1), and when Q2=Q1, calculates the delay time Td using equation (2).

In the formula (1), charges generated by reflected light are accumulated in the charge accumulation units CS2 and CS3, but are not accumulated in the charge accumulation unit CS4. On the other hand, in the formula (2), charges generated by reflected light are accumulated in the charge accumulation units CS3 and CS4, but are not accumulated in the charge accumulation unit CS2.
Note that, in the formula (1) or the formula (2), among the charge amounts accumulated in the charge accumulation units CS2, CS3, and CS4, the component corresponding to the external light component is the charge amount accumulated in the charge accumulation unit CS1. It is assumed that the amounts are the same.

The distance calculator 42 calculates the round-trip distance to the subject S by multiplying the delay time obtained by the formula (1) or (2) by the speed of light (velocity).
Then, the distance calculation unit 42 halves the calculated round-trip distance (delay time Td×c (light speed)/2), so that the distance image sensor 32 (that is, the distance image capturing device 1) to the object S is obtained.

FIG. 4 is a diagram showing an example of the arrangement (layout pattern) of each transistor of the pixel circuit 321 in this embodiment.
The layout pattern of FIG. 4 shows the layout pattern of the pixel circuit 321 of FIG. 3 (that is, the pixel circuit 321 of FIG. 2).
4, transfer transistors G1, G2, G3 and G4; source follower transistors SF1, SF2, SF3 and SF4; selection transistors SL1, SL2, SL3 and SL4; reset transistors RT1, RT2, RT3 and RT4; , the pattern arrangement of the charge discharge transistors GD1 and GD2, and the photoelectric conversion element PD. Each of the transistors described above is an n-channel MOS transistor formed on a p-type semiconductor substrate.

For example, the reset transistor RT1 is composed of a drain RT1_D (n diffusion layer (n-type impurity diffusion layer)), a source RT1_S (n diffusion layer), and a gate RT1_G on a p-type semiconductor substrate. .
A contact RT1_C is a pattern indicating a contact connected to a wiring (not shown) provided in each diffusion layer of the drain RT1_D (n diffusion layer) and the source RT1_S (n diffusion layer) of the reset transistor RT1. . Other transfer transistors G1 to G4, source follower transistors SF1 to SF4, selection transistors SL1 to SL4, reset transistors RT2 to RT4, and charge discharge transistors GD1 and GD2 have the same configuration.
The photoelectric conversion element PD is formed in a rectangular shape, and is composed of a long side PDL1, a PDL2 facing parallel to the long side PDL1, a short side PDS1, and a PDS2 facing parallel to the short side PDS1.

Also, the transfer transistor G1 is formed of a floating diffusion FD1 as a drain, a gate G1_G, and a source (n diffusion layer of the photoelectric conversion element PD). Here, the floating diffusion FD1 is a diffusion layer (n-diffusion layer) as a drain of the transfer transistor G1, and accumulates charges from the photoelectric conversion element PD.
Also, the drain G1_D is connected to the gate SF1_G of the source follower transistor SF1 and the source RT1_S of the reset transistor RT1 through a wiring (not shown) by a contact G1_C. Each of the other transfer transistors G2, G3 and G4 has the same configuration as the transfer transistor G1.

FIG. 4 shows the arrangement of each transistor on the semiconductor substrate of the pixel circuit 321, omitting the wiring pattern and the charge storage capacitors (C1 to C4). Therefore, each of the charge storage units CS1, CS2, CS3 and CS4 is arranged at the position of each of the floating diffusions FD1, FD2, FD3 and FD4.

In FIG. 4, for example, the floating diffusion FD1 in the charge storage section CS and each of the transfer transistor G1, the select transistor SL1, the source follower transistor SF1 and the reset transistor RT1 in the vicinity of the charge storage section CS1 has a pn junction. When irradiated with incident light, charges that become noise charges are generated.
Since the charge storage section CS1 has been reset to the voltage of the power supply VRD by the reset transistor and is in a high potential state, noise charges are injected into the charge storage section CS1 due to the electric field caused by the potential difference. .

As a result, noise charges are accumulated in the charge accumulation unit CS1. In addition, all the charge storage units CS1 to CS4 are continuously irradiated with the incident light during the period during which the charges are stored in the charge storage units CS and until the charges are read out from the charge storage units CS. , the ratio of the noise charge in the amount of charge accumulated in the charge accumulation section CS increases.
In addition, similarly to the charge storage section CS1 described above, each of the charge storage sections CS2 to CS3 also receives and stores noise charges generated by itself and other transistors in the vicinity thereof.

Therefore, the accuracy of the distances calculated by the above formulas (1) and (2) is degraded due to noise charges mixed into the charge amounts Q1, Q2, Q3 and Q4 used for calculation.
In this embodiment, as shown in FIG. 5, a process of detecting the amount of noise charges accumulated in the charge accumulation section CS is performed. FIG. 5 is a diagram for explaining the process of acquiring the amount of noise charges accumulated in the charge accumulation unit CS in this embodiment. In FIG. 5, the integration time is the period during which charges are accumulated in the charge storage section CS in a frame, and the previously described charge distribution process from the photoelectric conversion element PD to the charge storage section CS is repeated every predetermined number of accumulation cycles. The read time is a period for reading out the amount of charge accumulated in the charge accumulation section CS in the frame, and the charge amount accumulated in the charge accumulation section CS of each pixel circuit 321 in the distance image sensor 32 is sequentially read out. and output to the distance calculator 42 .
The measurement control unit 43 controls the timing control unit 41 to control the output timing of each of the accumulation drive signals TX1, TX2, TX3 and TX4 from the pixel drive circuit 322 and the drive signal RSTD in the noise charge amount acquisition mode and the output timing of the drive signal RSTD. Control is performed to change between the ranging charge amount acquisition modes.

Here, the noise charge amount acquisition mode indicates a mode of processing performed in the first frame when distance measurement is started, and the amount of noise charge QNS1 accumulated in each of the charge accumulation units CS1 to CS4 is This is the mode for obtaining each QNS4 from .
On the other hand, the distance measurement charge amount acquisition mode is a mode for acquiring the charge amount of the charge generated by the photoelectric conversion element PD due to the incident light including the reflected light RL, which is accumulated in each of the charge storage units CS1 to CS4.
Then, when the distance measurement is started in the distance image pickup device 1, the measurement control unit 43 sets the operation of the pixel driving circuit 322 in the first frame to the noise charge amount acquisition mode. The timing control unit 41 is controlled so that the drive circuit 322 operates in the distance measurement charge amount acquisition mode.

FIG. 6 is a conceptual diagram illustrating charge accumulation processing for the charge accumulation units CS (CS1, CS2, CS3, and CS4) in the noise charge amount acquisition mode.
FIG. 6A is a timing chart showing timings of light emission of the light pulse PO, each of the accumulation drive signals TX1, TX2, TX3 and TX4, and the drive signal RSTD in the noise charge amount acquisition mode. In FIG. 6A, the horizontal axis indicates time, and the vertical axis indicates signal level (H level (ON) or L level (OFF)).
FIG. 6B is a simplified schematic diagram of FIG. 4, showing the concept of the configuration of the photoelectric conversion element PD, the transfer transistors G1, GS2, GS3 and GS4, and the charge storage units CS1, CS2, CS3 and CS4. is.

As shown in FIG. 6A, in the noise charge amount acquisition mode, all of the accumulation drive signals TX1, TX2, TX3, and TX4 are set to L level during the integration time, that is, in all the accumulation cycles in the charge accumulation period. Therefore, the transfer transistors G1, G2, G3, and G4 are kept off.
At this time, the light pulse PO is emitted at a predetermined time in each accumulation period, the drive signal RSTD is set to H level, and the charge discharge transistors GD1 and GD2 are maintained in the ON state.

As a result, all the charges generated by the photoelectric conversion element PD are discharged to the power supply VDD. Since each of the transfer transistors G1, G2, G3, and G4 is in an off state, the charge from the photoelectric conversion element PD is not accumulated in each of the charge accumulation units CS1, CS2, CS3, and CS4.
Therefore, the charge accumulated in the charge storage section CS is a combination of the charge generated in the charge storage section CS and the charge generated in the circuit near the charge storage section CS by incident light (background light and reflected light RL). noise charge.
Then, in each of the charge storage units CS1, CS2, CS3, and CS4, the noise charges of the noise charge amounts QNS1, QNS2, QNS3, and QNS4 are accumulated during the integration time.

Further, the pixel drive circuit 322 reads the noise charge amounts QNS1, QNS2, QNS3, and QNS4 of the charge storage units CS1, CS2, CS3, and CS4 from each of the pixel circuits 321 during the read time, and converts them into predetermined electrical signals to determine the distance. Output to the calculation unit 42 . Here, for simplification of explanation, the noise charge amounts QNS1, QNS2, QNS3, and QNS4 will also be explained in the following explanation.
The distance calculation unit 42 writes and stores the noise charge amounts QNS1, QNS2, QNS3, and QNS4 for each pixel circuit 321 in a storage unit (for example, provided in the distance image processing unit 4) not shown.
With the above-described processing, the noise charge amount acquisition mode ends.

FIG. 7 is a conceptual diagram illustrating charge accumulation processing for the charge accumulation units CS (CS1, CS2, CS3, and CS4) in the distance measurement charge amount acquisition mode.
FIG. 7(a) is a timing chart showing the timing of the light emission of the light pulse PO, each of the accumulation drive signals TX1, TX2, TX3 and TX4, and the drive signal RSTD in the distance measurement charge amount acquisition mode. In FIG. 7A, the horizontal axis indicates time, and the vertical axis indicates signal level (H level (ON) or L level (OFF)).
FIG. 7B is a simplified conceptual diagram of FIG. 4, showing the concept of the configuration of the photoelectric conversion element PD, the transfer transistors G1, G2, G3 and G4, and the charge storage units CS1, CS2, CS3 and CS4. is.

As shown in FIG. 7(a), in the range-finding charge amount acquisition mode, each of the accumulation drive signals TX1 to TX4 is at H level/L at a predetermined timing during the integration time, that is, in all the accumulation cycles in the charge accumulation period. level, the transfer transistors G1, G2, G3, and G4 are controlled to be on/off, and charges are distributed from the photoelectric conversion element PD to the charge storage units CS1, CS2, CS3, and CS4, respectively.
At this time, the light pulse PO is emitted at a predetermined time in each accumulation cycle, and the drive signal RSTD is set to L level during the charge allocation period to the charge accumulation units CS1, CS2, CS3, and CS4, and the charge is discharged. The transistors GD1 and GD2 are kept off.

As a result, the charges generated by the photoelectric conversion elements PD are transferred to the charge storage units CS1, CS2, CS3, and CS4 by the transfer transistors G1, G2, G3, and G4, respectively. , CS4.
Therefore, the charge amount Q of the charges accumulated in the charge storage portion CS is generated in the charge storage portion CS by incident light (background light and reflected light RL) with respect to the charge distributed from the photoelectric conversion element PD. Charges and noise charges, which are charges generated in circuits near the charge storage section CS, are mixed.
That is, the noise charge amounts QNS1, QNS2, QNS3 and QNS4 are mixed in the charge amounts Q1, Q2, Q3 and Q4 accumulated in the charge accumulation units CS1, CS2, CS3 and CS4, respectively.

Further, the pixel drive circuit 322 reads the charge amounts Q1, Q2, Q3, and Q4 respectively accumulated in the charge accumulation units CS1, CS2, CS3, and CS4 from each of the pixel circuits 321 during the read time, It is output to the distance calculator 42 as a signal. Here, for the sake of simplification of explanation, the charge amounts Q1, Q2, Q3, and Q4 will also be explained in the following explanation.
The distance calculator 42 uses the charge amounts Q1, Q2, Q3, and Q4 supplied from the distance image sensor 32 to calculate the distance image sensor 32 using equation (1) or equation (2), as described above. (that is, each of the pixel circuits 321) and the subject S, the delay time Td corresponding to the distance is calculated.

At this time, the distance calculation unit 42 reads the noise charge amounts QNS1, QNS2, QNS3, and QNS4 stored in the storage unit corresponding to each pixel circuit 321 .
Then, the distance calculator 42 subtracts the noise charge amounts QNS1, QNS2, QNS3, and QNS4 from the charge amounts Q1, Q2, Q3, and Q4, respectively, to calculate the corrected charge amounts QC1, QC2, QC3, and QC4.
That is, the corrected charge amount QC1 (=Q1-QNS1) is the charge amount supplied from the photoelectric conversion element PS by the sorting process and stored in the charge storage unit CS1, excluding the noise charge amount QNS1 from the charge amount Q1. . Similarly, the corrected charge amount QC2 (=Q2-QNS2) is the charge amount supplied from the photoelectric conversion element PS by the sorting process and stored in the charge storage unit CS2, which is obtained by removing the noise charge amount QNS2 from the charge amount Q2. be. The corrected charge amount QC3 (=Q3-QNS3) is the charge amount supplied from the photoelectric conversion element PS by the sorting process and stored in the charge storage section CS3, which is obtained by subtracting the noise charge amount QNS3 from the charge amount Q3. The corrected charge amount QC4 (=Q4-QNS4) is the charge amount supplied from the photoelectric conversion element PS by the sorting process and stored in the charge storage section CS4, which is obtained by subtracting the noise charge amount QNS4 from the charge amount Q4.

Then, the distance calculation unit 42 calculates the delay time Td by substituting each of the corrected charge amounts QC1, QC2, QC3, and QC4 into the equation (1) or (2) as follows. .
Td=To×(QC3−QC1)/(QC2+QC3−2×QC1) (1)′
Td=To+To×(QC4−QC1)/(QC3+QC4−2×QC1) (2)′
In calculating the distance in each frame after the second frame, the noise charge amounts QNS1, QNS2, QNS3 and QNS4 stored in the storage section are read out and subtracted from the charge amounts Q1, Q2, Q3 and Q4. , corrected charge amounts QC1, QC2, QC3, and QC4 are sequentially obtained, and the distance is calculated using the equation (1)' or (2)'.

With the above-described configuration, according to the present embodiment, in each of the charge storage units CS (CS1, CS2, CS3, and CS4) in advance in the frame of the noise charge amount acquisition mode, and in the circuits near the charge storage units CS, incident The noise charge amount QNS of the noise charge generated by light and accumulated in the charge storage unit CS is acquired, and in subsequent frames in the distance measurement charge amount acquisition mode, the charge storage unit CS to remove the noise charge mixed in the charge amount Q accumulated in the charge accumulation unit CS, and the incident light in the photoelectric conversion element PD By extracting the corrected charge amount QC generated by the above, it is possible to suppress the decrease in accuracy due to the influence of the noise charge, and to calculate the distance between the object S and the distance image sensor 32 with high accuracy.

Further, in this embodiment, when the distance image pickup device 1 starts distance measurement, the first frame is set to be the frame in the noise charge amount acquisition mode. may be used as the frame of the noise charge amount acquisition mode. For example, if a frame group consists of 100 consecutive frames, the first frame of the frame group is set in the noise charge amount acquisition mode, and the second to 99th frames are set in the distance measurement charge amount acquisition mode. The information on the noise charge amount in the storage unit is overwritten with the newly acquired noise charge amounts QNS1, QNS2, QNS3, and QNS4.
Alternatively, a frame in the noise charge amount acquisition mode may be inserted at regular time intervals (eg, every 5 seconds, 1 minute, etc.) to perform distance measurement processing.

In the present embodiment, the noise charge amounts QNS1, QNS2, QNS3, and QNS4 in all the pixel circuits 321 in the distance image sensor 32 are stored in the storage unit. The average value of the noise charge amounts QNS1, QNS2, QNS3, and QNS4 of each of the charge storage units CS1, CS2, CS4, and CS4 is calculated, and the average value is used as the noise charge of all the charge storage units CS1, CS2, CS4, and CS4. may be configured.
Further, each of the pixel circuits 321 in the range image sensor 32 is divided into pixel circuit groups each including a predetermined number of pixel circuits 321, the pixel circuits 321 in the range image sensor 32 are divided, and any pixel circuit 321 in each group is divided into groups. The noise charge amounts QNS1, QNS2, QNS3, and QNS4 of the pixel circuits 321 may be used as representative values, or the average values thereof may be obtained and used by groups.

<Second embodiment>
A second embodiment of the present invention will be described below with reference to the drawings.
The second embodiment has the same configuration as the first embodiment shown in FIGS. The operation of the second embodiment, which differs from that of the first embodiment, will be described below.
FIG. 8 is a diagram for explaining the process of acquiring the amount of noise charges accumulated in the charge accumulation unit CS in this embodiment.
In FIG. 8, the integration time is the period for accumulating charges in the charge accumulating portion CS in a frame, and the previously described process of distributing charges from the photoelectric conversion element PD to the charge accumulating portion CS is repeated. The read time is a period for reading out the amount of charge accumulated in the charge storage section CS in the frame, and the charge amount of the charge accumulated in each charge storage section CS of the pixel circuit 321 in the distance image sensor 32 is read out sequentially. . In this embodiment, a frame has a frame width of 33.3 msec (milliseconds) and a frame period of 30 frames per second (30 fps (frames per second)).

In the second embodiment, each frame is divided into a first subframe and a second subframe as shown in FIG. The distance measurement charge amount acquisition mode is processed in two subframes. The processing in the noise charge amount acquisition mode and the processing in the distance measurement charge amount acquisition mode are the same as in the first embodiment.
An integration time and a lead time are provided in each of the first subframe and the second subframe, respectively.

During the integration time in the first subframe, the noise charge amounts QNS1, QNS2, QNS3, and QNS4 are obtained, read during the lead time, and output to the distance image processing section 4. FIG. As a result, the distance calculation unit 42 writes and stores the input noise charge amounts QNS1, QNS2, QNS3, and QNS4 in the storage unit.
Then, the charge amounts Q1, Q2, Q3, and Q4 are acquired during the integration time in the second subframe, read during the lead time, and output to the distance image processing section 4. FIG.

As a result, the distance calculation unit 42 subtracts the noise charge amounts QNS1, QNS2, QNS3, and QNS4 from the input charge amounts Q1, Q2, Q3, and Q4, respectively, to obtain corrected charge amounts QC1, QC2, QC3, and QC4. Ask for
Further, the distance calculation unit 42 calculates the distance from the distance image sensor 32 to the subject S by the formula (1)' or (2)' based on the calculated corrected charge amounts QC1, QC2, QC3, and QC4. Obtain the delay time Td.

Here, the lead time can be shortened because the noise charge amounts QNS1, QNS2, QNS3 and QNS4 or the charge amounts Q1, Q2, Q3 and Q4 are read out from each of the pixel circuits 321 in the distance image sensor 32. Can not.
Therefore, in order to maintain the read time for reading charges from the charge storage section CS, the integration time for storing charges in the charge storage section CS is shortened (the number of storage cycles is reduced).

With the above-described configuration, according to this embodiment, noise charge amount acquisition and distance measurement are performed in the same frame. can be removed from the charge amount Q in real time, the influence of the noise charge can be suppressed in response to changes in ambient light, and the accuracy of the distance to be measured can be improved. .

<Third Embodiment>
A third embodiment of the present invention will be described below with reference to the drawings.
The configuration of the third embodiment is similar to that of the first embodiment shown in FIGS. The operation of the third embodiment, which differs from that of the first embodiment, will be described below.
FIG. 9 is a diagram for explaining the process of acquiring the amount of noise charges accumulated in the charge accumulation unit CS in this embodiment. In FIG. 9, the integration time is the period for accumulating charges in the charge accumulating portion CS in a frame, and the previously described process of distributing charges from the photoelectric conversion element PD to the charge accumulating portion CS is repeated. The read time is a period for reading out the amount of charge accumulated in the charge storage section CS in the frame, and the charge amount of the charge accumulated in each charge storage section CS of the pixel circuit 321 in the distance image sensor 32 is read out sequentially. .

In this embodiment, as shown in FIG. 9, under the control of the measurement control unit 43, a frame of the first noise charge amount acquisition mode (first frame) and a second noise charge amount acquisition mode frame (first frame) are used as frames of the noise charge amount acquisition mode. Two types are used: the frame in the quantity acquisition mode (second frame). Also, the frames in the distance measurement charge amount acquisition mode are the frames after the third frame. In this embodiment, as in the first embodiment, when the imaging of the distance image is started, the noise charge amount acquisition mode may be set for the first 1 frame and 2 frames. A configuration may be employed in which the first noise charge amount acquisition mode and the second noise charge amount acquisition mode are executed between time-series frame blocks of a plurality of distance measurement charge amount acquisition modes.

In the present embodiment, the first noise charge amount acquisition mode is a mode for acquiring the first noise charge amount generated only by background light as incident light and accumulated in the charge accumulation unit CS.
In the second noise charge amount acquisition mode, the second noise charge amount (corresponding to the noise charge amount in the first and second embodiments) generated by background light and reflected light RL as incident light and accumulated in the charge accumulation unit CS ) in this mode. The third and subsequent frames are frames in the distance measurement charge amount acquisition mode for acquiring the distance image.

FIG. 10 is a conceptual diagram illustrating noise charges corresponding to the reflected light RL in the charge storage section CS in the distance measurement charge amount acquisition mode.
FIG. 10 shows the light emission of the light pulse PO, the reflected light RL, each of the accumulation drive signals TX1, TX2, TX3 and TX4, the drive signal RSTD, and the noise charge amount QNSP (QNSP1, 4 is a timing chart showing the timing with the occurrence of QNSP2, QNSP3, QNSP4). The noise charge amount QNSP is the amount of noise charge generated by the reflected light RL accumulated in the charge accumulation section CS. That is, the noise charge amount QNSP1 is the amount of noise charge due to the reflected light RL accumulated in the charge accumulation section CS1. Further, the noise charge amount QNSP2 is the amount of noise charge due to the reflected light RL accumulated in the charge accumulation section CS2. The noise charge amount QNSP3 is the amount of noise charge due to the reflected light RL accumulated in the charge accumulation section CS3. The noise charge amount QNSP4 is the amount of noise charge due to the reflected light RL accumulated in the charge accumulation section CS4.

As shown in FIG. 10, in the range-finding charge amount acquisition mode, each of the accumulation drive signals TX1, TX2, TX3, and TX4 is at H level/at a predetermined timing during the integration time, that is, in all the accumulation cycles in the charge accumulation period. The transfer transistors G1, G2, G3, and G4 are controlled to be at L level, and the transfer transistors G1, G2, G3, and G4 are controlled to be on/off, and the charges are distributed from the photoelectric conversion element PD to the charge storage units CS1, CS2, CS3, and CS4, respectively.
At this time, the light pulse PO is emitted at a predetermined time in each accumulation cycle, and the drive signal RSTD is set to L level during the charge allocation period to the charge accumulation units CS1, CS2, CS3, and CS4, and the charge is discharged. The transistors GD1 and GD2 are kept off.
At this time, each of the noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 changes according to the number of allocations for each frame in the distance measurement charge amount acquisition mode.

As a result, the charges generated by the photoelectric conversion elements PD are transferred to the charge storage units CS1, CS2, CS3, and CS4 by the transfer transistors G1, G2, G3, and G4, respectively. , CS4 are mixed with noise charges generated by the reflected light RL and noise charges generated by the background light.
Each of the noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 generated by the reflected light RL needs to be obtained each time the number of allocations changes by the auto-exposure function when an object with a significantly different reflectance is imaged. There is
For this reason, it is not possible to predict the occurrence of a change in the number of distributions by the auto exposure, so it is necessary to frequently execute frames in the noise charge amount acquisition mode.
However, frequent processing of frames in the noise charge amount acquisition mode reduces the resolution of the range image on the time axis. correspond to changes in

In the first frame of the first noise charge amount acquisition mode in FIG. 9, the noise charge amount acquisition mode process is performed during the integration time, as in the first embodiment.
However, unlike the noise charge amount acquisition mode of the first embodiment, the measurement control unit 43 does not cause the light source unit 2 to emit the light pulse PO in the first noise charge amount acquisition mode, and the incident light is only the background light, and the noise charge amount QNSN of the noise charges generated by the background light and accumulated in the charge accumulation unit CS is obtained. That is, the distance calculation unit 42 acquires the noise charge amounts QNSN1, QNSN2, QNSN3, and QNSN4 in each of the charge storage units CS1, CS2, CS3, and CS4 from the distance image sensor 32, respectively.
Then, the distance calculation unit 42 writes each of the acquired noise charge amounts QNSN1, QNSN2, QNSN3, and QNSN4 to the storage unit as the noise charge amount (first noise charge amount) acquired in the first noise charge amount acquisition mode. Memorize.

In addition, in the second noise charge amount acquisition mode, the measurement control unit 43 causes the light source unit 2 to emit the light pulse PO in the same manner as in the noise charge amount acquisition mode of the first embodiment. are the background light and the reflected light RL, and the noise charge amount QNS of the noise charges generated by the background light and the reflected light RL and accumulated in the charge accumulation unit CS is obtained. That is, the distance calculator 42 acquires from the distance image sensor 32 the noise charge amounts QNS1, QNS2, QNS3, and QNS4 in the charge storage units CS1, CS2, CS3, and CS4, respectively.
Then, the distance calculation unit 42 writes each of the acquired noise charge amounts QNS1, QNS2, QNS3, and QNS4 to the storage unit as the noise charge amount (second noise charge amount) acquired in the second noise charge amount acquisition mode. Memorize.

Further, the distance calculation unit 42 subtracts the first noise charge amounts QNSN1, QNSN2, QNSN3, and QNSN4 from the second noise charge amounts QNS1, QNS2, QNS3, and QNS4, respectively, and calculates the third noise charge amount QNSP1 (= QNS1-QNSN1), QNSP2 (=QNS2-QNSN2), QNSP3 (=QNS3-QNSN4), and QNSP4 (=QNS4-QNSN4) are obtained.
Each of the third noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 is the noise charge amount of noise charges generated only by the light pulse PO in each of the charge storage units CS1, CS2, CS3, and CS4. Here, each of the third noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 corresponds to the number of allocation times in the second noise charge amount acquisition mode, that is, the number of times the light source unit 2 emits light pulses PO. is the amount of noise charge generated by the reflected light RL.
Then, the distance calculation unit 42 writes and stores each of the calculated third noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 in the storage unit.

The measurement control unit 43 emits the light pulse PO to the light source unit 2 in the third and subsequent frames of the distance measurement charge amount acquisition mode, as in the frames of the distance measurement charge amount acquisition mode of the first embodiment. The background light and the reflected light RL are used as the incident light, and the charges generated in the photoelectric conversion element PD by the background light and the reflected light RL are distributed, and the charges (including noise charges) accumulated in the charge accumulation unit CS are reduced. Get the amount of charge Q. That is, the distance calculation unit 42 outputs the charge amounts Q1, Q2, Q3, and Q4 to which the charges generated by the photoelectric conversion elements PD in each of the charge storage units CS1, CS2, CS3, and CS4 are allocated to the distance image sensor 32, respectively. Get from
Then, the distance calculation unit 42 subtracts the noise charge amount from each of the obtained charge amounts Q1, Q2, Q3, and Q4 in the same manner as in the first embodiment, and obtains the corrected charge amounts QC1, QC2, QC3, and QC4. Calculate each.

In the present embodiment, when obtaining each of the corrected charge amounts QC1, QC2, QC3, and QC4, the third noise charge amounts QNSP1, QNSP2, QNSP3, Each QNSP4 is adjusted.
That is, when the distance measuring imaging device has an auto-exposure function, the measurement control unit 43 determines the number of distributions in the integration time, depending on the intensity of the reflected light RL, so that one of the charge storage units CS is saturated. change it to not
Therefore, each of the third noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 changes sequentially according to the number of distributions (the number of times the light pulse is emitted).

When calculating the distance in a frame in the distance measurement charge amount acquisition mode, the distance calculation unit 42 reads the third noise charge amounts QNSP1, QNSP2, QNSP3, and QNSP4 that are stored in the storage unit and are calculated in advance.
Then, the distance calculation unit 42 divides the number of allocations in the frame of the ranging charge amount acquisition mode for which the distance is to be calculated by the number of allocations in the second frame of the second noise charge amount acquisition mode, and calculates the adjustment coefficient k. do.
The distance calculator 42 multiplies each of the third noise charge amounts QNSP1, QNSP2, QNSP3 and QNSP4 by the obtained adjustment coefficient k to calculate the adjusted third noise charge amounts kQNSP1, kQNSP2, kQNSP3 and kQNSP4.

Then, the distance calculation unit 42 adds the adjusted third noise charge amounts kQNSP1, kQNSP2, kQNSP3, and kQNSP4 to each of the first noise charge amounts QNSN1, QNSN2, QNSN3, and QNSN4 to obtain the fourth noise charge amounts. Quantities QN1 (=QNSN1+kQNSP1), QN2 (=QNSN2+kQNSP2), QN3 (=QNSN3+kQNSP3), and QN4 (=QNSN4+kQNSP4) are calculated.
Further, the distance calculation unit 42 subtracts the fourth noise charge amounts QN1, QN2, QN3, and QN4 from the charge amounts Q1, Q2, Q3, and Q4, respectively, to obtain the corrected charge amounts QC1, QC2, QC3, and QC4. calculate.
Then, similarly to the first embodiment, the distance calculation unit 42 calculates the delay time Td used when obtaining the distance between the subject S and the distance image sensor 32 by the formula (1)′ or (2)′. is performed using each of the correction charge amounts QC1, QC2, QC3 and QC4.

With the above-described configuration, according to the present embodiment, each of the charge storage units CS (CS1, CS2, CS3, and CS4) and the vicinity of the charge storage unit CS in advance in the frames of the first and second noise charge amount acquisition modes In the circuit of (1), each of the first noise charge amount QNSN and the third noise charge amount QNSP of the noise charge generated by the incident light and accumulated in the charge accumulation section CS is obtained, and the frame of the distance measurement charge amount acquisition mode is obtained. The adjusted third noise charge amount kQNSP for each frame is calculated using the adjustment coefficient k corresponding to the number of allocations for each, the first noise charge amount QNSN and the adjusted third noise charge amount kQNSP are added, and the result of the addition is By subtracting the fourth noise charge amount QN from the charge amount Q, each of the corrected charge amounts QC1, QC2, QC3 and QC4 is obtained, and the adjustment time is calculated. Since the delay time Td is obtained from the corrected charge amounts QC1, QC2, QC3, and QC4 based on the fourth noise charge amount QN, the distance between the object S and the distance image sensor 32 can be calculated with high accuracy.

Further, according to the present embodiment, in the first and second noise charge amount acquisition modes, the first noise charge amount QNSN and the third noise charge amount QNSP are obtained in the first frame and the second frame. Since the amount of noise charge corresponding to the number of allocations for each frame can be obtained, it is not necessary to perform the process of obtaining the amount of noise charge for each frame as in the second embodiment. The corresponding noise charge amount can be obtained, and the accuracy of the distance to be measured can be easily improved.

FIG. 11 is a flowchart showing an operation example of processing for calculating the distance between the distance image sensor 32 and the subject S in the distance image pickup device 1 according to the third embodiment. In the following description, in each of the pixel circuits 321 in the distance image sensor 32, the distance from the pixel circuit 321 (that is, the distance image sensor 32) to the subject S is calculated.

Step S101:
When the distance image capturing device 1 starts distance measurement, the measurement control unit 43 outputs an instruction to the timing control unit 41 to perform processing in the first noise charge amount acquisition mode.
In the first frame, the timing control unit 41 outputs the accumulation drive signals TX1, TX2, TX3 and TX4 and the drive signal RSTD to acquire the noise charge amount QNSN of the noise charge generated only by the background light. 32.
Thereby, the distance image sensor 32 outputs the first noise charge amount QNSN (QNSN1, QNSN2, QNSN3, QNSN4) acquired in the first frame to the distance calculation section .

Step S102:
After the end of the first frame, the measurement control section 43 outputs an instruction to the timing control section 41 to perform processing in the second noise charge amount acquisition mode.
In the second frame, the timing control unit 41 causes the light source unit 2 to radiate the light pulse PO, and determines the noise charge amount QNS (QNS1, QNS2, QNS3, QNS4) of noise charges generated by the background light and the reflected light RL. It outputs accumulation drive signals TX1, TX2, TX3 and TX4 and a drive signal RSTD to the range image sensor 32 for acquisition.
Thereby, the distance image sensor 32 outputs the second noise charge amount QNS acquired in the second frame to the distance calculation section 42 .

Step S103:
Then, the distance calculator 42 subtracts each of the first noise charges QNSN from each of the second noise charges QNS to calculate third noise charges QNSP (QNSP1, QNSP2, QNSP3, QNSP4).
The distance calculation unit 42 writes and stores the third noise charge amount QNSP together with the first noise charge amount QNSN into the storage unit for each charge accumulation unit CS (CS1, CS2, CS3, CS4) in pixel circuit 321 units.

Step S104:
After the end of the second frame, the measurement control unit 43 outputs an instruction to the timing control unit 41 to perform processing in the range-finding charge amount acquisition mode.
In the third frame (that is, the frames subsequent to the third frame), the timing control unit 41 causes the light source unit 2 to emit a light pulse PO, and measures the distance between the subject S and the distance image sensor 32. As control for distributing charges from the conversion element PD to each charge storage section CS, storage drive signals TX1, TX2, TX3 and TX4 and a drive signal RSTD are output to the distance image sensor 32. FIG.
Thereby, the distance image sensor 32 outputs each of the charge amounts Q (Q1, Q2, Q3, Q4) acquired in the third frame to the distance calculation section 42 .

Step S105:
The distance calculation unit 42 divides the number of allocations for the third frame to be distance-measured by the number of allocations for the second frame to calculate an adjustment coefficient k.

Step S106:
The distance calculation unit 42 reads the third noise charge amount QNSP from the storage unit, multiplies each of the third noise charge amounts QNSP by the adjustment coefficient k, and obtains the adjusted third noise charge amount kQNSP (kQNSP1, kQNSP2, kQNSP3, kQNSP4 ) to calculate each.

Step S107:
The distance calculation unit 42 reads each of the first noise charge amounts QNSN from the storage unit, adds each of the read first noise charge amounts QNSN and the adjusted third noise charge amount kQNSP, and obtains a fourth noise charge amount QN ( QN1, QN2, QN3, QN4) are calculated.

Step S108:
The distance calculation unit 42 subtracts each of the calculated fourth noise charge amounts QN from each of the charge amounts Q to calculate a correction charge amount QC (QC1, QC2, QC3, QC4).

Step S109:
The distance calculation unit 42 calculates the delay time Td by using the obtained corrected charge amount QC by the formula (1)′ or (2)′.

Step S110:
The distance calculator 42 calculates the distance from the distance image sensor 32 to the subject S by multiplying the obtained delay time Td by the high speed and dividing by "2".

Step S111:
The measurement control unit 43 determines whether or not a control signal for ending distance measurement has been input.
At this time, the measurement control unit 43 terminates the processing when a control signal for terminating the distance measurement processing is input, and when the control signal for terminating the distance measurement processing is not input, the processing is stepped. Proceed to S104.

As the configuration of the above-described first, second, and third embodiments, the depth image capturing device using the TOF technique has been described, but the scope of application of the present invention is not limited to this. It can also be applied to a sensor having a structure in which a photodiode supplies one charge storage unit, such as an RGB-IR (Red Green Blue-Infrared) sensor.
In addition, it can be applied to a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc., as long as the charge generated by the photodiode due to incident light is accumulated in the charge accumulation section. .

DESCRIPTION OF SYMBOLS 1... Range image pick-up device 2... Light source part 3... Light-receiving part 31... Lens 32... Range image sensor (range image pick-up element)
321... Pixel circuit 322... Pixel driving circuit 4... Distance image processing unit 41... Timing control unit 42... Distance calculation unit 43... Measurement control unit CS1, CS2, CS3, CS4... Charge accumulation unit FD1, FD2, FD3, FD4... Floating Diffusion G1, G2, G3, G4 Transfer transistor GD Charge discharge transistor ML Microlens PD Photoelectric conversion element PO Optical pulse RT1, RT2, RT3, RT4 Reset transistor S Subject SF1, SF2, SF3, SF4 Source follower transistors SL1, SL2, SL3, SL4... select transistors

Claims (11)

a light source unit that irradiates a light pulse into the measurement space;
a plurality of pixel circuits each having a plurality of photoelectric conversion elements that generate charges according to the light incident from the measurement space; a plurality of charge storage units that store the charges in a frame cycle; a light-receiving unit having a pixel drive circuit for performing on/off processing of each transfer transistor in each of the charge storage units at the accumulation timing, and distributing and accumulating the charges;
a distance calculation unit that calculates the distance between the subject and the light receiving unit in the measurement space based on the charge amount determined by the first charge amount of the charges accumulated in each of the charge accumulation units;
The distance calculation unit calculates the distance by subtracting, from each of the first charge amounts, a second charge amount due to noise charges, which are accumulated charges other than the charges distributed and accumulated by the ON/OFF processing of the transfer transistors. A distance image capturing device characterized by: any one of the frames is a noise charge amount acquisition frame for acquiring the second charge amount;
In the noise charge amount acquisition frame, after the light pulse is irradiated from the light source unit,
the amount of charge accumulated in the charge storage unit in a state in which the pixel drive circuit turns off the transfer transistor and does not distribute the charge from the photoelectric conversion element to the charge storage unit as the second charge amount;
In frames subsequent to the noise charge amount acquisition frame,
2. The distance image pickup according to claim 1, wherein the pixel driving circuit turns on and off the transfer transistor to distribute and accumulate the charge in each of the charge accumulation units to obtain the first charge amount. Device. dividing each of the frames to generate a first subframe and a second subframe;
In the first subframe, after the light pulse is emitted from the light source unit,
the amount of charge accumulated in the charge storage unit in a state in which the pixel drive circuit turns off the transfer transistor and does not store the charge from the photoelectric conversion element in the charge storage unit as the second amount of charge;
In the second subframe,
2. The distance image pickup according to claim 1, wherein the pixel driving circuit turns on and off the transfer transistor to distribute and accumulate the charge in each of the charge accumulation units to obtain the first charge amount. Device. the frame is composed of an accumulation period for distributing and accumulating the electric charge in each of the charge accumulation units, and a readout period for reading the amount of electric charge accumulated from each of the charge accumulation units,
4. The range image pickup apparatus according to claim 3, wherein each of said first sub-frame and said second sub-frame is configured by shortening said accumulation period. In a state in which the light pulse is not irradiated from the light source section, the pixel driving circuit turns off the transfer transistor, and in a state in which the charge is not accumulated in the charge accumulation section from the photoelectric conversion element, the charge is accumulated in the charge accumulation section. The frame for acquiring the charge amount obtained as the third charge amount is the first noise charge amount acquisition frame,
After the light pulse is irradiated from the light source, the pixel driving circuit turns off the transfer transistor, and in a state in which the charge is not accumulated in the charge accumulation section from the photoelectric conversion element, the charge is accumulated in the charge accumulation section. A second noise charge amount acquisition frame is a frame for acquiring the charge amount obtained as the second charge amount,
The distance calculation unit subtracts the third charge amount from the second charge amount to obtain a fourth charge amount, which is the charge amount of the charge generated by the light pulse other than the photoelectric conversion element. The distance image pickup device according to claim 1. The distance calculation unit
With respect to the first number of allocations of the allocation in each of the first noise charge amount acquisition frame and the second noise charge amount acquisition frame, the number of allocations in each of the frames after the second noise charge amount acquisition frame When the second number of distributions changes, the result of dividing the second number of distributions by the first number of distributions is used as an adjustment coefficient, and the fourth charge amount is multiplied by the adjustment coefficient to obtain a fifth charge amount. to calculate
6. The range image pickup apparatus according to claim 5, wherein the fifth charge amount is added to the third charge amount to correct the second charge amount and use it for calculating the distance. 7. The distance imaging apparatus according to claim 1, wherein the light pulse emitted from the light source unit is a pulse of light in a near-infrared wavelength band with a predetermined width. The range imaging device according to any one of claims 1 to 7, wherein the structure of the pixel circuit is a BSI (Back Side Illumination) structure. The range imaging device according to any one of claims 1 to 8, wherein the pixel circuit has three or more charge storage units. 2. The pixel circuit is provided with one or more charge discharge transistors for discharging charges from the photoelectric conversion elements except for a period in which the charges are distributed and accumulated in each of the charge accumulation units. The range imaging device according to claim 9 . A range image capturing method for controlling a range image capturing apparatus comprising: each of a plurality of pixel circuits each composed of a photoelectric conversion element and a plurality of charge storage units; a light source unit; a pixel drive circuit;
a process in which the light source unit irradiates a light pulse into the measurement space;
a process in which the pixel circuit accumulates, in the charge accumulation unit, charges generated by the photoelectric conversion element in response to light incident from the measurement space in a frame period;
a pixel driving process in which the pixel drive circuit turns on and off the transfer transistors in each of the charge storage units at a predetermined accumulation timing synchronized with the irradiation of the light pulse, and distributes and accumulates the charges;
a distance calculation step in which the distance calculation unit calculates the distance between the subject and the light receiving unit in the measurement space from the charge amount determined by the first charge amount of the charges accumulated in each of the charge accumulation units; including
The distance calculation unit calculates the distance by subtracting, from each of the first charge amounts, a second charge amount due to noise charges, which are accumulated charges other than the charges distributed and accumulated by the ON/OFF processing of the transfer transistors. A distance image capturing method characterized by:

Images