JP2022112389A - Distance image pickup device, and distance image pickup method - Google Patents

Abstract

To provide a distance image pickup device and a distance image pickup method that, even when a distance calculated, on the basis of an amount of electric charges accumulated in charge accumulation units, has an error, can perform calculation to bring the distance close to an actual distance.SOLUTION: A distance image pickup device comprises: a light source unit; a light receiving unit that has pixels each provided with a photoelectric converter and a plurality of charge accumulation units, and a pixel drive circuit distributing electric charges to the charge accumulation units to cause the charge accumulation units to accumulate the electric charges; a storage unit that stores table information in which a charge ratio is associated with a corresponding distance to a subject; and a distance image processing unit that determines a measured distance to the subject by using an amount of electric charges accumulated in the charge accumulation units and the table information. In the table information, a table interval between variables in an uneven distribution state is smaller than a table interval between variables not in the uneven distribution state.SELECTED DRAWING: Figure 1

Description

The present invention relates to a range image capturing device and a range image capturing method.

Conventionally, as a technique for measuring the distance to an object, there is a technique for measuring the flight time of light pulses. Such a technique is called Time of Flight (hereinafter referred to as TOF). TOF utilizes the fact that the speed of light is known, and irradiates an object with light pulses in the near-infrared region. Then, the time difference between the time when the light pulse is applied and the time when the reflected light of the applied light pulse is received by the object is measured. The distance to the object is calculated based on this time difference. 2. Description of the Related Art Ranging sensors that detect light for measuring distance using photodiodes (photoelectric conversion elements) have been put to practical use.

In recent years, distance measuring sensors have been put into practical use that can obtain depth information for each pixel in a two-dimensional image including the object, that is, three-dimensional information about the object, in addition to the distance to the object. Such a distance measuring sensor is also called a distance image pickup device. In the depth image pickup device, a plurality of pixels including photodiodes are arranged in a two-dimensional matrix on a silicon substrate, and light reflected by an object is received by the pixel surface. A depth image pickup device outputs a photoelectric conversion signal for one image based on the amount of light (charge) received by each pixel. distance information can be obtained. For example, Japanese Laid-Open Patent Publication No. 2004-100003 discloses a technique in which three charge storage units are provided in one pixel and the charges are distributed in order to calculate the distance.

Japanese Patent No. 4235729

In such a distance image pickup device, pixels receive reflected light reflected by an object, photoelectrically convert the light amount of the received reflected light into electric charge, accumulate the converted electric charge in the electric charge accumulation unit, and store the accumulated electric charge. Calculate distance information based on the quantity. However, the distance calculated from the accumulated charge amount may have an error with respect to the actual distance (actual distance).

The present invention has been made based on the above problem, and even if there is an error in the distance calculated based on the amount of charge accumulated in the charge accumulation unit, calculation is performed so as to approximate the actual distance. It is an object of the present invention to provide a distance image pickup device and a distance image pickup method capable of capturing a distance image.

The distance image pickup device of the present invention includes a light source unit that irradiates a light pulse into a measurement space in which an object exists, a photoelectric conversion element that generates charges according to the incident light, and a plurality of charge storage units that store the charges. and a pixel driving circuit for distributing and accumulating the charge in each of the charge accumulating portions of the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a storage unit for storing table information in which corresponding distances to a subject are associated; and a distance image for determining a measured distance to the subject using the amount of charges accumulated in each of the charge storage units and the table information. and a processing unit, wherein the charge ratio is set for each of two or more charge storage units in which the charge corresponding to the light reflected by the light pulse from the subject is distributed and accumulated among the plurality of charge storage units. is a ratio indicated by using the distance calculation charge amount obtained by subtracting the charge amount corresponding to the external light component from the charge amount accumulated in the table information, and the table information includes the charge ratio included in the table information and the corresponding distance The charge ratio exceeds a predetermined upper limit value or falls below a predetermined lower limit value with respect to the table interval corresponding to the difference between the adjacent variables when each of the variables is arranged in ascending or descending order. The table interval of the variable in the unevenly distributed state is smaller than the table interval of the variable not in the unevenly distributed state. The corresponding distance corresponding to the calculated charge ratio is obtained from the table information, and the measured distance is determined using the obtained corresponding distance.

In the distance image pickup device of the present invention, the charge storage unit in which the charge corresponding to the reflected light reflected by the light pulse from the subject is first stored among the plurality of charge storage units is a first charge storage unit, and A second charge storage portion is defined as a charge storage portion in which charges corresponding to reflected light are stored next to the first charge storage portion, and the first charge amount stored in the first charge storage portion corresponds to the external light component. A charge amount obtained by subtracting the charge amount obtained by subtracting the charge amount for the first distance calculation is defined as a first distance calculation charge amount. The charge ratio is a ratio of the second distance calculation charge amount to the sum of the first distance calculation charge amount and the second distance calculation charge amount, and the uneven distribution state is , the charge ratio is above a predetermined upper limit or below a predetermined lower limit.

In the distance image capturing apparatus of the present invention, each of the variables included in the table information has a first range in which the charge ratio is less than a first threshold, and a range in which the charge ratio is greater than or equal to the first threshold and less than a second threshold. A value included in either the second range or the third range in which the charge ratio is equal to or greater than the second threshold and less than the third threshold, and the table interval in the third range is the value in the second range smaller than the table interval.

In the distance image capturing apparatus of the present invention, each of the variables included in the table information has a first range in which the charge ratio is less than a first threshold, and a range in which the charge ratio is greater than or equal to the first threshold and less than a second threshold. A value included in either the second range or the third range in which the charge ratio is greater than or equal to the second threshold and less than the third threshold, and the table interval in the first range is the value in the second range smaller than the table interval.

In the distance image pickup device of the present invention, the distance image processing unit calculates the charge ratio based on the amount of charge accumulated in each of the plurality of charge accumulation units, and corresponds to the charge ratio smaller than the calculated charge ratio. The attached first distance and the second distance associated with the charge ratio greater than the calculated charge ratio are extracted from the table information, and the extracted first distance and the extracted second distance are linearly interpolated. to determine the measured distance.

In the distance image pickup device of the present invention, the pixel includes three or more of the charge storage units, and the distance image processing unit calculates the amount of charge accumulated in each of the three or more charge storage units , the smallest charge amount is set as the charge amount corresponding to the external light component.

In the distance image pickup device of the present invention, the pixel includes three or more charge storage units, and the distance image processing unit stores predetermined external light from among the three or more charge storage units. The amount of charge accumulated in the external light charge accumulating unit is controlled by controlling the timing of accumulating the charge in the external light accumulating charge accumulating unit so that the charge corresponding to the reflected light is not accumulated in the external light charge accumulating unit. is the charge amount corresponding to the external light component.

In the range image pickup device of the present invention, the pixel is provided with three charge storage sections, namely, a first charge storage section, a second charge storage section, and a third charge storage section, and the range image processing section includes controlling the first charge storage unit, the second charge storage unit, and the third charge storage unit so that the charges are stored in this order at timing synchronized with the irradiation of the light pulse; A first calculated value that is a difference between the first charge amount and the third charge amount, using the first charge amount accumulated in the accumulation unit and the third charge amount accumulated in the third charge accumulation unit is calculated, and the first calculated value is obtained from two distance-calculating charge storage units in which the amount of charge corresponding to the reflected light reflected by the light pulse from the subject is divided and stored among the three charge storage units. The charge amount corresponding to the reflected light accumulated in one of them is assumed.

In the distance image pickup device of the present invention, the pixel is provided with a first charge storage unit, a second charge storage unit, a third charge storage unit, and a fourth charge storage unit, which are the four charge storage units, The distance image processing unit stores the charges in the order of the first charge storage unit, the second charge storage unit, the third charge storage unit, and the fourth charge storage unit at a timing synchronized with the irradiation of the light pulse. controlling the pixel drive circuit to accumulate the first charge using the first charge amount accumulated in the first charge accumulation unit and the third charge amount accumulated in the third charge accumulation unit; calculating a first calculated value that is a difference between the amount of charge and the amount of third charge, and calculating the amount of second charge accumulated in the second charge accumulation section and the amount of fourth charge accumulated in the fourth charge accumulation section; and calculating a second calculated value, which is the difference between the second charge amount and the fourth charge amount, and adding the absolute value of the first calculated value and the absolute value of the second calculated value The reflected light accumulated in each of two distance-calculating charge accumulators, among the four charge accumulators, in which the amount of charge corresponding to the reflected light reflected by the light pulse from the subject is divided and accumulated. , the first calculated value is the sum of the distance calculation charge amounts corresponding to the distance calculation charge amounts, the first calculated value is the distance calculation charge amount of one of the two distance calculation charge storage units, and the second calculation value is the sum of the two distance calculation charge amounts. The distance calculation charge amount in the other distance calculation charge accumulating section is calculated.

In the distance image pickup device of the present invention, the pixel includes three or more charge storage units, and the light pulses among the three or more charge storage units are charged corresponding to reflected light reflected from the subject. A first charge storage unit is a charge storage unit in which is stored first, a second charge storage unit is a charge storage unit in which a charge corresponding to the reflected light is stored next to the first charge storage unit, and the storage the distance image processing unit stores the table information for each combination of the first charge storage unit and the second charge storage unit, and the distance image processing unit stores the timing of storing the charge among the three or more charge storage units. The first charge storage unit and A combination of the second charge storage units is determined, and the table information is selected according to the determined combination of the first charge storage units and the second charge storage units.

In the distance image pickup device of the present invention, the charge storage unit in which the charge corresponding to the reflected light reflected by the light pulse from the subject is first stored among the plurality of charge storage units is a first charge storage unit, and A second charge storage unit is defined as a charge storage unit in which charges corresponding to reflected light are stored next to the first charge storage unit, and the storage unit stores charges corresponding to the reflected light reflected by the subject at a short distance. Short-distance information, which is the table information corresponding to the combination of the first charge storage section and the second charge storage section in which the amount is accumulated, and the charge amount corresponding to the reflected light reflected by the subject at a long distance. long-distance information, which is the table information corresponding to the combination of the first charge storage unit and the second charge storage unit in which is stored, and each of the variables included in the short-range information is A first range in which the charge ratio is less than the first threshold, a second range in which the charge ratio is greater than or equal to the first threshold and less than the second threshold, and a second range in which the charge ratio is greater than or equal to the second threshold and less than the third threshold. A value included in one of three ranges, the table interval in the third range is smaller than the table interval in the second range, and each of the variables included in the long-distance information is the A fourth range in which the charge ratio is less than the fourth threshold, a fifth range in which the charge ratio is greater than or equal to the fourth threshold and less than the fifth threshold, and a sixth range in which the charge ratio is greater than or equal to the fifth threshold and less than the sixth threshold. The table interval in the fourth range is smaller than the table interval in the fifth range.

A distance image capturing method of the present invention includes a light source unit that irradiates a measurement space in which an object exists with a light pulse, a photoelectric conversion element that generates charges according to the incident light, and a plurality of charge storage units that store the charges. and a pixel driving circuit for distributing and accumulating the charge in each of the charge accumulating portions of the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a storage unit that stores table information in which the corresponding distance to the object is associated with each variable based on the measurement; and a distance image processing unit for determining the distance image capturing method, wherein the charge ratio corresponds to reflected light reflected from the subject by the light pulse among the plurality of charge storage units. It is a ratio indicated by using the charge amount for distance calculation obtained by subtracting the charge amount corresponding to the external light component from the charge amount accumulated in each of two or more charge accumulation units in which the charge is distributed and accumulated, and is shown in the table. The information includes the charge ratio and the corresponding distance for a table interval corresponding to the difference between the adjacent variables when the variables that are either the charge ratio or the corresponding distance included in the table information are arranged in ascending or descending order. , the table interval of the variables in an unevenly distributed state that is above a predetermined upper limit value or below a predetermined lower limit value is smaller than the table interval of the variables that are not in the unevenly distributed state, and the distance image processing unit calculating the charge ratio based on the amount of charge accumulated in each of the plurality of charge accumulation units, obtaining the corresponding distance corresponding to the calculated charge ratio from the table information, and obtaining the obtained corresponding distance is used to determine the measured distance.

According to the present invention, even if there is an error in the distance calculated based on the amount of charge accumulated in the charge accumulation section, it is possible to calculate the distance closer to the actual distance.

1 is a block diagram showing a schematic configuration of a distance imaging device 1 of an embodiment; FIG. 3 is a block diagram showing a schematic configuration of a distance image sensor 32 of the embodiment; FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 of the embodiment; FIG. 4 is a timing chart showing an example of timing for driving a pixel 321 of the embodiment; It is a figure which shows the example of a structure of the table information 440 of embodiment. It is a figure explaining the table information 440 of embodiment. FIG. 10 is a diagram for explaining a process in which the distance image processing unit 4 of the embodiment performs linear interpolation using table information 440; FIG. 4 is a diagram for explaining processing for determining an external light component by the distance image processing unit 4 of the embodiment; FIG. 4 is a diagram for explaining processing for determining an external light component by the distance image processing unit 4 of the embodiment; 4 is a flow chart showing the flow of processing performed by the distance image processing unit 4 of the embodiment; It is a figure explaining the effect of embodiment. It is a figure explaining two time windows in the modification of embodiment. It is a figure which shows the example of a structure of 440 A of table information in the modification of embodiment. It is a figure which shows the example of a structure of the table information 440B in the modification of embodiment. It is a figure explaining the table information 440B in the modification of embodiment. It is a figure which shows the example of a structure of 440 C of table information in the modification of embodiment. It is a figure explaining 440 C of table information in the modification of embodiment.

Hereinafter, the distance image capturing device of the embodiment will be described with reference to the drawings.

<Embodiment>
First, an embodiment will be described. FIG. 1 is a block diagram showing a schematic configuration of a distance image pickup device according to a first embodiment of the present invention. A distance image pickup device 1 configured as shown in FIG. FIG. 1 also shows an object OB, which is an object whose distance is to be measured in the distance image pickup device 1 .

Under the control of the distance image processing unit 4 , the light source unit 2 irradiates the space of the imaging target in which the object OB whose distance is to be measured in the distance imaging device 1 exists, with a light pulse PO. The light source unit 2 is, for example, a surface emitting semiconductor laser module such as a vertical cavity surface emitting laser (VCSEL: Vertical Cavity Surface Emitting Laser). The light source unit 2 includes a light source device 21 and a diffuser plate 22 .

The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) as a light pulse PO to irradiate the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting device. The light source device 21 emits pulsed laser light under the control of the timing control section 41 .

The diffuser plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 over a surface that irradiates the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO, and is irradiated onto the object OB.

The light receiving unit 3 receives the reflected light RL of the light pulse PO reflected by the object OB whose distance is to be measured in the distance image pickup device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving section 3 includes a lens 31 and a distance image sensor 32 .

The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32 . The lens 31 emits the incident reflected light RL to the distance image sensor 32 side, and causes pixels provided in the light receiving area of the distance image sensor 32 to receive the light (incident).

The distance image sensor 32 is an image pickup device used in the distance image pickup device 1 . The distance image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the distance image sensor 32 is provided with one photoelectric conversion element, a plurality of charge storage units corresponding to this one photoelectric conversion element, and a component for distributing the charge to each charge storage unit. . In other words, the pixel is an imaging device having a distribution structure in which charges are distributed and accumulated in a plurality of charge accumulation units.

The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge storage units according to control from the timing control unit 41 . Also, the distance image sensor 32 outputs a pixel signal corresponding to the amount of charge distributed to the charge storage section. A plurality of pixels are arranged in a two-dimensional matrix in the range image sensor 32, and pixel signals for one frame corresponding to each pixel are output.

The distance image processing unit 4 controls the distance image capturing device 1 and calculates the distance to the subject OB. The distance image processing section 4 includes a timing control section 41 , a distance calculation section 42 , a measurement control section 43 and a storage section 44 .

The timing control section 41 controls the timing of outputting various control signals required for measurement under the control of the measurement control section 43 . The various control signals here include, for example, a signal for controlling the irradiation of the light pulse PO, a signal for allocating and accumulating the reflected light RL to a plurality of charge accumulation units, and controlling the number of times of distribution (number of times of accumulation) per frame. such as a signal to The number of distributions is the number of repetitions of the process of distributing charges to the charge storage section CS (see FIG. 3). The exposure time is the product of the number of times of electric charge distribution and the time (accumulation time Ta, which will be described later) for accumulating electric charges in each electric charge accumulating section for each electric charge distributing process.

The distance calculator 42 calculates the distance to the subject OB based on the pixel signals output from the distance image sensor 32, and outputs the calculated distance information. A distance calculation unit 42 calculates the distance to the subject OB based on the charge amounts accumulated in the plurality of charge accumulation units.

In this embodiment, the distance calculator 42 determines the distance to the object OB using table information 440, which will be described later. The table information 440 will be explained later in detail. A method for determining the distance to the object OB by the distance calculation unit 42 using the table information 440 will be described in detail later.

The measurement control section 43 controls the timing control section 41 . For example, the measurement control unit 43 sets the number of allocation times for one frame, the accumulation time Ta, and the like, and controls the timing control unit 41 so that imaging is performed according to the set contents.

The storage unit 44 is a storage medium such as a HDD (Hard Disk Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read/write Memory), ROM (Read Only Memory), or any of these any combination of storage media. The storage unit 44 stores table information 440, for example. The table information 440 will be explained later in detail.

With such a configuration, in the distance image capturing device 1, the light receiving unit 3 receives the reflected light RL that is reflected by the object OB from the light pulse PO in the near-infrared wavelength band that the light source unit 2 irradiates the object OB. A distance image processing unit 4 outputs distance information obtained by measuring the distance to the object OB.

Although FIG. 1 shows the distance image pickup device 1 having a configuration in which the distance image processing unit 4 is provided inside, the distance image processing unit 4 is a component provided outside the distance image pickup device 1. may

Next, the configuration of the distance image sensor 32 used as an imaging device in the distance image pickup device 1 will be described. FIG. 2 is a block diagram showing a schematic configuration of an imaging device (distance image sensor 32) used in the distance image pickup device 1 of the embodiment.

As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a sorting operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325 .

The light-receiving region 320 is a region in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which the pixels are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixel 321 accumulates electric charge corresponding to the amount of light received. A control circuit 322 controls the distance image sensor 32 in an integrated manner. The control circuit 322 controls the operation of the components of the range image sensor 32 according to instructions from the timing control section 41 of the range image processing section 4, for example. It should be noted that the components provided in the distance image sensor 32 may be controlled directly by the timing control section 41, in which case the control circuit 322 may be omitted.

The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving region 320 for each row according to control from the control circuit 322 . The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge accumulated in each charge accumulation portion CS of the pixel 321 . In this case, the vertical scanning circuit 323 distributes the charges converted by the photoelectric conversion elements to the respective charge accumulating portions of the pixels 321 . That is, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing) on the voltage signals output to the corresponding vertical signal lines from the pixels 321 in each column under the control of the control circuit 322 . , A/D conversion processing, etc.).

The horizontal scanning circuit 324 is a circuit that sequentially outputs signals output from the pixel signal processing circuit 325 to horizontal signal lines under the control of the control circuit 322 . As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to the distance image processing section 4 via the horizontal signal line.

In the following description, it is assumed that the pixel signal processing circuit 325 performs A/D conversion processing and the pixel signal is a digital signal.

Here, the configuration of the pixels 321 arranged in the light receiving area 320 provided in the distance image sensor 32 will be described. FIG. 3 is a circuit diagram showing an example of the configuration of the pixels 321 arranged within the light receiving area 320 of the distance image sensor 32 of the embodiment. FIG. 3 shows an example of the configuration of one pixel 321 among the plurality of pixels 321 arranged in the light receiving region 320. As shown in FIG. The pixel 321 is an example of a configuration including three pixel signal readout units.

As shown in FIG. 3, the pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU for outputting voltage signals from corresponding output terminals O. As shown in FIG. Each pixel signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, the floating diffusion FD and the charge storage capacitor C constitute a charge storage unit CS.

In FIG. 3, three pixel signal readout units RU are distinguished by adding numerals "1", "2" or "3" after the symbol "RU" of the three pixel signal readout units RU. do. Similarly, each component provided in the three pixel signal readout units RU is also indicated by a numeral representing each pixel signal readout unit RU after the symbol, so that each component corresponds to the pixel signal readout unit. RUs are distinguished.

In the pixel 321 shown in FIG. 3, the pixel signal readout unit RU1 that outputs a voltage signal from the output terminal O1 includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitor C1, a reset gate transistor RT1, and a source follower. It includes a gate transistor SF1 and a selection gate transistor SL1. In the pixel signal readout unit RU1, the charge storage unit CS1 is composed of the floating diffusion FD1 and the charge storage capacitor C1. The pixel signal readout unit RU2 and the pixel signal readout unit RU3 also have the same configuration.

The photoelectric conversion element PD is an embedded photodiode that photoelectrically converts incident light to generate charges and accumulate the generated charges. The structure of the photoelectric conversion element PD may be arbitrary. The photoelectric conversion element PD may be, for example, a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. It may be a PIN photodiode. Further, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

In the pixel 321, the charge generated by photoelectrically converting the light incident on the photoelectric conversion element PD is distributed to each of the three charge storage units CS, and each voltage signal corresponding to the charge amount of the distributed charge is output to the pixel. Output to the signal processing circuit 325 .

The configuration of the pixels arranged in the distance image sensor 32 is not limited to the configuration provided with three pixel signal readout units RU as shown in FIG. Any pixel of the configuration may be used. That is, the number of pixel signal readout units RU (charge storage units CS) provided in the pixels arranged in the distance image sensor 32 may be two, or may be four or more.

Also, in the pixel 321 having the configuration shown in FIG. 3, an example in which the charge storage portion CS is configured by the floating diffusion FD and the charge storage capacitor C is shown. However, the charge storage section CS may be composed of at least the floating diffusion FD, and the pixel 321 may be configured without the charge storage capacitor C. FIG.

Further, although the pixel 321 having the configuration shown in FIG. 3 has an example of the configuration including the drain gate transistor GD, the configuration is not limited to this. For example, when it is not necessary to discard the charge remaining in the photoelectric conversion element PD without being accumulated in the charge accumulation section CS, the configuration without the drain gate transistor GD may be employed.

Next, driving timing of the pixel 321 will be described with reference to FIG. FIG. 4 is a timing chart showing timings for driving the pixels 321 of the embodiment.

In FIG. 4, the timing of emitting the light pulse PO is "L", the timing of receiving the reflected light is "R", the timing of the driving signal TX1 is "G1", the timing of the driving signal TX2 is "G2", and the driving signal The timing of TX3 and the timing of the drive signal RSTD are indicated by item names of "G3" and "GD," respectively. The drive signal TX1 is a signal for driving the readout gate transistor G1. The same applies to the drive signals TX2 and TX3.

As shown in FIG. 4, it is assumed that the light pulse PO is emitted for the irradiation time To, and the reflected light RL is received by the distance image sensor 32 after the delay time Td. The vertical scanning circuit 323 accumulates charges in the order of the charge accumulation units CS1, CS2, and CS3 in synchronization with the irradiation of the light pulse PO. In FIG. 4, in one sorting process, the time from the irradiation of the light pulse PO until the charges are sequentially accumulated in the charge accumulation units CS is expressed as a "unit accumulation period". After repeating the allocation process performed in the "unit accumulation period" for the number of times of accumulation corresponding to one frame, the process of reading out the amount of charge accumulated during that period is performed. The time during which the process of reading out the accumulated charge amount is expressed as a “readout period”.

First, the case of receiving reflected light RL from an object at a short distance will be described with reference to FIG. The vertical scanning circuit 323 turns off the drain gate transistor GD and turns on the readout gate transistor G1 in synchronization with the timing of irradiating the light pulse PO. The vertical scanning circuit 323 turns off the readout gate transistor G1 after the accumulation time Ta has elapsed since turning on the readout gate transistor G1. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the readout gate transistor G1 is controlled to be on is accumulated in the charge storage unit CS1 via the readout gate transistor G1.

Next, the vertical scanning circuit 323 turns on the readout gate transistor G2 for the accumulation time Ta at the timing when the readout gate transistor G1 is turned off. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the readout gate transistor G2 is controlled to be on is accumulated in the charge storage unit CS2 via the readout gate transistor G2.

Next, the vertical scanning circuit 323 turns on the readout gate transistor G3 at the timing when the charge accumulation in the charge accumulation unit CS2 is finished, and turns off the readout gate transistor G3 after the accumulation time Ta has elapsed. do. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the readout gate transistor G3 is controlled to be on is accumulated in the charge storage unit CS3 via the readout gate transistor G3.

Next, the vertical scanning circuit 323 turns on the drain gate transistor GD at the timing when the charge accumulation in the charge accumulation section CS3 is completed to discharge the charge. As a result, charges photoelectrically converted by the photoelectric conversion element PD are discarded via the drain gate transistor GD.

The vertical scanning circuit 323 repeats the above-described driving a predetermined number of times over one frame. After that, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge distributed to each charge storage section CS. Specifically, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in the charge storage unit CS1 via the pixel signal readout unit RU1 by turning on the selection gate transistor SL1 for a predetermined time. Output from O1. Similarly, the vertical scanning circuit 323 sequentially turns on the selection gate transistors SL2 and SL3 to output voltage signals corresponding to the amounts of charge accumulated in the charge accumulation units CS2 and CS3 from the output terminals O2 and O3. Let Then, an electric signal corresponding to the amount of charge for one frame accumulated in each of the charge accumulation units CS is output to the distance calculation unit 42 via the pixel signal processing circuit 325 and the horizontal scanning circuit 324 .

In the above description, the case where the light source unit 2 irradiates the light pulse PO at the timing when the readout gate transistor G1 is turned on has been described as an example. However, it is not limited to this. The light source unit 2 may irradiate at a timing such that at least the reflected light RL from an object at a short distance is received across the charge storage units CS1 and CS2. For example, the light source unit 2 may emit light at a timing before the readout gate transistor G1 is turned on. Further, in the above description, the case where the irradiation time To for irradiating the light pulse PO is the same length as the accumulation time Ta has been described as an example. However, it is not limited to this. The irradiation time To and the accumulation time Ta may be different time intervals.

In the short-distance light-receiving pixel as shown in FIG. 4, the relationship between the timing of irradiating the light pulse PO and the timing of accumulating the charge in each of the charge accumulating portions CS causes the reflected light RL and the amount of charge corresponding to the external light component is distributed and held. Also, the charge storage section CS3 holds a charge amount corresponding to an external light component such as background light. In this case, the charge storage units CS1 and CS2 are an example of the "distance calculation charge storage unit".

The distribution (distribution ratio) of the amount of charge distributed to the charge accumulating units CS1 and CS2 is a ratio corresponding to the delay time Td until the light pulse PO is reflected by the object OB and is incident on the range image pickup device 1 .

Using this principle, the distance calculation unit 42 calculates the delay time Td in the conventional short-distance light-receiving pixel using the following equation (1).

Td=To×(Q2-Q3)/(Q1+Q2-2×Q3) (1)

Here, To is the period during which the light pulse PO is irradiated, Q1 is the amount of charge accumulated in the charge storage section CS1, Q2 is the amount of charge accumulated in the charge storage section CS2, and Q3 is the amount of charge accumulated in the charge storage section CS3. shows the amount of charge. Note that the formula (1) assumes that the charge amount corresponding to the external light component among the charge amounts accumulated in the charge storage units CS1 and CS2 is the same as the charge amount accumulated in the charge storage unit CS3. and

The distance calculation unit 42 multiplies the delay time Td obtained by the formula (1) by the speed of light (velocity) in the short-distance light-receiving pixel, thereby calculating the round-trip distance to the object OB. Then, the distance calculation unit 42 obtains the distance to the object OB by halving the round trip distance calculated above.

Here, in the conventional distance image capturing device 1, the factors that cause an error in the distance (measured distance) calculated from the amount of accumulated charge will be described.

One possible cause of the error is that the light incident on the distance image pickup device 1 is diffusely reflected by the optical system such as the lens 31 . In other words, such diffused reflection affects the amount of light received by the pixels, causing an error in the distance to be measured. As a countermeasure, for example, a method of correcting the error using a correspondence table (an example of "table information") in which the charge ratio R is associated with the distance D is adopted. The distance here is the distance to the subject OB. Also, the charge ratio R is an example of a "variable". The distance D is an example of the "correspondence distance."

The charge ratio R here is the ratio of the amount of charge accumulated in each of the two charge accumulation units CS (charge accumulation units CS1 and CS2 in FIG. 4) in which the reflected light RL is divided and accumulated. In this case, the charge ratio R is represented by, for example, the ratio H1 in the following equation (2) or the ratio H2 in the following equation (3). The ratio H1 is an example of a "charge ratio." The ratio H2 is an example of a "charge ratio." The ratios H1 and H2 are simply referred to as the charge ratio R hereinafter.

H1=Q1#/(Q1#+Q2#) (2)
H2=Q2#/(Q1#+Q2#) (3)
However, Q1#=Q1-Qb
Q2#=Q2-Qb
Q1 is the amount of charge accumulated in the charge accumulation section CS1
Q2 is the amount of charge accumulated in the charge accumulation section CS2
Qb is the charge amount corresponding to the external light component accumulated in the charge accumulation section CS.

By using the correspondence table in which the distance D is associated with the charge ratio R, even if an error occurs in the distance calculated by the formula (1), it is possible to correct the distance so as to reduce the error. Become. A method for calculating the charge amount Qb corresponding to the external light component will be described later in detail.

In addition, as a case in which an error is likely to occur, the ratio of the amount of charge accumulated in each of the two charge accumulation units CS (charge accumulation units CS1 and CS2 in FIG. 4) in which the reflected light RL is divided and accumulated is biased. It is conceivable that there may be Here, the unevenness means that one of the two charge storage portions CS stores substantially the entire amount of charge due to the reflected light RL, and the remaining charge storage portion CS has a very small amount of charge due to the reflected light RL. It is that little is accumulated. That is, the charge ratio R becomes a value close to 0 or a value close to 1.

For example, a charge amount corresponding to about 90% of the light amount of the reflected light RL is accumulated in the charge storage section CS1, and a charge amount corresponding to the remaining light amount of about 10% is accumulated in the charge storage section CS2. . In this case, the charge ratio R based on the formula (2) is about R=0.9, and the charge ratio R based on the formula (3) is about R=0.1. Alternatively, the charge amount corresponding to about 10% of the light amount of the reflected light RL is accumulated in the charge storage section CS1, and the charge amount corresponding to the remaining light amount of about 90% is accumulated in the charge storage section CS2. . In this case, the charge ratio R based on the formula (2) is about R=0.1, and the charge ratio R based on the formula (3) is about R=0.9.

Here, charges that become noise components are also accumulated in the charge accumulation section CS. Here, the noise component is light different from the reflected light RL, for example, the amount of charge caused by the inside of the range image pickup device 1, for example, dullness of light pulses and gate pulses, delay in charge transfer, and the like. A constant amount of such noise components is also accumulated in the two charge accumulation units CS in which the reflected light RL is distributed and accumulated, regardless of the magnitude of the amount of charge caused by the reflected light RL.

When the reflected light RL is unevenly accumulated, both the charge accumulation portion CS in which substantially the entire amount of charge due to the reflected light RL is accumulated and the charge accumulation portion CS in which only a small amount of charge due to the reflected light RL are accumulated are A certain amount of noise component accumulates. Therefore, the SN ratio of the charge storage section CS in which only a small amount of charge is accumulated due to the reflected light RL is lowered. Here, S is the target signal component, which is the amount of charge caused by the reflected light RL used for distance calculation. Also, N here is the amount of charge caused by the noise component. If the distance is calculated using a charge amount with a low SN ratio, an error is likely to occur in the calculation result. When the SN ratio is low, that is, when the charge ratio R exceeds a predetermined threshold value (for example, 0.9), it deviates from an ideal calculation formula that does not consider noise, resulting in a large error in the measured distance. Become. As a countermeasure, it is conceivable to make the table intervals uniformly finer (smaller). However, when the SN ratio is high and there is no "maldistribution", the table interval becomes unnecessarily fine, increasing the memory capacity of the table.

As a countermeasure against this, in the present embodiment, the table information 440 is created so that the table interval differs depending on whether the reflected light RL is unevenly accumulated or not. Here, the table interval is an interval corresponding to the difference between adjacent variables when the variables (charge ratio R or distance D) included in the table information 440 are arranged in ascending order or descending order.

In the following, a case where the variable in the table information 440 is the charge ratio R will be described as an example. However, it is not limited to this. Even when the variable in the table information 440 is the distance D, it is possible to create the table information 440 so that the table interval differs depending on whether the reflected light RL is accumulated unevenly or not. be.

The case where the reflected light RL is unevenly accumulated is, for example, the amount of charge accumulated in each of the two charge accumulation units CS (the charge accumulation units CS1 and CS2 in FIG. 4) in which the reflected light RL is distributed and accumulated. is equal to or greater than a predetermined value. The predetermined value here may be arbitrarily set according to the target accuracy of the measured distance. Such a state in which the reflected light RL is unevenly accumulated is an example of an “unevenly distributed state”.

For example, among the charge ratios R included in the table information 440, the charge ratios R within a certain range (the range in which the charge ratio can be taken when the reflected light RL is biased and accumulated) are considered as fine table intervals with small intervals. The table information 440 is created so that On the other hand, for the charge ratio R in another range (the range in which the charge ratio can be taken when the reflected light RL is not accumulated unevenly), the table information 440 is created so that the table interval is large and coarse ( 6, 15 and 17). As a result, when attempting to determine the distance D from the charge ratio R within a range where errors are likely to occur, an appropriate charge ratio can be selected from a charge ratio group created with fine table intervals. Therefore, it is possible to improve the accuracy of the determined distance.

Here, the table information 440 will be explained using FIGS. 5 and 6. FIG. FIG. 5 is a diagram showing an example of the configuration of table information 440 according to the embodiment. As shown in FIG. 5, the table information 440 includes, for example, charge ratio threshold, table interval, charge ratio, and distance items. The charge ratio threshold is information indicating a threshold for changing the table interval. The table interval is information indicating the interval between charge ratios in the table information 440 . The charge ratio is a ratio represented by formula (2) or formula (3). The distance is the distance associated with the charge ratio. In the example of this figure, the table interval of the charge ratio group (charge ratios R1 to R4) less than the charge ratio threshold value R5 is set to interval TK2, and the table interval of the charge ratio group (charge ratios R5 to R9) of the charge ratio threshold value R5 or more is set to TK2. An example with an interval TK3 is shown.

FIG. 6 is a diagram illustrating the table information 440 of the embodiment. The horizontal axis of FIG. 6 indicates the charge ratio R, and the vertical axis indicates the distance D. In FIG. In FIG. 6, two ranges E2 and E3 are set according to the magnitude of the charge ratio R on the horizontal axis. The range E2 is a range in which the charge ratio is less than the charge ratio R5. The range E3 is a range in which the charge ratio is equal to or greater than the charge ratio R5. Here, the range E2 is an example of the "second range". The range E3 is an example of the "third range".

FIG. 6 shows the relationship between the charge ratio R and the distance D based on the table information 440 of FIG. A charge ratio R5 is set as a charge ratio threshold in the table information 440 of FIG. In this case, the table interval of the charge ratios R1 to R4 included in the range E2 is the interval TK2. Also, the table interval of the charge ratios R5 to R9 included in the range E3 is the interval TK3. Interval TK3 is smaller than interval TK2.

Note that FIG. 6 shows an example in which the charge ratio R is represented by the formula (3). That is, FIG. 6 shows a characteristic that the distance D increases as the charge ratio R increases. When the charge ratio R is represented by the formula (2), the characteristic is a downward slope in which the distance D decreases as the charge ratio R increases. In this case, the charge ratio threshold is set to (1.0-charge ratio R5), the table interval for charge ratios below the charge ratio threshold is fine, and the table interval for charge ratios above the charge ratio threshold is coarsely set.

7A and 7B are diagrams for explaining the process of performing linear interpolation using the table information 440 by the distance image processing unit 4 of the embodiment. FIG. 7 shows part of the characteristics of FIG. 6 (range of charge ratios R1 to R2). In this example, the charge ratio calculated based on the amount of charge accumulated in the photoelectric conversion element PD is a charge ratio R corresponding to an intermediate value between the charge ratios R1 and R2. In this case, the distance calculator 42 may calculate the distance D corresponding to the charge ratio R by linearly interpolating the distance D11 corresponding to the charge ratio R1 and the distance D12 corresponding to the charge ratio R2. . By linear interpolation, it is possible to calculate the distance with higher accuracy.

Further, the interval of the charge ratio in the table information 440 may be set so that the accuracy of the interpolated distance is within a predetermined range when linear interpolation is performed. In this case, the charge ratio interval in the table information 440 is set within a linear range, for example.

As described above, when the reflected light RL is accumulated unevenly, the SN ratio in the charge ratio R decreases due to the dullness of the light pulse and the gate pulse, the delay in charge transfer, and the like, and the relationship between the charge ratio R and the distance D becomes nonlinear. tends to be The table interval is reduced within such a non-linear range. This makes it possible to improve the accuracy of the distance when linear interpolation is used to determine the distance.

Here, a method for calculating the charge amount Qb corresponding to the external light component will be described with reference to FIGS. 8 and 9. FIG. 8 and 9 are diagrams for explaining the process of determining the external light component by the distance image processing unit 4 of the embodiment.

(Method 1 for calculating the charge amount Qb corresponding to the external light component)
FIG. 8 shows a timing chart when receiving reflected light RL from an object at a longer distance than in FIG. 8, the timing at which the vertical scanning circuit 323 irradiates the light pulse PO, the timing at which the readout gate transistors G1 to G3 and the drain gate transistor GD are turned on, etc. are the same as in FIG. omitted.

As shown in FIG. 8, when the delay time Td is longer than in the timing chart of FIG. A charge amount Qb corresponding to the component is accumulated, and the charge amount Qb corresponding to the reflected light RL and the external light component are distributed and accumulated in the charge accumulation units CS2 and CS3. In this case, the charge storage units CS2 and CS3 are an example of the "distance calculation charge storage unit".

That is, when the delay time Td is not large (in the case of FIG. 4), the charge amount Qb corresponding to the external light component is accumulated in the charge storage section CS3, and when the delay time Td is large (in the case of FIG. 8), A charge amount Qb corresponding to the external light component is accumulated in the charge accumulation portion CS1. In either case of FIGS. 4 and 8, the charge amount Qb corresponding to the same amount of external light component is accumulated in each of the charge accumulation units CS1 to CS3. Therefore, the charge storage section CS in which the reflected light RL is distributed and accumulated accumulates a larger amount of charge than the charge storage section CS in which only other external light components are accumulated.

Using this property, the distance calculation unit 42 determines the smallest charge amount among the charge amounts accumulated in the charge accumulation units CS1 to CS3 as the charge amount Qb corresponding to the external light component.

(Method 2 for calculating the charge amount Qb corresponding to the external light component)
The distance image pickup device 1 may control the timing so that only the charge amount Qb corresponding to the external light component is accumulated in the predetermined specific charge accumulation section CS. In this case, the distance calculation unit 42 can determine the charge amount accumulated in the specific charge accumulation unit CS as the charge amount Qb corresponding to the external light component regardless of the magnitude of the delay time Td.

FIG. 9 shows a timing chart when control is performed so that only the charge amount Qb corresponding to the external light component is accumulated in the charge accumulation section CS1. 9, the timing at which the vertical scanning circuit 323 irradiates the light pulse PO, the timing at which the readout gate transistors G1 to G3 and the drain gate transistor GD are turned on, etc. are the same as in FIG. omitted.

As shown in the example of FIG. 9, by accumulating charges in the charge accumulating section CS1 before irradiating the light pulse PO, only the charge amount Qb corresponding to the external light component is accumulated in the charge accumulating section CS1. can be In this case, the distance calculation unit 42 determines the charge amount accumulated in the charge accumulation unit CS1 as the charge amount Qb corresponding to the external light component. In this case, the charge storage section CS1 is an example of a "predetermined external light storage charge storage section".

FIG. 10 is a flow chart showing the flow of processing performed by the distance image processing unit 4 of the embodiment. The distance calculation unit 42 acquires the charge amounts Q1 to Q3 accumulated in the charge accumulation units CS1 to CS3, respectively (step S10). The distance calculator 42 uses the acquired charge amounts Q1 to Q3 to calculate the charge amount Qb corresponding to the external light component (step S11). The distance calculation unit 42 may set the smallest charge amount among the charge amounts Q1 to Q3 as the charge amount Qb, or may be accumulated in a predetermined specific charge storage unit CS (for example, the charge storage unit CS1 in FIG. 9). The charge amount obtained may be used as the charge amount Qb.

The distance calculator 42 calculates the charge ratio R using the charge amounts Q1 to Q3 and the charge amount Qb (step S12). The distance calculator 42 calculates the charge ratio R by, for example, substituting the charge amounts Q1 to Q3 and the charge amount Qb into the equation (2) or (3). In this case, the distance calculation unit 42 selects either formula (2) or formula (3) so as to match the configuration of the charge ratio shown in the table information 440, and using the selected formula, A charge ratio R is calculated.

The distance calculator 42 uses the table information 440 to select the distance corresponding to the charge ratio R calculated in step S12 (step S13). In this case, the distance calculation unit 42 may select two distances associated with the two charge ratios for linear interpolation, or may select the two distances that are most closely related to the charge ratio R calculated in step S12. One distance may be selected that corresponds to a close charge ratio.

When two distances are selected (step S14, YES), the distance calculator 42 determines the distance (measured distance) by linear interpolation (step S15). On the other hand, when one distance is selected (step S14, NO), the distance calculator 42 determines the selected distance as the corrected distance (measured distance) (step S16).

In the above description, the case where the pixel 321 of the distance image pickup device 1 includes the three charge storage units CS1 to CS3 has been described as an example. However, it is not limited to this. The pixel 321 of the distance image pickup device 1 may be configured to include four or more (for example, N, N≧4) charge accumulation units CS.

When the pixel 321 of the distance image pickup device 1 includes N (N≧4) charge storage units CS, in step S10, the distance calculation unit 42 calculates the charge amount Q1 accumulated in each of the charge storage units CS1 to CSN. ~ Get QN. In step S11, the distance calculator 42 uses the acquired charge amounts Q1 to Q3 to calculate the charge amount Qb corresponding to the external light component. The method by which the distance calculation unit 42 calculates the amount of charge Qb is the same as in the case where the pixel 321 of the distance image pickup device 1 has three charge storage units CS1 to CS3.

In step S12, the distance calculation unit 42 selects two charge storage units CS (distance-measuring charge storage units) in which charges corresponding to the reflected light RL are distributed and stored from the charge storage units CS1 to CSN. The method by which the distance calculation unit 42 selects two charge storage units CS is, for example, based on the amount of charge stored in each charge storage unit CS out of a combination of two charge storage units CS in which charges are continuously stored. are the two charge accumulators CS (distance-measuring charge accumulators) in which the charges corresponding to the reflected light RL are distributed and accumulated. The distance calculation unit 42 uses the amount of charge accumulated in each of the two charge accumulation units CS in which the charges corresponding to the reflected light RL are distributed and accumulated, and the amount of charge Qb corresponding to the external light component, to calculate the charge Calculate the ratio R. The processing of steps S13 to S16 is the same as in the case where the pixel 321 of the distance image pickup device 1 has three charge storage units CS1 to CS3.

Alternatively, the pixel 321 of the distance image pickup device 1 may be configured to include two charge storage units CS. In this case, the distance image pickup device 1 performs a process of accumulating only the charge corresponding to the external light component (referred to as the first process) and a process of accumulating the charge including the reflected light RL (second process) for each measurement. ) and two processes related to charge accumulation. For example, the range image capturing apparatus 1 performs the first process on the first frame and the second process on the next frame. When performing the first process, the distance image pickup device 1 accumulates charges in each of the charge accumulation units CS1 and CS2 without irradiating the light pulse PO. When performing the second process, the distance imaging device 1 irradiates a light pulse PO to accumulate electric charge in each of the electric charge accumulation units CS1 and CS2.

In this case, in step S10, the distance calculation unit 42 acquires the charge amounts Q1f and Q2f accumulated in the charge accumulation units CS1 and CS2 in the first process, respectively. Also, the distance calculation unit 42 acquires the charge amounts Q1s and Q2s accumulated in the charge accumulation units CS1 and CS2 in the second process. In step S11, the distance calculation unit 42 sets either or both of the acquired charge amount Q1f and Q2f as the charge amount Qb corresponding to the external light component. In step S12, the distance calculator 42 calculates the charge ratio R using the acquired charge amounts Q1s, Q2s, and charge amount Qb. The method by which the distance calculation unit 42 calculates the charge ratio R is the same as in the case where the pixel 321 of the distance image pickup device 1 has three charge storage units CS1 to CS3.

As described above, the distance image capturing device 1 of the embodiment includes the light source unit 2, the light receiving unit 3, the storage unit 44, and the distance image processing unit 4. The light source unit 2 irradiates a light pulse PO into the measurement space in which the subject OB exists. The light receiving unit 3 includes pixels 321 and a vertical scanning circuit 323 (an example of a driving circuit). The pixel 321 includes a photoelectric conversion element PD and three or more charge storage units CS. The vertical scanning circuit 323 distributes and accumulates electric charges in each of the charge accumulation units CS in the pixels 321 at a predetermined timing synchronized with the irradiation of the light pulse PO. The storage unit 44 stores table information 440 . The table information 440 is information in which the charge ratio R is associated with the distance to the object OB (corresponding distance). The distance image processing unit 4 determines the measured distance to the subject OB using the amount of charge accumulated in each charge accumulation unit CS and the table information 440 . The charge ratio R is the amount of charge accumulated in each of the two or more charge accumulation units CS in which the charges corresponding to the reflected light RL are distributed and accumulated among the three or more charge accumulation units CS to the external light component. It is a ratio indicated by using the charge amount (distance calculation charge amount) obtained by subtracting the corresponding charge amount. Specifically, the charge ratio R is a ratio indicated using the charge amounts Q1# and Q2#, for example, the ratio of the charge amount Q2# to the sum of the charge amounts Q1# and Q2#. The charge amount Q1# is the charge amount (first distance calculation charge amount) obtained by subtracting the charge amount Qb corresponding to the external light component from the charge amount Q1 (first charge amount) accumulated in the charge storage section CS1 in FIG. is. Among the plurality of charge storage units CS, the charge storage unit CS1 is the charge storage unit CS to which the amount of charge corresponding to the reflected light RL reflected by the light pulse PO from the object OB is applied first. is an example of The charge amount Q2# is the charge amount (second distance calculation charge amount) obtained by subtracting the charge amount Qb corresponding to the external light component from the charge amount Q2 (second charge amount) accumulated in the charge storage section CS2 in FIG. is. Among the plurality of charge storage units CS, the charge storage unit CS2 is the charge storage unit CS in which the amount of charge corresponding to the reflected light RL reflected by the light pulse PO reflected by the object OB is stored next to the charge storage unit CS1. It is an example of a "second charge storage section". In the table information 440, the table interval of variables in the unevenly distributed state is smaller than the table interval of variables that are not in the unevenly distributed state. Being unevenly distributed means that the amount of charge Q1# and the amount of charge Q2# are extremely uneven, and the ratio of the amount of charge Q1# to the amount of charge Q2# (charge ratio R) is larger than a predetermined threshold. state. The uneven distribution state is, for example, a state in which the charge ratio R exceeds a predetermined upper limit or falls below a predetermined lower limit. The variable is either the charge ratio R or the distance D in table information 440 . A table interval is a difference between adjacent variables when the variables in the table information 440 are arranged in ascending order. The distance image processing unit 4 calculates the charge ratio R based on the charge amount accumulated in each of the plurality of charge accumulation units CS, acquires the corresponding distance corresponding to the calculated charge ratio R from the table information 440, and acquires The corresponding distances obtained are used to determine the measured distance.

Thereby, the distance image capturing device 1 of the embodiment can obtain the distance D corresponding to the charge ratio R using the table information 440 . Therefore, even if there is an error in the distance calculated based on the charge amounts Q1 to Q3 accumulated in the charge accumulation units CS1 to CS3, the distance can be corrected so as to approach the actual distance. Moreover, in the present embodiment, the table information 440 is created such that when the charge amounts Q1# and Q2# are accumulated in an unbalanced state, the table interval is smaller than when they are not accumulated. Therefore, even if the charges Q1# and Q2# are accumulated in a biased state and errors are likely to occur, the distance can be accurately corrected. Moreover, memory increase can be suppressed compared to the case where the table interval in the table information 440 is uniformly reduced. Therefore, by making the table interval non-linear, it is possible to accurately calculate the distance deviation region while suppressing an increase in memory. The distance deviation region is a region in which the SN ratio is low and the charge ratio R exceeds a predetermined threshold value and is in an “unevenly distributed state”.

Further, in the distance image pickup device 1 of the embodiment, each of the charge ratios R (variables) included in the table information 440 is a value included in either range E2 or E3 in FIG. The range E2 is a range in which the charge ratio R is less than the charge ratio threshold value R5 (second threshold value), and is an example of the "second range." The range E3 is a range in which the charge ratio R is equal to or greater than the charge ratio threshold value R5, and is an example of the "third range". The table spacing in range E3 is smaller than the table spacing in range E2. As a result, the distance can be accurately corrected even if the charge amount Q2# is accumulated in a biased state in which a large amount is accumulated compared to the charge amount Q1#, and errors are likely to occur.

Further, in the distance image capturing device 1 of the embodiment, the distance image processing unit 4 determines the distance after correction by linear interpolation. Using the table information 440, the distance image processing unit 4 uses the calculated first distance (for example, the distance D11) associated with the charge ratio (for example, the charge ratio R1) smaller than the calculated charge ratio R, and the distance from the charge ratio R A second distance (for example, distance D12) associated with a large charge ratio (for example, charge ratio R2) is extracted. The distance image processing unit 4 determines the distance obtained by linearly interpolating the extracted first distance (for example, distance D11) and second distance (for example, distance D12) as the corrected distance (measured distance). do. In the present embodiment, the table information 440 is created such that when the charge amounts Q1# and Q2# are accumulated in a biased state, the table interval is smaller than when the charge amounts are not biased. When interpolating, the distance can be corrected with higher accuracy.

Further, the distance image processing unit 4 of the embodiment sets the smallest charge amount among the charge amounts accumulated in each of the charge accumulation units CS1 to CS3 as the charge amount Qb corresponding to the external light component. As a result, in the distance image pickup device 1 of the embodiment, the charge amount Qb corresponding to the external light component can be calculated by a simple process of comparing the charge amount accumulated in each of the charge accumulation units CS1 to CS3. can.

Further, the distance image processing section 4 of the embodiment controls the accumulation timing so that only the amount of light corresponding to the external light component is accumulated in a specific charge accumulation section CS among the charge accumulation sections CS1 to CS3. The distance calculation unit 42 sets the charge amount accumulated in the specific charge accumulation unit CS as the charge amount Qb corresponding to the external light component. As a result, in the distance image pickup device 1 of the embodiment, the charge amount accumulated in the specific charge accumulation unit CS can be calculated as the charge amount Qb corresponding to the external light component, and the external light component can be easily handled. It is possible to determine the amount of charge Qb to be applied.

(Effect of Embodiment)
Here, the effect of the distance image capturing device 1 of the embodiment will be described with reference to FIG. 11 . FIG. 11 is a diagram explaining the effect of the embodiment. FIG. 11 shows the relationship between the actual distance (actual distance) and the measured distance. The horizontal axis of FIG. 11 indicates the actual distance, and the vertical axis indicates the measured distance. The distance here is the distance to the subject OB. In FIG. 11, the measured distance indicated by black circles when the table information 440 is not used (table information not used) is, for example, the distance calculated by substituting the charge amounts Q1 to Q3 into the equation (1). The measured distance when using the table information 440 indicated by a black triangle (using table information) is the distance calculated using the table information 440 . As shown in this figure, the measured distance when using table information matches the actual distance. On the other hand, when the table information is not used, the measured distance does not match the actual distance and is a value including an error. That is, in the distance image pickup device 1 of the present embodiment, by determining the measurement distance using the table information 440, it is possible to calculate a value closer to the actual distance.

(Modification 1 of Embodiment)
Modification 1 of the embodiment will now be described. This modification differs from the above-described embodiment in that the storage unit 44 stores table information 440 for each of a plurality of time windows. The time window here corresponds to a combination of charge storage units for distance measurement. The distance-measuring charge accumulators are two charge accumulators CS in which charges corresponding to the reflected light RL are distributed and accumulated.

For example, in FIG. 4, the combination of the charge storage units for distance measurement is the charge storage units CS1 and CS2, and this combination corresponds to the first time window. The distance D is determined according to the charge ratio R of the charges distributed and accumulated in the first time window.

Also, in FIG. 8, the combination of the charge storage units for distance measurement is the charge storage units CS2 and CS3. This combination corresponds to the second time window. The distance D is determined according to the charge ratio R of the charges distributed and accumulated in the second time window.

FIG. 12 is a diagram illustrating characteristics of two time windows in the modified example of the embodiment. The characteristic here is a characteristic indicating the correspondence relationship between the actual distance and the measured distance. The horizontal axis of FIG. 12 indicates the actual distance, and the vertical axis indicates the measured distance. A characteristic L0 indicates an ideal relationship between the actual distance and the measured distance. A characteristic L1 indicates the relationship between the actual distance and the measured distance in the first time window. A characteristic L2 indicates the relationship between the actual distance and the measured distance in the second time window. As shown in the example of this figure, the time window and another time window often exhibit different characteristics in the correspondence relationship between the actual distance and the measured distance. Therefore, when correcting the distance in another time window using the table information 440 that allows accurate correction in a certain time window, accurate correction is not always possible. Further, as described above, errors are likely to occur when the charges corresponding to the reflected light RL are accumulated in a biased state. For this reason, the distance error increases at the connecting portion TS of the time windows.

Here, in the first time window, the reflected light RL from the subject OB, which is closer than the second time window, is received. In this case, the table information 440 corresponding to the first time window is an example of "short distance information". On the other hand, in the second time window, the reflected light RL from the object OB, which is farther than the first time window, is received. In this case, the table information 440 corresponding to the second time window is an example of "long distance information".

As a countermeasure against this, in this modified example, table information 440 for each time window is created in advance in the storage unit 44 and stored in the storage unit 44 .

13 and 14 are diagrams showing examples of the configuration of table information in modifications of the embodiment. FIG. 13 shows table information 440A used for a combination of charge storage units for distance measurement (charge storage units CS1 and CS2 in this example) corresponding to a certain time window. FIG. 14 shows table information 440B used for a combination of charge storage units for distance measurement (for example, charge storage units CS2 and CS3) corresponding to another time window. Thus, by creating the table information 440A (400B) for each time window, it becomes possible to perform correction according to each time window.

As shown in FIG. 13, the table information 440A includes items such as time window, charge ratio threshold, table interval, charge ratio, and distance. The time window is information indicating a combination of charge storage units for distance measurement. In the example of this figure, (Q1Q2) is shown in the time window, indicating that the combination of the charge storage units for distance measurement is the charge storage units CS1 and CS2. Since the charge ratio threshold value, table interval, charge ratio, and distance are the same as in FIG. 5, description thereof will be omitted.

As shown in FIG. 14, the table information 440B includes items such as time window, charge ratio threshold, table interval, charge ratio, and distance. The time window is the same as in FIG. 13, and the charge ratio threshold value, table interval, charge ratio, and distance are the same as in FIG. 5, so description thereof will be omitted. In the example of this figure, (Q2Q3) is shown in the time window, indicating that the combination of the charge storage units for distance measurement is the charge storage units CS2 and CS3. Also, the table interval of the charge ratio group (charge ratios R11 to R14) less than the charge ratio threshold R15 is defined as interval TK4, and the table interval of the charge ratio group (charge ratios R15 to R19) equal to or higher than the charge ratio threshold R15 is defined as interval TK5. An example is given.

FIG. 15 is a diagram illustrating table information 440B in a modification of the embodiment. 15, the horizontal axis indicates the charge ratio R, and the vertical axis indicates the distance D. In FIG. In FIG. 15, two ranges E11 and E12 are set according to the magnitude of the charge ratio R on the horizontal axis. The range E11 is a range in which the charge ratio is less than the charge ratio R15. The range E12 is a range in which the charge ratio is equal to or greater than the charge ratio R15. Here, the range E11 is an example of the "first range". The range E12 is an example of the "second range". Also, the range E11 is an example of the "fourth range". The range E12 is an example of the "fifth range".

FIG. 15 shows the relationship between the charge ratio R and the distance D based on the table information 440B of FIG. A charge ratio R15 is set as a charge ratio threshold in the table information 440B. In this case, the table interval of the charge ratios R11 to R14 included in the range E11 is the interval TK4. Also, the table interval of the charge ratios R15 to R19 included in the range E12 is the interval TK5. Interval TK4 is smaller than interval TK5.

Note that FIG. 15 shows an example in which the charge ratio R is represented by the formula (3). In other words, FIG. 15 shows a rising characteristic in which the distance D increases as the charge ratio R increases. When the charge ratio R is represented by the formula (2), the characteristic is a downward slope in which the distance D decreases as the charge ratio R increases. In this case, the charge ratio threshold is set to (1.0-charge ratio R15), the table interval for charge ratios below the charge ratio threshold is set coarsely, and the table interval for charge ratios above the charge ratio threshold is set finely.

As described above, in this modification, the table information 440 is created for each time window, and the interval between the tables is made smaller and finer at the joint portion TS of the time windows than at the portion other than the joint portion TS. This makes it possible to select the appropriate table information 440 for each time window, and to determine the distance more accurately even at the connecting portion TS where distance errors are likely to occur.

Alternatively, the table information 440 may be created so that the table intervals become smaller and finer not only at the connecting portion TS of the time windows but also at both ends of the second time window. This makes it possible to more accurately determine the distance at the end of the time window where distance errors are likely to occur.

FIG. 16 is a diagram showing an example of the configuration of table information 440C in a modified example of the embodiment. As shown in FIG. Each item of distance is provided. The time window is the same as in FIG. 13, and the table interval, charge ratio, and distance are the same as in FIG. 5, so description thereof will be omitted.

Here, two charge ratio thresholds, an upper limit and a lower limit, are set. The example of this figure shows that the upper limit charge ratio threshold is set to the charge ratio R20 and the lower limit charge ratio threshold is set to the charge ratio R15. Then, the table interval of the charge ratio group (charge ratios R11 to R14) less than the lower limit charge ratio threshold R15 is set to interval TK4, and the charge ratio group (charge ratio R15 to R19) is set to the interval TK5, and the table interval of the charge ratio group (charge ratios R20 to R24) equal to or higher than the upper limit charge ratio threshold value R20 is set to the interval TK6.

FIG. 17 is a diagram illustrating table information 440C in a modification of the embodiment. The horizontal axis of FIG. 17 indicates the charge ratio R, and the vertical axis indicates the distance D. In FIG. In FIG. 17, three ranges E11 to E13 are set according to the magnitude of the charge ratio R on the horizontal axis. The range E11 is a range in which the charge ratio is less than the charge ratio R15. The range E12 is a range in which the charge ratio is equal to or greater than the charge ratio R15 and less than the charge ratio R20. The range E13 is a range in which the charge ratio is equal to or greater than the charge ratio R20. Here, the range E11 is an example of the "first range". The range E12 is an example of the "second range". The range E13 is an example of the "third range". Also, the range E11 is an example of the "fourth range". The range E12 is an example of the "fifth range". The range E13 is an example of the "sixth range".

FIG. 17 shows the relationship between the charge ratio R and the distance D based on the table information 440C of FIG. In the table information 440C, the charge ratio R20 is set as the upper limit charge ratio threshold, and the charge ratio R15 is set as the lower limit charge ratio threshold. In this case, the table interval of the charge ratios R11 to R14 included in the range E11 is the interval TK4. Also, the table interval of the charge ratios R15 to R19 included in the range E12 is the interval TK5. Also, the table interval of the charge ratios R20 to R24 included in the range E13 is the interval TK6. Interval TK4 is smaller than interval TK5. Interval TK6 is smaller than interval TK5.

Note that FIG. 16 shows an example in which the charge ratio R is represented by the formula (3). That is, FIG. 16 shows a characteristic that rises to the right in which the distance D increases as the charge ratio R increases. When the charge ratio R is represented by the formula (2), the characteristic is a downward slope in which the distance D decreases as the charge ratio R increases. In this case, the charge ratio thresholds are set to (1.0-charge ratio R15) and (1.0-charge ratio R20). A table interval that is greater than or equal to the charge ratio threshold (1.0-charge ratio R20) and less than (1.0-charge ratio R15) is roughly set. Table intervals below the charge ratio threshold (1.0-charge ratio R20) are finely set. Also, the table interval of (1.0-charge ratio R15) or more is finely set.

As described above, in this modification, the table information 440 is created for each time window, and the table intervals are made smaller and finer at both ends of the second time window. This makes it possible to select appropriate table information 440 for each time window, and to determine the distance more accurately even at the ends of the time window where distance errors are likely to occur.

In the above description, the case where the table information 440C is created so that the table interval becomes small and fine at both ends of the second time window has been exemplified and explained, but the present invention is not limited to this. The table information 440A may be created so that the table intervals become smaller and finer at both ends of the first time window. Further, the table information 440 may be created so that the table interval becomes small and fine at both ends of one time window when the range image capturing apparatus 1 does not have a plurality of time windows.

As described above, in the distance image pickup device 1 according to the modification 1 of the embodiment, each of the charge ratios R (variables) included in the table information 440 falls within either range E11 or E12 in FIG. The included value. The range E11 is a range in which the charge ratio R is less than the charge ratio threshold value R15, and is an example of the "first range." The range E12 is a range in which the charge ratio R is equal to or greater than the charge ratio threshold value R15, and is an example of the "second range." The table interval in range E11 is smaller than the table interval in range E12. As a result, the distance can be accurately corrected even if the charge amount Q2# is accumulated in a biased state in which the charge amount Q2# is very small compared to the charge amount Q1#, and errors are likely to occur.

Further, in the distance image capturing device 1 according to Modification 1 of the embodiment, the pixel 321 includes three or more charge storage units CS. The storage unit 44 stores table information 440 for each combination of the first charge storage unit and the second charge storage unit among the three or more charge storage units CS. The distance image processing section 4 determines a combination of the first charge storage section and the second charge storage section based on the amount of charge stored in the three or more charge storage sections CS. In the distance image processing unit 4, among the three or more charge storage units CS, the sum of the amount of charge stored in each of the two charge storage units whose charge storage timings are consecutive is equal to that of the other two charge storage units. A combination larger than the amount of charge accumulated in each of the accumulation units is determined as the combination of the first charge accumulation unit and the second charge accumulation unit. The distance image processing section 4 selects the table information 440 according to the determined combination of the first charge storage section and the second charge storage section. Accordingly, in the modified example of the embodiment, appropriate table information 440 can be selected for each time window, and distance can be determined more accurately even when characteristics differ for each time window.

In addition, in the distance image capturing device 1 according to Modification 1 of the embodiment, the storage unit 44 stores table information 440A and table information 440B. The table information 440A is the table information 440 corresponding to the first time window, and is an example of "short distance information". Table information 440B is table information 440 corresponding to the second time window, and is an example of "long-distance information." Each charge ratio R (variable) included in table information 440A is a value included in either range E2 or E3 in FIG. The table spacing in range E3 is smaller than the table spacing in range E2. Each of the charge ratios R (variables) included in table information 440B is a value included in either range E11 or E12 in FIG. The table interval in range E11 is smaller than the table interval in range E12. As a result, the range image capturing apparatus 1 according to the modification of the embodiment can select the appropriate table information 440 for each time window, and can select a more appropriate distance at the time window connecting portion TS where distance errors are likely to occur. can be selected. Therefore, the distance can be determined with high accuracy.

(Modification 2 of Embodiment)
Modification 2 of the embodiment will now be described. In this modification, the charge ratio R is calculated without specifying the charge storage portion CS in which only the external light is stored or the two charge storage portions CS (distance calculation charge storage portions) in which the reflected light RL is divided and stored. is different from the above-described embodiment in that it calculates .

In this modification, the distance calculator 42 uses the method described in Patent Document WO2019/031510. Patent document WO2019/031510 describes a technique of selecting an arithmetic expression to be used for distance calculation depending on whether an index value exceeds a predetermined threshold. The index value here is the "distance data validity determination signal" in Patent Document WO2019/031510. Further, the arithmetic expression here is the "distance reference value" in Patent Document WO2019/031510, and corresponds to the "charge ratio R" of the present embodiment. Specific methods for calculating the charge ratio R will be described below separately for the case where the pixel 321 includes three charge storage units CS and the case where the pixel 321 includes four charge storage units CS.

(When the pixel 321 has three charge storage units CS)
In this modification, the distance calculator 42 calculates the charge ratio R using the following equation (8) or (9). Here, according to the timing charts shown in FIGS. 4 and 8, charges are accumulated in the order of the charge accumulation units CS1, CS2, and CS3. That is, the distance calculation unit 42 controls the charge accumulation units CS1, CS2, and CS3 so that charges are accumulated in this order at the timing synchronized with the irradiation of the light pulse PO. In this case, the charge storage section CS1 is an example of the "first charge storage section". The charge storage section CS2 is an example of a "second charge storage section". The charge storage section CS3 is an example of a "third charge storage section". Also, the charge amount accumulated in the charge accumulation unit CS1 is an example of the "first charge amount". The charge amount accumulated in the charge accumulation unit CS2 is an example of the "second charge amount". The charge amount accumulated in the charge accumulation unit CS3 is an example of the "third charge amount".

R=1-(Q1-Q3)/SA (8)
R=(Q1-Q3)/SA (9)
however,
SA=|Q1−Q3|+Q2−0.5×SB
SB=|Q1+Q3|-|Q1-Q3|
Q1 is the amount of charge accumulated in the charge accumulation section CS1
Q2 is the amount of charge accumulated in the charge accumulation section CS2
Q3 is the charge amount accumulated in the charge accumulation section CS3

The storage unit 44 stores table information 440 in which the distance is associated with the charge ratio R for each unit accumulated charge amount Qint. The distance calculator 42 calculates the unit accumulated charge amount Qint, and uses the table information 440 corresponding to the calculated unit accumulated charge amount Qint to set the distance associated with the charge ratio R as the measured distance.

(When the pixel 321 has four charge storage units CS)
First, the drive timing of the pixel 321 when the pixel 321 has four charge storage units CS will be described. In this case, for example, the item of the readout gate transistor G4 is added to FIGS. 4 and 8, and charges are accumulated in the order of the charge accumulation units CS1, CS2, CS3, and CS4. In this case, the charge storage section CS1 is an example of the "first charge storage section". The charge storage section CS2 is an example of a "second charge storage section". The charge storage section CS3 is an example of a "third charge storage section". The charge storage section CS4 is an example of a "fourth charge storage section". Also, the charge amount accumulated in the charge accumulation unit CS1 is an example of the "first charge amount". The charge amount accumulated in the charge accumulation unit CS2 is an example of the "second charge amount". The charge amount accumulated in the charge accumulation unit CS3 is an example of the "third charge amount". The charge amount accumulated in the charge accumulation unit CS4 is an example of the "fourth charge amount".

Specifically, at the timings shown in FIGS. 4 and 8, irradiation of the light pulse PO, control to turn off the drain gate transistor GD, and control to turn on the readout gate transistors G1 to G3 are performed. Next, the vertical scanning circuit 323 turns on the readout gate transistor G4 at the timing when the charge accumulation in the charge accumulation unit CS3 is completed, and turns off the readout gate transistor G4 after the accumulation time Ta has elapsed. do. As a result, the charge photoelectrically converted by the photoelectric conversion element PD while the readout gate transistor G4 is controlled to be on is accumulated in the charge storage unit CS4 via the readout gate transistor G4. Next, the vertical scanning circuit 323 turns on the drain gate transistor GD at the timing when the charge accumulation in the charge accumulation unit CS4 is completed to discharge the charge. As a result, charges photoelectrically converted by the photoelectric conversion element PD are discarded via the drain gate transistor GD.

Based on the amount of charge accumulated in each of the charge accumulation units CS controlled at the timing as described above, the distance calculation unit 42 uses the following equation (10) or (11) to calculate the charge ratio Calculate the XR.

XR=1-(Q1-Q3)/SA (10)
XR=(Q1-Q3)/SA (11)
however,
SA=|Q1−Q3|+|Q2−Q4|
Q1 is the amount of charge accumulated in the charge accumulation section CS1
Q2 is the amount of charge accumulated in the charge accumulation section CS2
Q3 is the charge amount accumulated in the charge accumulation section CS3
Q4 is the amount of charge accumulated in the charge accumulation section CS4

Further, the distance calculator 42 calculates the charge ratio YR using the following equation (12) or (13).

YR=2-(Q2-Q4)/SA (12)
YR=1+(Q2-Q4)/SA (13)
however,
SA=|Q1−Q3|+|Q2−Q4|
Q1 is the amount of charge accumulated in the charge accumulation section CS1
Q2 is the amount of charge accumulated in the charge accumulation section CS2
Q3 is the charge amount accumulated in the charge accumulation section CS3
Q4 is the amount of charge accumulated in the charge accumulation section CS4

The distance calculator 42 compares the charge ratio XR with the threshold value ThR. The threshold ThR is set in the vicinity of the value of the charge ratio XR corresponding to the vicinity of switching of the time windows. The distance calculator 42 selects the charge ratio XR as the charge ratio R when the charge ratio XR is equal to or less than the threshold ThR. On the other hand, the distance calculator 42 selects the charge ratio YR as the charge ratio R when the charge ratio XR exceeds the threshold ThR.

The storage unit 44 stores table information 440 in which the distance is associated with the charge ratio R for each unit accumulated charge amount Qint. The distance calculator 42 calculates the unit accumulated charge amount Qint, and uses the table information 440 corresponding to the calculated unit accumulated charge amount Qint to set the distance associated with the charge ratio R as the measured distance.

As described above, in the distance image pickup device 1 according to Modification 2 of the embodiment, the pixel 321 includes three charge storage units CS1 to CS3. The distance image processing unit 4 controls the charge storage units CS1, CS2, and CS3 so that charges are accumulated in this order at timing synchronized with the irradiation of the light pulse PO. The distance calculation unit 42 converts (Q1-Q3) into the reflected light accumulated in one of the two distance calculation charge accumulation units CS as shown in the formula (8) or (9). It is assumed that the amount of charge corresponding to RL (the amount of charge for distance calculation). Q1 is the charge amount accumulated in the charge accumulation section CS1. Q3 is the amount of charge accumulated in the charge accumulation section CS3.

As a result, in the distance image pickup device 1 according to the modification 2 of the embodiment, the distance calculation accumulated in either one of the two distance calculation charge accumulation units is not specified without specifying the two distance calculation charge accumulation units. The amount of charge used can be calculated. For this reason, as indicated by SA in equation (8) or (9), the sum of the distance calculation charge amounts accumulated in the two distance calculation charge accumulation units is calculated to obtain the charge ratio R. It is possible to calculate Therefore, in the distance image pickup device 1 according to the fourth modification of the embodiment, the charge storage section CS in which only the external light is accumulated is specified according to the length of the delay time Td, and the charge amount corresponding to the external light component is determined. The charge ratio R can be easily calculated without calculating Qb. Furthermore, since the same arithmetic expression (equation (8) or (9)) can be applied to the boundaries of the two time windows, it is possible to improve the discontinuity at the boundaries of the time windows.

In addition, in the distance image capturing device 1 according to Modification 2 of the embodiment, the pixel 321 includes four charge storage units CS1 to CS4. The distance image processing unit 4 controls the charge storage units CS1, CS2, CS3, and CS4 so that charges are accumulated in this order at timing synchronized with the irradiation of the light pulse PO. The distance calculation unit 42 sets (Q1-Q3) as the signal amount calculated from the charge amount accumulated in one of the two distance calculation charge accumulation units CS. The distance calculation unit 42 sets (Q2-Q4) to the signal amount calculated from the charge amount accumulated in the other charge accumulation unit CS of the two distance calculation charge accumulation units. The distance calculation unit 42 sets |Q1−Q3|+|Q2−Q4| to be the sum of the signal amounts calculated from the charge amounts accumulated in the respective charge accumulation units CS of the two distance calculation charge accumulation units. .

As a result, in the distance image pickup device 1 according to Modification 2 of the embodiment, either the sum of the distance calculation charge amount accumulated in each of the two distance calculation charge accumulation units or the two distance calculation charge accumulation units The amount of charge for distance calculation accumulated in one of them and the amount of charge for distance calculation accumulated in the other can be calculated. Therefore, the charge ratio R can be calculated without specifying the two distance calculation charge storage units.

Furthermore, in this case, the charge ratio XR in the equation (10) and the charge ratio YR in the equation (12) have the same value at the boundaries of the time windows. Therefore, it is possible to improve the discontinuity at the boundaries of the time windows.

In the above-described embodiment, the distance image processing unit 4 includes the storage unit 44, but the configuration is not limited to this. The storage unit 44 may be provided at least in the distance image capturing device 1 , and may be provided in a functional unit different from the distance image processing unit 4 .

Further, in the above-described embodiment, the case where the difference between the charge ratios R is used as the table interval has been described as an example, but the present invention is not limited to this. The table interval may be the distance D difference. When creating the table information 440, an object such as a wall whose measurement distance is uniform is placed as the object OB in the measurement space, and the reflected light RL Charges are accumulated and the relationship between the charge ratio R and the distance D is obtained. In this case, the amount of charge corresponding to the reflected light RL is unevenly accumulated in the vicinity of the minimum value and maximum value of the measurement range, and errors in the distance are likely to occur.

For example, consider a configuration in which the measurable range in one time window is approximately 1.8 m and three time windows are provided. In this case, errors are likely to occur in the range of distances corresponding to the ends of each time window. The distance corresponding to the end of the time window is, for example, a range of approximately 1.6 m to 1.8 m corresponding to the end of the first time window. Also, the distances are in the range of about 1.8 m to 2.0 m and 3.4 m to 3.6 m corresponding to both ends of the second time window. Also, the distances are in the range of about 3.6 m to 3.8 m and 5.2 m to 5.4 m corresponding to both ends of the third time window. In these distance ranges, the table information 440 is created so that the table intervals are small and fine. Specifically, the table information 440 is created by accumulating the reflected light RL charge while changing the distance from the depth image capturing device 1 to the object OB (wall) finely (for example, in increments of 0.05 m). On the other hand, the table information 440 is created so that the table intervals are large and coarse in the distance range that does not correspond to the edge of the time window. Specifically, the table information 440 is created by accumulating the reflected light RL charges while roughly changing the distance from the depth image capturing device 1 to the object OB (wall) (for example, in increments of 0.5 m).

Further, in at least one embodiment described above, the charge ratio R of either one of the two distance-calculation charge storage units to the sum of the distance-calculation charge amount accumulated in each of the two distance-calculation charge storage units is is the ratio of charge amounts for distance calculation. However, it is not limited to this. The charge ratio R may be a ratio indicated by using the distance calculation charge amount accumulated in each of at least two distance calculation charge accumulation units. For example, the charge ratio R may be the ratio of the distance calculation charge amount of one of the two distance calculation charge storage units to the distance calculation charge amount of the other.

Further, in FIG. 9, by accumulating charges in the charge accumulating section CS1 before irradiating the light pulse PO, control is performed so that only the charge amount Qb corresponding to the external light component is accumulated in the charge accumulating section CS1. was explained as an example. However, it is not limited to this. After irradiating the light pulse PO and receiving the reflected light RL, the electric charge may be accumulated in the electric charge accumulating section CS3. In this case, only the charge amount Qb corresponding to the external light component is accumulated in the charge accumulation section CS3. In this case, the charge storage section CS3 is an example of the "predetermined external light storage charge storage section".

In addition, as the charge amount Qb corresponding to the external light component, the charge amount corresponding to the external light component acquired in another frame, or the charge amount acquired by providing the specific external light measurement pixel 321 in the distance image capturing device 1. A charge amount corresponding to the external light component may be used. Using the above method, for example, even if the pixel 321 has only two charge storage units CS, the amount of charge corresponding to the external light component can be determined from the amount of charge stored in each of the two charge storage units CS. Amount can be subtracted. Therefore, it is possible to calculate the charge ratio R even if the pixel 321 has only two charge storages CS.

All or part of the distance image capturing device 1 and the distance image processing unit 4 in the above-described embodiments may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing a part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be implemented using a programmable logic device such as FPGA.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

REFERENCE SIGNS LIST 1 distance image pickup device 2 light source unit 3 light receiving unit 32 distance image sensor 321 pixel 323 vertical scanning circuit 4 distance image processing unit 41 timing control unit 42 distance calculation unit 43 measurement control unit 44 Memory unit 440 Table information CS Charge accumulation unit PO Optical pulse

Claims (12)

a light source unit that irradiates a light pulse into a measurement space in which an object exists;
A pixel including a photoelectric conversion element that generates a charge according to incident light and a plurality of charge storage units that store the charge, and the charge is stored in the pixel at a predetermined timing synchronized with the irradiation of the light pulse. a light-receiving unit having a pixel drive circuit that distributes and accumulates the electric charge in each of the units;
a storage unit that stores table information in which charge ratios are associated with corresponding distances to a subject;
a distance image processing unit that determines a measured distance to the subject using the amount of charge accumulated in each of the charge accumulation units and the table information;
with
The charge ratio is calculated from the amount of charge accumulated in each of two or more charge accumulation units, among the plurality of charge accumulation units, in which the charges corresponding to the light reflected by the light pulse from the subject are distributed and accumulated. A ratio indicated by using the distance calculation charge amount obtained by subtracting the charge amount corresponding to the external light component,
The table information includes the charge ratio and the corresponding distance for table intervals corresponding to the difference between the adjacent variables when the variables, which are either the charge ratio or the corresponding distance, are arranged in ascending or descending order. the table interval of the variables in the unevenly distributed state in which the ratio is above a predetermined upper limit value or below the predetermined lower limit value is smaller than the table interval of the variables not in the unevenly distributed state;
The distance image processing unit calculates the charge ratio based on the charge amount accumulated in each of the plurality of charge accumulation units, acquires the corresponding distance corresponding to the calculated charge ratio from the table information, determining the measured distance using the obtained corresponding distance;
Range imaging device. Among the plurality of charge storage units, the charge storage unit in which the charge corresponding to the reflected light reflected by the light pulse from the subject is first stored is defined as a first charge storage unit, and the charge corresponding to the reflected light is stored in the first charge storage unit. The charge storage portion that is stored next to the first charge storage portion is defined as a second charge storage portion, and the charge amount obtained by subtracting the charge amount corresponding to the external light component from the first charge amount stored in the first charge storage portion is obtained. a first distance calculation charge amount, and a second distance calculation charge amount obtained by subtracting a charge amount corresponding to an external light component from the second charge amount accumulated in the second charge accumulation unit;
The charge ratio is a ratio of the second distance calculation charge amount to the sum of the first distance calculation charge amount and the second distance calculation charge amount,
The uneven distribution state is a state in which the charge ratio exceeds a predetermined upper limit or falls below a predetermined lower limit.
The distance image pickup device according to claim 1. Each of the variables included in the table information includes a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is greater than or equal to the first threshold and less than a second threshold, and A value included in any one of a third range equal to or greater than a second threshold, and the table interval in the third range is smaller than the table interval in the second range.
3. The distance image pickup device according to claim 1 or 2. Each of the variables included in the table information includes a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is greater than or equal to the first threshold and less than a second threshold, and A value included in any one of a third range equal to or greater than a second threshold, and the table interval in the first range is smaller than the table interval in the second range.
The range imaging device according to any one of claims 1 to 3. The distance image processing unit calculates the charge ratio based on the charge amount accumulated in each of the plurality of charge accumulation units, and calculates a first distance associated with a charge ratio smaller than the calculated charge ratio; A second distance associated with a charge ratio greater than the calculated charge ratio is extracted from the table information, and the measured distance is determined by linearly interpolating the extracted first distance and the second distance.
The range imaging device according to any one of claims 1 to 3. the pixel comprises three or more of the charge storage units;
The distance image processing unit sets the smallest charge amount among the charge amounts accumulated in each of the three or more charge accumulation units as the charge amount corresponding to the external light component,
The range imaging device according to any one of claims 1 to 5. the pixel comprises three or more of the charge storage units;
The distance image processing unit controls the external light charge accumulation unit so that the charge corresponding to the reflected light is not accumulated in a predetermined external light charge accumulation unit among the three or more charge accumulation units. controlling the timing for accumulating electric charge in the external light accumulating charge accumulating unit, and setting the amount of electric charge accumulated in the electric charge accumulating unit for accumulating external light as the electric charge amount corresponding to the external light component;
The range imaging device according to any one of claims 1 to 5. the pixel is provided with a first charge storage unit, a second charge storage unit, and a third charge storage unit, which are the three charge storage units;
The distance image processing unit is
controlling so that the charge is accumulated in the order of the first charge storage unit, the second charge storage unit, and the third charge storage unit at a timing synchronized with the irradiation of the light pulse;
A difference between the first charge amount and the third charge amount using the first charge amount accumulated in the first charge accumulation unit and the third charge amount accumulated in the third charge accumulation unit A first calculated value is calculated, and the first calculated value is stored in two distance calculation units in which the amount of charge corresponding to the reflected light reflected by the light pulse from the subject is divided and accumulated among the three charge accumulation units. A charge amount corresponding to the reflected light accumulated in one of the charge accumulation units;
The range imaging device according to any one of claims 1 to 5. The pixel is provided with a first charge storage portion, a second charge storage portion, a third charge storage portion, and a fourth charge storage portion, which are the four charge storage portions,
The distance image processing unit is
The pixels are arranged such that the charges are accumulated in the order of the first charge accumulation unit, the second charge accumulation unit, the third charge accumulation unit, and the fourth charge accumulation unit at timing synchronized with the irradiation of the light pulse. control the drive circuit,
A difference between the first charge amount and the third charge amount using the first charge amount accumulated in the first charge accumulation unit and the third charge amount accumulated in the third charge accumulation unit Calculate a first calculated value,
A difference between the second charge amount and the fourth charge amount using the second charge amount accumulated in the second charge accumulation unit and the fourth charge amount accumulated in the fourth charge accumulation unit Calculate a second calculated value,
The added value obtained by adding the absolute value of the first calculated value and the absolute value of the second calculated value is distributed to the amount of charge corresponding to the reflected light of the light pulse reflected by the object among the four charge storage units. and the first calculated value is the sum of the distance calculation charge amounts corresponding to the reflected light accumulated in the two distance calculation charge accumulating units respectively. one of the charge amounts for distance calculation in the charge accumulating section, and the second calculated value as the other of the charge amounts for distance calculation in the two distance calculation charge accumulating sections;
The range imaging device according to any one of claims 1 to 5. The pixel includes three or more charge storage units, and the charge storage unit in which, among the three or more charge storage units, the charge corresponding to the light reflected by the light pulse from the subject is accumulated first. is a first charge storage unit, and the charge storage unit in which the charge corresponding to the reflected light is stored next to the first charge storage unit is a second charge storage unit,
the storage unit stores the table information for each combination of the first charge storage unit and the second charge storage unit;
In the distance image processing section, the sum of the amount of charge accumulated in each of two of the three or more charge accumulation sections whose charge accumulation timings are consecutive is equal to the sum of the amounts of charges accumulated in the other two charge accumulation sections. determining a combination of the first charge storage section and the second charge storage section that is larger than the amount of charge stored in each of the storage sections; and determining the determined first charge storage section and the second charge storage section. selecting the table information according to a combination of
The distance image pickup device according to claim 1. A first charge accumulator is defined as one of the plurality of charge accumulators in which the amount of charge corresponding to the reflected light reflected by the light pulse from the subject is accumulated first, and the charge amount corresponding to the reflected light is the first charge accumulator. A second charge storage unit is a charge storage unit that is stored next to the first charge storage unit;
The storage section stores the table information corresponding to the combination of the first charge storage section and the second charge storage section in which an amount of charge corresponding to the reflected light reflected by the subject at a short distance is accumulated. long-distance use information, and the long-distance use information, which is the table information corresponding to the combination of the first charge storage section and the second charge storage section in which the amount of charge corresponding to the reflected light reflected by the subject at the long distance is accumulated. remember information and
Each of the variables included in the short distance information includes a first range in which the charge ratio is less than a first threshold, a second range in which the charge ratio is greater than or equal to the first threshold and less than a second threshold, and the charge. a value within any one of a third range in which the ratio is equal to or greater than a second threshold, wherein the table interval in the third range is smaller than the table interval in the second range;
Each of the variables included in the long distance information includes a fourth range in which the charge ratio is less than a fourth threshold, a fifth range in which the charge ratio is greater than or equal to the fourth threshold and less than a fifth threshold, and the charge. A value included in any one of a sixth range in which the ratio is equal to or greater than a fifth threshold, and the table interval in the fourth range is smaller than the table interval in the fifth range.
The distance image pickup device according to claim 1. A light source unit that irradiates a measurement space in which an object exists with a light pulse, a pixel that includes a photoelectric conversion element that generates charges according to the incident light, and a plurality of charge storage units that store the charges, and the light pulse. a pixel drive circuit that distributes and accumulates the charge in each of the charge storage units in the pixel at a predetermined timing synchronized with the irradiation of the light receiving unit, and the corresponding distance to the subject for each variable based on the charge ratio. and a distance image processing unit that determines the measured distance to the subject by using the charge amount accumulated in each of the charge accumulation units and the table information. A range image capturing method using a range image capturing device comprising:
The charge ratio is calculated from the amount of charge accumulated in each of two or more charge accumulation units, among the plurality of charge accumulation units, in which the charges corresponding to the light reflected by the light pulse from the subject are distributed and accumulated. A ratio indicated by using the distance calculation charge amount obtained by subtracting the charge amount corresponding to the external light component,
The table information includes the charge ratio and the corresponding distance for table intervals corresponding to the difference between the adjacent variables when the variables that are either the charge ratio or the corresponding distance included in the table information are arranged in ascending or descending order. the table interval of the variables in the unevenly distributed state in which the ratio is above a predetermined upper limit value or below the predetermined lower limit value is smaller than the table interval of the variables not in the unevenly distributed state;
The distance image processing unit calculates the charge ratio based on the charge amount accumulated in each of the plurality of charge accumulation units, acquires the corresponding distance corresponding to the calculated charge ratio from the table information, determining the measured distance using the obtained corresponding distance;
Range image capturing method.

Images