JP2024011621A - Distance image capturing device and distance image capturing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform correspondence according to a tendency of a multi-pass.

SOLUTION: A distance image capturing device performs first measurement having plural measurements in which a combination of radiation time and storage time is a first condition, a time difference between radiation timing being a reference and storage timing is a first time difference and the time differences between the radiation timing and the storage timing with the first time difference as the reference are different from each other, performs second measurement having the plural measurements in which a combination of the radiation time and the storage time is a second condition, the time difference between the radiation timing being the reference and the storage timing is a second time difference and the time differences between the radiation timing and the storage timing with the second time difference as the reference are different from each other, performs the measurement in which any of the second condition and the second time difference is different from that of the first measurement in the second measurement, and calculates a distance to a subject on the basis of a tendency of a feature amount based on an electric charge amount stored in the first measurement and the second measurement.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a distance image capturing device and a distance image capturing method.

Time of Flight (TOF) uses the fact that the speed of light is known to measure the distance between a measuring instrument and an object based on the flight time of light in space (measurement space). A distance image imaging device based on the above method has been realized (see, for example, Patent Document 1). In such distance image capturing devices, the delay time from the time when a light pulse is irradiated until the reflected light that has reflected from the subject returns is determined by making the reflected light enter the image sensor and generating a charge according to the amount of reflected light. The charge is determined by distributing and accumulating the charge in a plurality of charge storage units, and the distance to the subject is calculated using the delay time and the speed of light.

Patent No. 4235729

In a distance image capturing device, an arithmetic expression for calculating a distance is defined on the assumption that a pixel receives a direct wave (single pass) that directly travels back and forth between a light source of a light pulse and an object. However, there are cases where the optical pulse is multiple-reflected at the corners of the object, or where the surface of the object has an uneven structure, and a multipath including a mixture of direct waves and indirect waves is received. When such multipath light is received, if the distance is calculated by assuming that a single path has been received, an error will occur in the measured distance.
On the other hand, in distance image capturing devices, in order to widen the distance measurement range, the time for emitting light pulses (irradiation time) and the time for accumulating charge in the charge storage section (accumulation time) are changed depending on the distance to the subject. There is. When the irradiation time and accumulation time are changed, there is a possibility that the tendency of multipath light received by a pixel differs, and it has been difficult to take measures according to such a tendency of multipath.

The present invention has been made based on the above-mentioned problem, and an object of the present invention is to provide a distance image capturing device and a distance image capturing method that can take measures according to the tendency of multipath.

The distance image imaging device of the present invention includes a pixel including a light source unit that irradiates a subject with a light pulse, a photoelectric conversion element that generates charges according to incident light, and a plurality of charge storage units that accumulate charges, and a pixel drive circuit that distributes and accumulates charges in each of the charge accumulation sections at an accumulation timing synchronized with the irradiation timing of irradiating the light pulse; and a pixel drive circuit that distributes and accumulates charges in each of the charge accumulation sections; and a distance image processing unit that calculates a distance between the distance image processing unit and the distance image processing unit, wherein the distance image processing unit has a first combination of an irradiation time for irradiating the light pulse and an accumulation time for distributing and accumulating charges in each of the charge accumulation units. The condition is that the time difference between the irradiation timing and the accumulation timing, which is a reference, is a first time difference, and the first time difference is a first time difference, and a first time difference is made of a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other with respect to the first time difference. 1 measurement is performed, the combination of the irradiation time and the accumulation time is a second condition, the time difference between the reference irradiation timing and the accumulation timing is a second time difference, and the irradiation is performed based on the second time difference. A second measurement is performed that includes a plurality of measurements in which the time difference between the timing and the accumulation timing is different from each other, and in the second measurement, either the second condition or the second time difference is different from the first measurement. A distance image imaging device that extracts a feature amount based on the amount of charge accumulated in each of the first measurement and the second measurement, and calculates a distance to the subject based on a tendency of the feature amount. .

In the distance image imaging device of the present invention, the distance image processing unit may perform a measurement in which the second time difference is the same as in the first measurement and the second condition is different from the first measurement in the second measurement. conduct.

In the distance image imaging device of the present invention, the distance image processing unit performs a measurement in which the second time difference is different from the first measurement and the second condition is the same as the first measurement in the second measurement. .

In the distance image imaging device of the present invention, the distance image processing unit may be configured to determine whether the reflected light of the light pulse is received by the pixel in a single pass, or the reflected light of the light pulse is received by the pixel in multiple passes. A multi-pass determination is performed to determine whether or not the distance is high, and the distance to the subject is calculated according to the result of the multi-pass determination.

In the distance image imaging device of the present invention, the distance image processing unit may be arranged to determine the irradiation timing and the accumulation timing when the reflected light is received by the pixel in a single pass for each combination of the irradiation time and the accumulation time. The multipath determination is performed based on the degree of similarity between the tendency of the lookup table and the tendency of the feature amount by referring to a lookup table in which the time difference between the time difference and the feature amount is associated with the feature amount.

In the distance image imaging device of the present invention, a plurality of lookup tables are created for each combination of the shape of the light pulse and the irradiation time and the accumulation time, and the distance image processing unit is configured to create a plurality of lookup tables for each combination of the shape of the light pulse and the irradiation time and the accumulation time, and the distance image processing unit Among the tables, the multipath determination is performed using the lookup table corresponding to each of the measurement conditions of the first measurement and the second measurement.

In the distance image imaging device of the present invention, the feature amount is a value calculated using at least the amount of charge corresponding to the reflected light of the light pulse, out of the amount of charge accumulated in each of the charge storage sections. .

In the distance image imaging device of the present invention, the pixel is provided with a first charge storage section, a second charge storage section, a third charge storage section, and a fourth charge storage section, and the distance image processing section is configured to At a timing when a charge corresponding to the reflected light of the optical pulse is accumulated in at least one of the first charge accumulation section, the second charge accumulation section, the third charge accumulation section, or the fourth charge accumulation section, Charges are accumulated in the order of the first charge accumulation section, the second charge accumulation section, the third charge accumulation section, and the fourth charge accumulation section, and the feature amount is determined by the first charge accumulation section, the second charge accumulation section, and the second charge accumulation section. It is a complex number whose variable is the amount of charge stored in each of the storage section, the third charge storage section, and the fourth charge storage section.

In the distance image imaging device of the present invention, the feature amount is a first charge amount that is a difference between a first charge amount accumulated in the first charge storage section and a third charge amount accumulated in the third charge storage section. A complex number whose real part is a variable and whose imaginary part is a second variable that is the difference between the second amount of charge stored in the second charge storage section and the fourth amount of charge stored in the fourth charge storage section. is the value represented.

In the distance image imaging device of the present invention, the distance image processing section delays the irradiation timing with respect to the accumulation timing in the first measurement and the second measurement, thereby adjusting the irradiation timing and the accumulation timing. Perform multiple measurements with different time differences.

In the distance image imaging device of the present invention, the distance image processing unit performs a temporary measurement to calculate the distance to the subject without determining whether the image is a single pass or a multipass, and according to the distance calculated in the temporary measurement. and determining at least one of the first condition and the second condition.

In the distance image imaging device of the present invention, when determining that the subject is relatively close according to the distance calculated in the provisional measurement, the distance image processing unit may adjust the irradiation time under the second condition to When the second condition is determined such that the combination of the accumulation times is shorter than the first condition, and it is determined that the subject is relatively far away, the irradiation time and the accumulation under the second condition are determined such that the combination of the accumulation times is shorter than the first condition. The second condition is determined such that the combination of times is longer than the first condition.

In the distance image imaging device of the present invention, the distance image processing unit performs a temporary measurement to calculate the distance to the subject without determining whether the image is a single pass or a multipass, and according to the distance calculated in the temporary measurement. and determining the second time difference.

In the distance image capturing device of the present invention, the distance image processing section corrects the distance calculated in the second measurement according to the distance based on the second time difference, and sets the corrected distance to the distance to the subject. do.

In the distance image imaging device of the present invention, the distance image processing unit calculates an index value indicating the degree of similarity between the tendency of the lookup table and the tendency of the feature amount of each of the plurality of measurements, and The value is the difference between the first feature quantity, which is the feature quantity calculated from each of the plurality of measurements, and the second feature quantity, which is the feature quantity, corresponding to each of the plurality of measurements in the lookup table. is an added value obtained by adding the difference normalized values of each of the plurality of measurements to the difference normalized value normalized by the absolute value of the second feature amount, and the distance image processing unit calculates the index value If does not exceed a threshold, it is determined that the reflected light has been received by the pixel in a single pass, and if the index value exceeds the threshold, it is determined that the reflected light has been received by the pixel in a multi-pass. .

In the distance image imaging device of the present invention, when the distance image processing unit determines that the reflected light is received by the pixel in a multipath, the distance image processing unit minimizes the distance corresponding to each of the light paths included in the multipath. Calculated using the square method.

In the distance image imaging device of the present invention, the distance image processing section controls the intensity of irradiating the light pulse in the first measurement and the second measurement, depending on the distance calculated in the temporary measurement.

The distance image imaging device of the present invention further includes a charge discharging unit that discharges the charges generated by the photoelectric conversion element, and the range image processing unit discards the charges generated by the photoelectric conversion element at a timing different from the accumulation timing. control is performed so that the discharged charges are discharged by the charge discharging section.

The distance image imaging method of the present invention includes a pixel including a light source section that irradiates a subject with a light pulse, a photoelectric conversion element that generates charges according to incident light, and a plurality of charge storage sections that accumulate charges; a pixel drive circuit that distributes and accumulates charges in each of the charge accumulation sections at an accumulation timing synchronized with the irradiation timing of irradiating the light pulse; and a pixel drive circuit that distributes and accumulates charges in each of the charge accumulation sections; A distance image imaging method carried out by a distance image imaging device comprising: a distance image processing unit that calculates a distance of the charge storage unit; A first condition is a combination of accumulation times for distributing and accumulating charges, a time difference between the reference irradiation timing and the accumulation timing is a first time difference, and the irradiation timing and the accumulation are determined based on the first time difference. A first measurement consisting of a plurality of measurements with different time differences from each other is performed, a combination of the irradiation time and the accumulation time is a second condition, and a time difference between the reference irradiation timing and the accumulation timing is a second condition. A second measurement is performed that includes a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other based on the second time difference, and in the second measurement, the second measurement is performed under the second condition or the second time difference. Either one performs a measurement different from the first measurement, extracts a feature amount based on the amount of charge accumulated in each of the first measurement and the second measurement, and based on the tendency of the feature amount. A distance to the subject is calculated.

According to the present invention, it is possible to take measures according to the tendency of multipath.

FIG. 1 is a block diagram showing a schematic configuration of a distance image capturing device 1 according to an embodiment. It is a block diagram showing a schematic structure of distance image sensor 32 of an embodiment. FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 according to the embodiment. FIG. 3 is a diagram illustrating multipath according to an embodiment. It is a figure explaining the process which the distance image processing part 4 of embodiment performs. FIG. 2 is a diagram schematically showing an example in which a conventional distance image capturing device measures a subject OB. FIG. 2 is a diagram schematically showing an example in which a conventional distance image capturing device measures a subject OB. FIG. 2 is a diagram schematically showing an example in which a conventional distance image capturing device measures a subject OB. FIG. 2 is a diagram schematically showing an example in which a conventional distance image capturing device measures a subject OB. It is a figure explaining the measurement method of a 1st embodiment. It is a figure explaining the measurement method of a 1st embodiment. It is a figure explaining the measurement method of a 1st embodiment. It is a figure explaining the measurement method of a 1st embodiment. It is a figure showing an example of complex function CP (phi) of an embodiment. It is a figure showing an example of complex function CP (phi) of an embodiment. It is a figure explaining the process which the distance image processing part 4 of embodiment performs. It is a figure explaining the process which the distance image processing part 4 of embodiment performs. It is a figure explaining the process which the distance image processing part 4 of embodiment performs. It is a figure explaining the process which the distance image processing part 4 of embodiment performs. It is a flowchart showing the flow of processing performed by the distance image imaging device 1 of the embodiment. FIG. 3 is a diagram showing an example of a lookup table. It is a figure explaining the measurement method of 2nd Embodiment. It is a figure explaining the measurement method of 2nd Embodiment.

Hereinafter, a distance image capturing device according to an embodiment will be described with reference to the drawings.

FIG. 1 is a block diagram showing a schematic configuration of a distance image capturing device according to an embodiment. The distance image imaging device 1 includes, for example, a light source section 2, a light receiving section 3, and a distance image processing section 4. FIG. 1 also shows a subject OB, which is an object whose distance is to be measured in the distance image capturing device 1.

The light source section 2 irradiates a measurement target space in which a subject OB whose distance is to be measured in the distance image capturing apparatus 1 exists with a light pulse PO under control from the distance image processing section 4 . The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source section 2 includes a light source device 21 and a diffusion plate 22.

The light source device 21 is a light source that emits laser light in a near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes a light pulse PO that is irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light emitting device. The light source device 21 emits pulsed laser light under control from the timing control section 41.

The diffuser plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 over a surface that irradiates the subject OB. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated onto the object OB.

The light receiving unit 3 receives the reflected light RL of the optical pulse PO reflected by the subject OB whose distance is to be measured in the distance image capturing device 1, and outputs a pixel signal according to the received reflected light RL. The light receiving section 3 includes a lens 31 and a distance image sensor 32.

The lens 31 is an optical lens that guides the incident reflected light RL to the distance image sensor 32. The lens 31 emits the incident reflected light RL to the distance image sensor 32 side, and causes the light to be received (incident) by a pixel provided in a light receiving area of the distance image sensor 32.

The distance image sensor 32 is an image sensor used in the distance image imaging device 1. The distance image sensor 32 includes a plurality of pixels in a two-dimensional light receiving area. Each pixel of the distance image sensor 32 is provided with one photoelectric conversion element, a plurality of charge storage sections corresponding to this one photoelectric conversion element, and a component that distributes charge to each charge storage section. . In other words, a pixel is an image sensor having a distribution configuration in which charges are distributed and accumulated in a plurality of charge storage sections.

The distance image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge storage sections in accordance with the control from the timing control section 41. Further, the distance image sensor 32 outputs a pixel signal according to the amount of charge distributed to the charge storage section. The distance image sensor 32 has a plurality of pixels arranged in a two-dimensional matrix, and outputs pixel signals for one frame to which each pixel corresponds.

The distance image processing unit 4 controls the distance image imaging device 1 and calculates the distance to the object OB. The distance image processing section 4 includes a timing control section 41, a distance calculation section 42, and a measurement control section 43.

The timing control section 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control section 43. The various control signals here include, for example, a signal for controlling the irradiation of the optical pulse PO, a signal for distributing and accumulating the reflected light RL in a plurality of charge storage units, a signal for controlling the number of accumulations per frame, etc. be. The number of times of accumulation is the number of times that the process of distributing and accumulating charges in the charge accumulating section CS (see FIG. 3) is repeated. The exposure time is the product of this number of times of accumulation and the time width (accumulation time width) in which charges are accumulated in each charge accumulation section per process of distributing and accumulating charges.

The distance calculation unit 42 outputs distance information obtained by calculating the distance to the object OB based on the pixel signal output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time from irradiation of the optical pulse PO to reception of the reflected light RL based on the amount of charge accumulated in the plurality of charge storage units. The distance calculation unit 42 calculates the distance to the object OB according to the calculated delay time.

The measurement control section 43 controls the timing control section 41. For example, the measurement control unit 43 sets the number of times of accumulation of one frame and the accumulation time width, and controls the timing control unit 41 so that imaging is performed according to the set contents.

With such a configuration, in the distance image imaging device 1, the light receiving unit 3 receives the reflected light RL, which is the light pulse PO in the near-infrared wavelength band that the light source unit 2 irradiated onto the subject OB, and is reflected by the subject OB. The distance image processing unit 4 outputs distance information obtained by measuring the distance to the object OB.

Note that although FIG. 1 shows a distance image imaging device 1 having a configuration in which the distance image processing unit 4 is provided inside the distance image imaging device 1, the distance image processing unit 4 is provided outside the distance image imaging device 1. It may be a component provided.

Here, the configuration of the distance image sensor 32 used as an image sensor in the distance image imaging device 1 will be explained using FIG. 2. FIG. 2 is a block diagram showing a schematic configuration of an image sensor (distance image sensor 32) used in the distance image imaging device 1 of the embodiment.

As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, A pixel signal processing circuit 325 is provided.

The light receiving area 320 is an area in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which they are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixel 321 accumulates charges corresponding to the amount of light received. The control circuit 322 controls the distance image sensor 32 in an integrated manner. The control circuit 322 controls the operations of the components of the distance image sensor 32, for example, in accordance with instructions from the timing control section 41 of the distance image processing section 4. Note that the components included in the distance image sensor 32 may be directly controlled by the timing control section 41, and in this case, the control circuit 322 may be omitted.

The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving area 320 row by row in accordance with the control from the control circuit 322. The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge accumulated in each charge accumulation section CS of the pixel 321. In this case, the vertical scanning circuit 323 distributes and accumulates the charges converted by the photoelectric conversion element in each of the charge storage sections of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel drive circuit."

The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing) on the voltage signal output from the pixels 321 of each column to the corresponding vertical signal line in accordance with the control from the control circuit 322. This circuit performs A/D conversion processing, etc.).

The horizontal scanning circuit 324 is a circuit that sequentially outputs the signals output from the pixel signal processing circuit 325 to the horizontal signal line in accordance with the control from the control circuit 322. As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to the distance image processing section 4 via the horizontal signal line.

In the following description, it is assumed that the pixel signal processing circuit 325 performs A/D conversion processing and the pixel signal is a digital signal.

Here, the configuration of the pixel 321 arranged in the light receiving area 320 provided in the distance image sensor 32 will be explained using FIG. 3. FIG. 3 is a circuit diagram showing an example of the configuration of a pixel 321 arranged within the light receiving area 320 of the distance image sensor 32 of the embodiment. FIG. 3 shows an example of the configuration of one pixel 321 among the plurality of pixels 321 arranged in the light receiving area 320. The pixel 321 is an example of a configuration including four pixel signal readout sections.

The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and three pixel signal readout units RU that output voltage signals from the corresponding output terminals O. Each of the pixel signal readout units RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a selection gate transistor SL. In each pixel signal readout unit RU, a charge storage unit CS is configured by a floating diffusion FD and a charge storage capacitor C.

In addition, in FIG. 3, each pixel signal readout unit RU is distinguished by adding any number from “1” to “3” after the code “RU” of the three pixel signal readout units RU. do. Similarly, each component included in the three pixel signal readout units RU is indicated by a number representing each pixel signal readout unit RU after the code, so that each component can read out the pixel signal to which it corresponds. The unit RU is distinguished from each other.

In the pixel 321 shown in FIG. 3, the pixel signal readout unit RU1 that outputs a voltage signal from the output terminal O1 includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitor C1, a reset gate transistor RT1, and a source follower. It includes a gate transistor SF1 and a selection gate transistor SL1. In the pixel signal readout section RU1, a charge storage section CS1 is configured by a floating diffusion FD1 and a charge storage capacitor C1. The pixel signal reading units RU2 to RU3 also have a similar configuration.

Note that the configuration of the pixels arranged in the distance image sensor 32 is not limited to the configuration including three pixel signal readout units RU as shown in FIG. 3, but may include a plurality of pixel signal readout units RU. Any pixel with the above configuration may be used. That is, the number of pixel signal readout units RU (charge storage units CS) provided in pixels arranged in the distance image sensor 32 may be two, or may be four or more.

Further, in the pixel 321 having the configuration shown in FIG. 3, an example is shown in which the charge storage section CS is configured by a floating diffusion FD and a charge storage capacitor C. However, the charge storage section CS only needs to be configured by at least the floating diffusion FD, and the pixel 321 may not include the charge storage capacitor C.

In addition, in the pixel 321 having the configuration shown in FIG. 3, an example of the configuration including the drain gate transistor GD is shown, but when there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, may be configured without the drain-gate transistor GD.

The photoelectric conversion element PD is an embedded photodiode that photoelectrically converts incident light to generate charges and stores the generated charges. The structure of the photoelectric conversion element PD may be arbitrary. The photoelectric conversion element PD may be, for example, a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. It may also be a PIN photodiode. Further, the photoelectric conversion element PD is not limited to a photodiode, and may be a photogate type photoelectric conversion element, for example.

In the pixel 321, the photoelectric conversion element PD converts the incident light at the accumulation timing synchronized with the irradiation timing of the light pulse PO into electric charge, and distributes the converted electric charge to each of the four charge storage parts CS. Let it accumulate. Furthermore, regarding light incident on the pixel 321 at a timing other than the accumulation timing, the charges converted by the photoelectric conversion element PD are discharged from the drain gate transistor GD to prevent them from being accumulated in the charge accumulation section CS.

In this way, after the accumulation of charges at the accumulation timing and the discarding of charges at timings other than the accumulation timing are repeatedly performed over one frame, a readout period is provided. During the read period, the horizontal scanning circuit 324 outputs to the distance calculation section 42 an electrical signal corresponding to the amount of charge for one frame, which is accumulated in each of the charge storage sections CS.

In this way, by driving the pixel 321 over one frame, the amount of charge corresponding to the reflected light RL is adjusted at a ratio corresponding to the delay time Td until the reflected light RL enters the distance image capturing device 1. The charges are distributed and stored in two of the four charge storage units CS included in the pixel 321. The distance calculation unit 42 uses this property to calculate the delay time Td using the following equation (1). Note that in formula (1), it is assumed that the amount of charge corresponding to the external light component out of the amount of charge accumulated in the charge storage units CS1 and CS2 is the same amount as the amount of charge accumulated in the charge accumulation unit CS3. shall be.

Td=To×(Q2-Q3)/(Q1+Q2-2×Q3)...Equation (1)
However, To is the period during which the optical pulse PO was irradiated, Q1 is the amount of charge accumulated in the charge storage section CS1, Q2 is the amount of charge accumulated in the charge accumulation section CS2, and Q3 is the amount of charge accumulated in the charge accumulation section CS3.

The distance calculation unit 42 calculates the round-trip distance to the subject S by multiplying the delay time Td obtained by equation (1) by the speed of light (velocity). Then, the distance calculation unit 42 calculates the distance to the subject S by halving the round trip distance calculated above.

Next, the multipath of the embodiment will be explained using FIG. 4. FIG. 4 is a diagram illustrating multipath according to the embodiment. The distance image capturing device 1 uses a light source with a wider irradiation range than a Lider (Light Detection and Ranging) or the like. For this reason, while it has the advantage of being able to measure a certain range of space at once, it has the disadvantage of being susceptible to multipath. The example in FIG. 4 schematically shows how the distance image capturing device 1 irradiates the measurement space E with a light pulse PO and receives a plurality of reflected waves (multipaths) including a direct wave W1 and an indirect wave W2. has been done. In the following explanation, a case where a multipath is constituted by two reflected waves will be exemplified and explained. However, the present invention is not limited to this, and the multipath may be composed of three or more reflected waves. The method described below can also be applied when a multipath is composed of three or more reflected waves.

When multi-pass light is received, the shape (time-series change) of the reflected light received by distance image imaging device 1 is different from that when only single pass light is received.

For example, in the case of a single pass, reflected light (direct wave W1) having the same shape as the optical pulse is received by the distance image imaging device 1 after a delay time Td. On the other hand, in the case of multipath, in addition to the direct wave, reflected light (indirect wave W2) having the same shape as the optical pulse is received with a delay of Td+α. α here is the time that the indirect wave W2 is delayed with respect to the direct wave W1. That is, in the case of multi-pass, the distance image imaging device 1 receives reflected light in which a plurality of lights having the same shape as the light pulse are added together with a time difference between them.

That is, in the case of multi-pass and single-pass, reflected light having different shapes (time-series changes) is received. Equation (1) above is a mathematical expression based on the premise that the delay time is the time required for the optical pulse to directly travel back and forth between the light source and the object. That is, equation (1) assumes that the distance image capturing device 1 receives light in a single path. Therefore, if the distance image capturing device 1 calculates the distance using equation (1) even though it receives multipath light, the calculated distance will be a distance that does not correspond to the position of the actual object OB. It becomes. For this reason, the difference between the calculated distance (measured distance) and the actual distance deviates, causing an error.

As a countermeasure for this, in this embodiment, a plurality of measurements are performed with different time differences between the irradiation timing and the accumulation timing. The irradiation timing here is the timing at which the optical pulse PO is irradiated. The accumulation timing is the timing at which charges are accumulated in each charge accumulation section CS.

FIG. 5 is a diagram illustrating a method in which the distance image processing unit 4 performs measurements multiple times while changing the time difference between the irradiation timing and the accumulation timing. FIG. 5 shows a timing chart of the pixel 321 that receives the reflected light RL after the delay time Td has elapsed after being irradiated with the optical pulse PO.

In FIG. 5, the timing of irradiating the optical pulse PO is "L", the timing of receiving the reflected light is "R", the timing of the drive signal TX1 is "G1", the timing of the drive signal TX2 is "G2", and the timing of the drive signal TX1 is "G2". The timing of TX3 is shown as "G3", and the timing of drive signal RSTD is shown as "GD". Note that the drive signal TX1 is a signal that drives the read gate transistor G1. The same applies to drive signals TX2 and TX3.

As shown in FIG. 5, the distance image processing unit 4 performs measurement multiple times (M times in the example shown) while changing the time difference between the irradiation timing and the accumulation timing. Here, M is an arbitrary natural number of 2 or more.

The irradiation time To in FIG. 5 is the time width for irradiating the optical pulse PO. The accumulation time Ta is a time width for accumulating charges in each charge accumulation section CS. The irradiation time To and the accumulation time Ta have the same time width. The equivalent time width includes a case where the irradiation time To and the accumulation time Ta are the same time width, and a case where the irradiation time To is longer than the accumulation time Ta by a predetermined time. The predetermined time here is determined depending on the rounding of the waveform of the optical pulse PO, the amount of noise accumulated in the charge accumulation section CS, and the like.

First, the distance image processing unit 4 performs the first measurement. In the first measurement, the time difference between the irradiation timing and the accumulation timing is set to 0 (zero). That is, in the first measurement, the irradiation timing and the accumulation timing are set to be the same timing. During the unit accumulation time UT, the distance image processing section 4 turns on the charge storage section CS1 at the same time as irradiating the optical pulse PO, and thereafter turns on the charge storage sections CS2 and CS3 in order, thereby turning on the charge storage sections CS1 to CS3. An accumulation process is performed to accumulate charge in each of the . After repeating such accumulation processing a predetermined number of accumulation times, the distance image processing section 4 reads out a signal value corresponding to the amount of charge accumulated in each of the charge accumulation sections CS during the readout time RD.

Next, the distance image processing unit 4 performs a second measurement. In the second measurement, the time difference between the irradiation timing and the accumulation timing is set as the irradiation delay time Dtm2. That is, in the second measurement, the irradiation timing is delayed by the irradiation delay time Dtm2 with respect to the accumulation timing. Since the irradiation timing is delayed by the irradiation delay time Dtm2 in the second measurement, the reflected light RL is received by the pixel 321 with a delay of (delay time Td+irradiation delay time Dtm2) from the irradiation timing. After repeating the accumulation process having such an irradiation delay time Dtm2 a predetermined number of accumulation times, the distance image processing unit 4 calculates a signal value corresponding to the amount of charge accumulated in each of the charge accumulation units CS during the readout time RD. Read out.

Next, the distance image processing unit 4 performs the (M-1)th measurement. In the (M-1)th measurement, the time difference between the irradiation timing and the accumulation timing is set as the irradiation delay time Dtm3. That is, in the (M-1)th measurement, the irradiation timing is delayed by the irradiation delay time Dtm3 with respect to the accumulation timing. Since the irradiation timing is delayed by the irradiation delay time Dtm3 in the (M-1)th measurement, the reflected light RL is received by the pixel 321 with a delay of (delay time Td+irradiation delay time Dtm3) from the irradiation timing. After repeating the accumulation process having such an irradiation delay time Dtm3 a predetermined number of accumulation times, the distance image processing unit 4 calculates a signal value corresponding to the amount of charge accumulated in each of the charge accumulation units CS during the readout time RD. Read out.

Next, the distance image processing unit 4 performs the M-th measurement. In the M-th measurement, the time difference between the irradiation timing and the accumulation timing is set as an irradiation delay time Dtm4. That is, in the M-th measurement, the irradiation timing is delayed by the irradiation delay time Dtm4 with respect to the accumulation timing. Since the irradiation timing is delayed by the irradiation delay time Dtm4 in the M-th measurement, the reflected light RL is received by the pixel 321 with a delay of (delay time Td+irradiation delay time Dtm4) from the irradiation timing. After repeating the accumulation process having such an irradiation delay time Dtm4 a predetermined number of accumulation times, the distance image processing unit 4 calculates a signal value corresponding to the amount of charge accumulated in each of the charge accumulation units CS during the readout time RD. Read out.

In this embodiment, the distance image processing unit 4 performs measurements multiple times while changing the time difference between the irradiation timing and the accumulation timing, and each time the measurement is performed, the distance image processing unit 4 A feature amount (complex variable CP to be described later) based on the amount of charge is calculated. A specific method by which the distance image processing unit 4 calculates the complex variable CP will be described in detail later.

The distance image processing unit 4 determines whether the pixel 321 has received single-pass light or multi-pass light according to the calculated feature amount.

If the tendency of the feature amount calculated according to each of the plurality of measurements is similar to the tendency of the feature amount when the pixel 321 receives a single pass, the distance image processing unit 4 determines that the pixel 321 has received a single pass. It is determined that For example, the distance image processing unit 4 stores in advance information associating the time difference between the irradiation timing and the accumulation timing with the feature amount when the pixel 321 receives a single pass as data (a lookup table LUT described later). I'll let it happen. The specific contents of the lookup table LUT will be explained in detail later.

The distance image processing unit 4 calculates the degree to which the tendency of the feature amount calculated for each of the plurality of measurements is similar to the tendency of the lookup table LUT (SD index described later). The distance image processing unit 4 determines whether the pixel 321 has received single-pass light by comparing the calculated SD index with a threshold value. A specific method by which the distance image processing unit 4 calculates the SD index will be described in detail later.

Thereby, the distance image processing unit 4 determines that the pixel 321 has received a single pass when the trend of the feature amount is similar to the trend of the lookup table LUT, and the trend of the feature amount is similar to the trend of the lookup table LUT. If they are not similar, it can be determined that the pixel 321 has received multipath light.

When the distance image processing unit 4 determines that the pixel 321 has received single-pass light, the distance image processing unit 4 calculates the distance using a relational expression assuming a single reflector, for example, Expression (1). On the other hand, when the distance image processing unit 4 determines that the pixel 321 has received multipath light, the distance image processing unit 4 calculates the distance by another means without using equation (1). Thereby, the distance image processing unit 4 can calculate the distance depending on whether or not a single path has been received, and it is possible to reduce errors occurring in the distance.

However, if an attempt is made to perform multiple measurements while changing the time difference between the irradiation timing and the accumulation timing, it may be difficult to determine whether or not there is multipath depending on the position where the object OB is present. A case in which it is difficult to determine whether or not there is multipath will be described using FIG. 6 (FIGS. 6A and 6B) and FIG. 7 (FIGS. 7A and 7B). FIGS. 6 and 7 are diagrams schematically showing the timing at which a conventional distance image capturing device measures a subject OB. Note that FIGS. 6 and 7 illustrate a configuration in which the pixel 321 includes four charge storage units CS. Even when the number of charge storage sections CS included in the pixel 321 is changed depending on the structure of the pixel 321, the above may be changed depending on the irradiation time To of the optical pulse PO and the length of the accumulation time Ta in the charge storage section CS. Similarly, if an attempt is made to perform multiple measurements while changing the time difference between the irradiation timing and the accumulation timing, it may be difficult to determine whether or not there is multipath depending on the position where the object OB is present. That is, regardless of the number of charge storage units CS included in the pixel 321, it may be difficult to determine whether or not there is multipath.

Note that in the following description, a subject OB that is located relatively close to the imaging position will be referred to as a "near object." Furthermore, a subject OB that is located relatively far from the imaging position is referred to as a "long distance object".

FIG. 6A shows an example in which a short-distance object is measured for the first time. FIG. 6B shows an example in which a close-range object is measured for the Kth time. K is any natural number greater than or equal to 1 and less than or equal to M.

The delay time Tdk in FIG. 6 is the delay time from irradiation of the optical pulse PO until the reflected light RL is received, and is shorter than the delay time Td in FIG. 5. That is, FIG. 6 shows an example of measuring a short-distance object that is located relatively close to the imaging position. Further, the irradiation delay time Dtmk in FIG. 6B indicates the time difference between the irradiation timing and the accumulation timing in the K-th measurement.

In the case of a short-distance object, the amount of reflected light RL is larger than in the case of measuring a long-distance object. Further, when the optical path difference between the single pass and the multipath is small, the single pass and the multipath are received by the pixel 321 almost simultaneously or with a slight time difference. Therefore, the difference between the tendency of the feature amount when the pixel 321 receives single-pass light and the tendency of the feature amount when the pixel 321 receives multi-pass light becomes small, and it may be difficult to determine whether the pixel 321 receives single-pass light or not. be.

FIG. 7A shows an example in which a long-distance object is measured for the first time. FIG. 7B shows an example in which a long-distance object is measured for the Kth time. The delay time Tde in FIG. 7 is the delay time from irradiation of the optical pulse PO until the reflected light RL is received, and is longer than the delay time Td in FIG. 5. That is, FIG. 7 shows an example of measuring a long-distance object that is located relatively far from the imaging position.

In the case of a long-distance object, since the delay time Tde is large, the timing at which the pixel 321 receives the reflected light RL in the K-th measurement deviates from the accumulation timing, and the charge corresponding to the reflected light RL is transferred to the charge storage section CS. may not be accumulated. In this case, it becomes difficult to calculate the feature amount for determining whether or not there is a single pass.

In order to solve the problem that it is difficult to determine whether or not multipath is occurring depending on the position where the object OB exists, the first embodiment has a method of performing multiple measurements with different combinations of irradiation time and accumulation time. I made it.

In the first embodiment, the distance image processing unit 4 performs a first measurement and a second measurement. In the first measurement, the first condition is the combination of the irradiation time and the accumulation time, the time difference between the reference irradiation timing and the accumulation timing is the first time difference, and the difference between the irradiation timing and the accumulation timing based on the first time difference is the first condition. These are multiple measurements with different time differences. In the second measurement, the combination of irradiation time and accumulation time is a second condition different from the first condition, the time difference between the reference irradiation timing and the accumulation timing is the second time difference, and the second time difference is used as the reference. These are a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other.
Note that in this embodiment, the first time difference is set to 0 (zero). That is, in this embodiment, the time difference between the reference irradiation timing and the accumulation timing is 0 (zero), and the reference initial (first) irradiation timing and accumulation timing are the same timing.
Furthermore, in this embodiment, the second time difference is set to the same value as the first time difference. That is, in the present embodiment, in the second measurement, the time difference between the reference irradiation timing and the accumulation timing is 0 (zero), and the reference initial (first) irradiation timing and accumulation timing are the same timing. .
However, it is not limited to this. The first time difference does not have to be 0 (zero) and may be set arbitrarily.

For example, the distance image processing unit 4 sets a reference combination of irradiation time and accumulation time, for example, the combination of irradiation time To and accumulation time Ta in FIG. 5, as the first condition.
When measuring a short-distance object, the distance image processing unit 4 sets a combination of an irradiation time and an accumulation time shorter than the first condition, for example, a combination of an irradiation time Tok and an accumulation time Tak in FIG. 8, which will be described later, as a second condition. do.
When measuring a long-distance object, the distance image processing unit 4 sets a combination of an irradiation time and an accumulation time longer than the first condition, for example, a combination of an irradiation time Toe and an accumulation time Tae in FIG. 9, which will be described later, as a second condition. do.

Further, the distance image processing unit 4 stores in advance a first lookup table LUT, which is a lookup table corresponding to the first condition, and a second lookup table LUT, which is a lookup table corresponding to the second condition. I'll keep it.

In the first measurement, the distance image processing unit 4 calculates a feature quantity based on the amount of charge accumulated in the charge accumulation unit CS for each measurement. After performing multiple measurements in the first measurement, the distance image processing unit 4 calculates a first SD index as the degree of similarity between the tendency of the feature amount calculated for each measurement and the tendency of the first lookup table LUT. .

In the second measurement, the distance image processing unit 4 calculates a feature quantity based on the amount of charge accumulated in the charge accumulation unit CS for each measurement. After performing a plurality of measurements in the second measurement, the distance image processing unit 4 calculates a second SD index as the degree of similarity between the tendency of the calculated feature amount and the tendency of the second lookup table LUT.

The distance image processing unit 4 calculates the distance to the object OB using the first SD index and the second SD index.

For example, the distance image processing unit 4 compares the first SD index with a threshold value, and when the first SD index indicates that the pixel 321 has received single-pass light, calculates the distance using equation (1).
On the other hand, the distance image processing unit 4 compares the first SD index and the threshold, and when the first SD index indicates that the pixel 321 has received multipath light, the distance image processing unit 4 compares the second SD index and the threshold. Here, the threshold value corresponding to the first SD index and the threshold value corresponding to the second SD index may be the same value or may be different values. When the second SD index indicates that the pixel 321 has received single-pass light, the distance image processing unit 4 calculates the distance using equation (1). When the second SD index indicates that the pixel 321 has received multipath light, the distance image processing unit 4 uses another means, for example, the least squares method as described below, without using equation (1). Calculate distance.

Here, a method for measuring a short-distance object and a long-distance object in the first embodiment will be described using FIGS. 8 (FIGS. 8A and 8B) and FIG. 9 (FIGS. 9A and 9B). 8 and 9 are diagrams schematically showing the timing at which the distance image imaging device 1 of the first embodiment measures the subject OB.

FIG. 8A shows an example in which a short distance object is measured for the first time in the second measurement. FIG. 8B shows an example in which a close-range object is measured for the Kth time in the second measurement.

The irradiation time Tok in FIG. 8 has a shorter time width than the irradiation time To. The accumulation time Tak has a shorter time width than the accumulation time Ta. The irradiation time Tok and the accumulation time Tak have approximately the same time width.

By setting the irradiation time and accumulation time short in the second measurement, the measurable distance range is narrowed, but this does not pose much of a problem since it is assumed that a short distance object is to be measured. On the other hand, by setting the irradiation time and accumulation time short, it is possible to improve measurement accuracy. By setting the irradiation time and accumulation time short, when performing multiple measurements, the amount of charge accumulated in the charge storage section CS is different from that of a single pass when compared with the case where the irradiation time and accumulation time are not shortened. It becomes easier to separate multipaths that receive light at different timings, and a difference in the tendency of the feature amount tends to appear when the pixel 321 receives light from a single path and when it receives light from a multipath.

Also, if the amount of reflected light RL is large in the first measurement and the amount of charge accumulated in the charge storage section CS exceeds the upper limit of the storage capacity of the charge storage section CS, saturation occurs where the amount of charge cannot be measured. However, by setting the irradiation time and accumulation time short in the second measurement, saturation can be made difficult.

FIG. 9A shows an example in which a long-distance object is measured for the first time in the second measurement. FIG. 9B shows an example in which a distant object is measured for the Kth time in the second measurement.

The irradiation time Toe in FIG. 9 has a longer time width than the irradiation time To. The accumulation time Tae has a longer time width than the accumulation time Ta. The irradiation time Toe and the accumulation time Tae have approximately the same time width.

By setting the irradiation time and accumulation time long in the second measurement, it is possible to widen the measurable distance range, and even in the K-th measurement in which the irradiation timing is delayed, the charge corresponding to the reflected light RL remains in the charge storage section. It is possible to store the information in the CS. Therefore, the feature amount can be calculated from each of the plurality of measurements in the second measurement, and it is possible to determine whether the pixel 321 receives single-pass light or multi-pass light.

Furthermore, by setting the irradiation time and accumulation time to be long in the second measurement, it is possible to increase the amount of charge accumulated in the charge accumulation section CS. When measuring a long-distance object, the amount of reflected light RL is smaller than that of a short-distance object. For this reason, the amount of charge stored in the charge storage section CS is small, making it susceptible to noise and causing measurement errors. In contrast, in the first embodiment, it is possible to increase the amount of charge stored in the charge storage section CS, and it is possible to reduce the influence of noise.

In this way, the distance image processing unit 4 performs the first measurement and the second measurement, extracts the feature amount based on the amount of charge accumulated in each of the first measurement and the second measurement, and determines the tendency of the feature amount. Based on this, the distance to the object OB is calculated. Thereby, it is possible to perform a second measurement with a different combination of irradiation time and accumulation time, and it is possible to decrease or increase the amount of charge accumulated in the charge accumulation section CS. Therefore, without changing the number of accumulations, it is possible to achieve auto-exposure (auto exposure) that avoids saturation and HDR (High Dynamic Range) that extends measurable distance, as well as to make the determination between single-pass and multi-pass. This makes it easier to measure and improve measurement accuracy.

Here, the method by which the distance image processing unit 4 calculates the feature amount, the contents of the lookup table LUT, and the method by which the SD index is calculated will be explained.

The distance image processing unit 4 calculates a complex variable CP shown in the following equation (2) based on the amount of charge accumulated in each of the charge accumulation units CS. The complex variable CP is an example of a "feature amount."

CP=(Q1-Q2)+j(Q2-Q3)...Equation (2)
However, j is an imaginary unit
Q1 is the amount of charge accumulated in charge storage section CS1
Q2 is the amount of charge accumulated in charge storage section CS2
Q3 is the amount of charge accumulated in charge storage section CS3

Further, the distance image processing unit 4 expresses the complex variable CP shown in equation (2) as a function GF of phase (2πfτ _A ) using equation (3). The phase (2πfτ _A ) here indicates the delay time τ _A with respect to the irradiation timing of the optical pulse PO as a phase delay with respect to the period (1/f=2To) of the optical pulse PO. Equation (3) assumes that only the reflected light from the object OB _A at the distance LA, ie, _a single path, is received. The function GF is an example of a "feature amount."

CP= _DA ×GF( _2πfτA )...Equation (3)
However, _DA is the intensity (constant) of the reflected light from _{the object OB A at} distance LA
τ _A is the time required for light to travel back and forth to object OB _A at distance _LA
τ _A =2L _A /c
c is the speed of light

In equation (3), if the value of the function GF corresponding to phases 0 (zero) to 2π can be determined, all single paths that can be received by the distance image capturing device 1 can be defined. Therefore, the distance image processing unit 4 defines a complex function CP(φ) of phase φ for the complex variable CP shown in Equation (3), and expresses it as shown in Equation (4). φ is the amount of phase change when the phase of the complex variable CP in equation (3) is set to 0 (zero).

CP (φ) = D _A × GF (2πfτ _A −φ) … Equation (4)
However, _DA is the intensity of reflected light from _{the object OB A at} distance LA
τ _A is the time required for light to travel back and forth to object OB _A at distance _LA
τ _A =2L _A /c
c is the speed of light
φ is the phase

Here, the behavior of the complex function CP(φ) (change in complex number due to change in phase) will be explained using FIGS. 10 and 11. 10 and 11 are diagrams showing examples of the complex function CP(φ) of the embodiment. The horizontal axis in FIG. 10 is the phase x, and the vertical axis is the value of the function GF(x). In FIG. 10, the solid line indicates the real part of the complex function CP(φ), and the dotted line indicates the value of the imaginary part of the complex function CP(φ). FIG. 11 shows an example of the function GF(x) in FIG. 10 shown on a complex plane. In FIG. 11, the horizontal axis represents the real axis, and the vertical axis represents the imaginary axis. The value obtained by multiplying the function GF(x) in FIGS. 10 and 11 by a constant (D _A ) corresponding to the signal strength becomes the complex function CP(φ).

The change in the complex function CP(φ) is determined according to the shape (time-series change) of the optical pulse PO. FIG. 10 shows, for example, a trajectory associated with a change in phase in the complex function CP(φ) when the optical pulse PO is a rectangular wave.

At phase x=0 (that is, delay time Td=0), all the charges corresponding to the reflected light are accumulated in the charge storage section CS1, and no charges corresponding to the reflected light are accumulated in the charge accumulation sections CS2 and CS3. . Therefore, the real part (Q1-Q2) of the function GF (x=0) becomes the maximum value max, and the imaginary part (Q2-Q3) becomes 0 (zero). max is a signal value corresponding to the amount of charge corresponding to total reflection light. At phase x = π/2 (that is, delay time Td = irradiation time To), all the charges corresponding to the reflected light are accumulated in the charge storage part CS2, and the charges corresponding to the reflected light are stored in the charge storage parts CS1 and CS3. No charge is accumulated. Therefore, the real part (Q1-Q2) of the function GF (x=π/2) becomes the minimum value (-max), and the imaginary part (Q2-Q3) becomes the maximum value max. At phase x = π (that is, delay time Td = irradiation time To x 2), all the charges corresponding to the reflected light are accumulated in the charge storage part CS3, and the charges corresponding to the reflected light are stored in the charge storage parts CS1 and CS2. No charge is accumulated. Therefore, the real part (Q1-Q2) of the function GF (x=π) becomes 0 (zero), and the imaginary part (Q2-Q3) becomes the minimum value (-max).

As shown in Fig. 11, in the complex plane, when the phase x=0, the function GF (x=0) has the coordinate (max, 0), and when the phase x=π/2, the function GF (x=π/2) has the coordinate (-max, max), phase x=π, and function GF (x=π) has coordinates (0, -max).

The distance image processing unit 4 determines whether the pixel 321 receives single-pass light or multi-pass light based on the tendency of the behavior of the function GF(x) (change in complex number due to change in phase) as shown in FIGS. 10 and 11. Determine whether a path has been received. The distance image processing unit 4 determines that the pixel 321 has received a single pass when the tendency of change in the complex function CP(φ) calculated by measurement matches the tendency of change in the function GF(x) in a single pass. do. On the other hand, if the tendency of the change in the complex function CP(φ) calculated by measurement does not match the tendency of change in the function GF(x) in a single pass, the distance image processing unit 4 determines that the pixel 321 has received multipath light. It is determined that

For example, the distance image processing unit 4 calculates the complex function CP(0) in the first measurement. The distance image processing unit 4 calculates the complex function CP(φ1) based on the second measurement. The phase φ1 is a phase (2πf×Dtm2) corresponding to the irradiation delay time Dtm2. f is the irradiation frequency (frequency) of the optical pulse PO. The distance image processing unit 4 calculates the complex function CP(φ2) based on the (M-1)th measurement. The phase φ2 is a phase (2πf×Dtm3) corresponding to the irradiation delay time Dtm3. The distance image processing unit 4 calculates the complex function CP(φ3) based on the M-th measurement. The phase φ3 is a phase (2πf×Dtm4) corresponding to the irradiation delay time Dtm4.

Here, a specific method for determining whether the distance image processing unit 4 receives single-pass light or multi-pass light will be described using FIGS. 12 to 15. Similar to FIG. 11, FIGS. 12 to 15 are shown on a complex plane in which the horizontal axis is the real axis and the vertical axis is the imaginary axis.

For example, as shown in FIG. 12, the distance image processing unit 4 plots the lookup table LUT and the measured points P1 to P3 on a complex plane. The lookup table LUT is information that associates the function GF(x) with its phase x when the pixel 321 receives single-pass light. The lookup table LUT is, for example, measured in advance and stored in a storage unit (not shown). The actual measurement points P1 to P3 are the values of the complex function CP(φ) calculated by measurement. As shown in FIG. 12, the distance image processing unit 4 determines that the pixel 321 has received a single pass in the measurement when the change trend in the lookup table LUT matches the change trend at the actual measurement points P1 to P3. judge.

As shown in FIG. 13, the distance image processing unit 4 plots the lookup table LUT and the actual measurement points P1# to P3# on a complex plane. The lookup table LUT is similar to the lookup table LUT in FIG. 12. Actual measurement points P1# to P3# are values of the complex function CP(φ) calculated by measurement in a measurement space different from that in FIG. As shown in FIG. 13, the distance image processing unit 4 determines whether the pixel 321 receives multipath light during measurement when the trend of change in the lookup table LUT and the trend of change at the actual measurement points P1# to P3# do not match. It is determined that the

Here, the distance image processing unit 4 determines whether the trend of the lookup table LUT matches the trend of the actual measurement points P1 to P3 (match determination). Here, a method in which the distance image processing unit 4 performs a match determination using scale adjustment and an SD index will be described.

(About scale adjustment)
Here, the distance image processing unit 4 performs scale adjustment as necessary. Scale adjustment is a process of adjusting the scale (absolute value of a complex number) of the lookup table LUT and the scale (absolute value of a complex number) of the actual measurement point P to be the same value. As shown in equation (4), the complex function CP(φ) is the value obtained by multiplying the function GF(x) by the constant _DA . The constant _DA is a constant value determined according to the amount of reflected light received. That is, the constant _DA is a value determined for each measurement depending on the irradiation time, irradiation intensity, and number of distributions per frame of the optical pulse PO. Therefore, the actual measurement point P has coordinates that are expanded (or reduced) by a constant _DA with the origin as a reference, compared to the corresponding point in the lookup table LUT.

In such a case, the distance image processing unit 4 performs scale adjustment to make it easier to determine whether the trend of change in the lookup table LUT matches the trend of change at the actual measurement points P1 to P3.

As shown in FIG. 14, the distance image processing unit 4 extracts a specific measured point P (for example, measured point P1) among the measured points P1 to P3. The distance image processing unit 4 scales the extracted actual measurement point so that the actual measurement point Ps after scale adjustment (for example, actual measurement point P1s), which is obtained by multiplying the extracted actual measurement point by a constant D with the origin as a reference, becomes a point on the lookup table LUT. Make adjustments. Then, the distance image processing unit 4 multiplies the remaining actual measurement points P (for example, actual measurement points P2, P3) by the same multiplication value (constant D) to the actual measurement point Ps after scale adjustment (for example, actual measurement points points P2s, P3s).

Note that even if the distance image processing unit 4 does not perform scale adjustment, if the specific actual measurement point P (for example, actual measurement point P1) becomes a point on the lookup table LUT, the scale adjustment is not necessary. In this case, the distance image processing unit 4 can omit scale adjustment.

(About matching determination using SD index)
Here, matching determination using the SD index will be explained using FIG. 15. FIG. 15 shows a complex plane, with the horizontal axis representing the real axis and the vertical axis representing the imaginary axis. FIG. 15 shows a lookup table LUT indicating the function GF(x) when the pixel 321 receives single-pass light, and points G(x0), G(x0+Δφ), and G(x0+2Δφ) on the lookup table LUT. It is shown. Further, in FIG. 15, complex functions CP(0), CP(1), and CP(2) are shown as actual measurement points.

The distance image processing unit 4 first creates (defines) a function GG(n) whose starting point matches the complex function CP(n) obtained by measurement. n is a natural number indicating a measurement number. For example, in the first measurement among multiple measurements (n = 0), in the second measurement among multiple measurements (n = 1), ..., in the NNth measurement (n = NN-1). ).

The function GG(x) is a function obtained by shifting the phase of the function GF(x) so as to match the starting point of the complex function CP(n) obtained by measurement. For example, as shown in equation (5), the distance image processing unit 4 sets the phase amount (x ₀ ) corresponding to the complex function CP (n=0) obtained in the first measurement as the initial phase, and sets the initial phase to A function GG(x) is created by shifting . In equation (5), x ₀ is the initial phase, n is the measurement number, and Δφ is the amount of phase shift for each measurement.

Next, the distance image processing unit 4 creates (defines) a complex function CP(n), a function GG(x), and a function SD(n) indicating the difference, as shown in equation (6). n in formula (6) indicates a measurement number.

Then, the distance image processing unit 4 uses the function SD(n) to calculate an SD index indicating the degree of similarity between the complex function CP(n) and the function GG(x), as shown in equation (7). do. n in equation (7) is the measurement number, and NN indicates the number of measurements. Note that the SD index defined here is an example. The SD index replaces the degree of dissociation on the complex plane of the complex function CP(n) and the function GG(n) with a single real number. , the functional form is of course adjustable. The SD index may be arbitrarily defined as long as it is an index indicating at least the degree of dissociation on the complex plane between the complex function CP(n) and the function GG(n).

The distance image processing unit 4 compares the calculated SD index with a predetermined threshold value. The distance image processing unit 4 determines that the pixel 321 has received single-pass light if the SD index does not exceed a predetermined threshold. On the other hand, when the SD index exceeds a predetermined threshold value, the distance image processing unit 4 determines that the pixel 321 has received multipath light.

Here, a method for the distance image processing unit 4 to calculate the measured distance according to the determination result will be described. The determination result here is the result of determining whether single-path light or multi-path light was received.

When receiving single-pass light, the distance image processing unit 4 calculates the measured distance using equation (8). In equation (8), n indicates the measurement number, _x0 indicates the initial phase, n indicates the measurement number, and Δφ indicates the phase shift amount for each measurement. Note that the internal distance in equation (8) may be arbitrarily set depending on the structure of the pixel 321 and the like. For example, if you do not take into account the setting position of the distance to the sensor, such as setting the light receiving surface of the sensor as the origin of the distance, or the internal distance, which is a correction distance due to the performance of the sensor's photoelectric conversion, etc., the internal distance is set to 0. .

Alternatively, when the distance image processing unit 4 determines that the pixel 321 has received a single path, the distance image processing unit 4 calculates the delay time Td based on equation (1), and calculates the measured distance using the calculated delay time Td. You may also do so.

When receiving multipath light, the distance image processing unit 4 calculates the complex function CP obtained by measurement as the sum of the reflected lights arriving from multiple (here, two) paths, as shown in equation (9). represent. _DA in equation (9) is the intensity of the reflected light from _{the object OB A located} at the distance LA. _xA is the phase required for the light to travel back and forth to _{the object OB A located} at the distance LA. n is the measurement number. Δφ indicates the amount of phase shift for each measurement. _DB is the intensity of reflected light from the object OB _B located at the distance _LB. x _B is the phase required for the light to travel back and forth to _{the object OB B located} at the distance LB.

The distance image processing unit 4 determines a combination of {phases x _A , x _B and intensities D _A , D _B } that minimizes the difference J shown in equation (10). The difference J corresponds to the sum of squares of the absolute values of the differences between the complex function CP(n) and the function G in equation (9). The distance image processing unit 4 determines the combination of {phases x _A , x _B and intensities D _A , D _B } by applying the least squares method, for example.

Note that in the above description, a case has been described as an example in which a look-up table LUT is used to determine whether single-pass light or multi-pass light is received. However, it is not limited to this. The distance image processing unit 4 may use a formula representing the function GF(x) instead of the lookup table LUT.

The formula representing the function GF(x) is, for example, a formula defined according to the phase range. In the example of Fig. 11, the function GF(x) is defined as a linear function with a slope (-1/2) and an intercept (max/2) in the range (0≦x≦2/π) for the phase x. . Further, in the range (2/π0<x≦π), the function GF(x) is defined as a linear function with a slope (-2) and an intercept (-max).

Further, the look-up table LUT may be created based on actual measurement results conducted in an environment where only a single pass is received, or may be created based on calculation results from a simulation or the like. There may be.

Moreover, although the case where the complex variable CP shown in Formula (2) is used has been described above as an example, the present invention is not limited to this. The complex variable CP may be a variable calculated using at least the amount of charge accumulated in the charge storage section CS that accumulates the amount of charge according to the reflected light RL. For example, the complex variable CP2=(Q2-Q3)+j(Q1-Q2) may be obtained by swapping the real and imaginary parts, or the complex variable CP3=(Q1-Q3) may be obtained by changing the combination of the real and imaginary parts. )+j(Q2-Q3), etc.

Further, in the above description, the timing of turning on the charge storage unit CS (accumulation timing) is fixed in FIG. 5, and the irradiation timing of irradiating the optical pulse PO is delayed. It never happens. In a plurality of measurements, it is only necessary that the accumulation timing and the irradiation timing change at least relatively. For example, it is of course possible to fix the irradiation timing and advance the accumulation timing. Furthermore, in the above description, the case where the function SD(n) is defined by equation (6) has been described as an example. However, it is not limited to this. The function SD(n) may be arbitrarily defined as long as it is a function indicating at least the difference between the complex function CP(n) and the function GG(n) on the complex plane.

Here, the flow of processing performed by the distance image imaging device 1 of the embodiment will be described using FIG. 16. FIG. 16 is a flowchart showing the flow of processing performed by the distance image capturing device 1 of the embodiment.

(Step S10)
The distance image processing unit 4 performs provisional measurements. The provisional measurement is a measurement that is performed separately from the first measurement and the second measurement, and is a measurement that calculates the distance using equation (1) regardless of whether it is a single pass or not. In the provisional measurement, each of the irradiation time, irradiation timing, accumulation time, and accumulation timing may be set arbitrarily, but is set to the same value as the first measurement in FIG. 5, for example.
(Step S11)
The distance image processing unit 4 determines the first condition and the second condition based on the distance calculated by the temporary measurement.
For example, when the distance image processing unit 4 determines that the subject OB is a short-distance object based on the distance calculated by provisional measurement, the distance image processing unit 4 sets the irradiation time and accumulation time under the second condition to be shorter than the first condition. I will make it happen. When the distance image processing unit 4 determines that the subject OB is a long-distance object based on the distance calculated by the temporary measurement, the distance image processing unit 4 sets the irradiation time and accumulation time under the second condition to be longer than the first condition. Make it.
Further, when the distance image processing unit 4 determines that the subject OB is a long-distance object based on the distance calculated by the temporary measurement, the charge corresponding to the reflected light RL is transferred to the charge storage unit CS in the M-th measurement. The irradiation time and accumulation time under the first condition may be determined so that the light is accumulated.
(Step S12)
The distance image processing unit 4 sets a first condition. The first condition is, for example, an irradiation time To and an accumulation time Ta that are set in advance as a reference. Alternatively, if the irradiation time and accumulation time under the first condition are determined in step S11, the first condition becomes the determined value.
(Step S13)
The distance image processing unit 4 performs first measurements and calculates feature amounts corresponding to each measurement. Each time the distance image processing unit 4 performs a measurement, the distance image processing unit 4 calculates a complex function CP(n) as a feature amount using the signal value corresponding to the amount of charge accumulated in the charge accumulation unit CS obtained by the measurement. do.
(Step S14)
The distance image processing unit 4 calculates a first SD index. The distance image processing unit 4 uses each of the feature quantities calculated in the first measurement and the first lookup table LUT to generate a first SD index as a degree of similarity between the warning of the feature quantity and the tendency of the first lookup table LUT. Calculate.
(Step S15)
The distance image processing unit 4 sets the second condition. The second condition is, for example, the irradiation time and accumulation time determined in step S11.
(Step S16)
The distance image processing unit 4 performs second measurements and calculates feature amounts corresponding to each measurement. Each time the distance image processing unit 4 performs a measurement, the distance image processing unit 4 calculates a complex function CP(n) as a feature amount using the signal value corresponding to the amount of charge accumulated in the charge accumulation unit CS obtained by the measurement. do.
(Step S17)
The distance image processing unit 4 calculates a second SD index. The distance image processing unit 4 uses each of the feature quantities calculated in the second measurement and the second lookup table LUT to determine a second SD index as a degree of similarity between the tendency of the feature quantity and the tendency of the second lookup table LUT. Calculate.
(Step S18)
The distance image processing unit 4 calculates the distance based on the first SD index and the second SD index. For example, the distance image processing unit 4 compares the first SD index with a threshold value, and when the first SD index indicates that the pixel 321 has received single-pass light, calculates the distance using equation (1). On the other hand, the distance image processing unit 4 compares the first SD index and the threshold, and when the first SD index indicates that the pixel 321 has received multipath light, the distance image processing unit 4 compares the second SD index and the threshold. Here, the threshold value corresponding to the first SD index and the threshold value corresponding to the second SD index may be the same value or may be different values. When the second SD index indicates that the pixel 321 has received single-pass light, the distance image processing unit 4 calculates the distance using equation (1). When the second SD index indicates that the pixel 321 has received multipath light, the distance image processing unit 4 calculates the distance by another means without using equation (1).

As explained above, the distance image imaging device 1 of the first embodiment performs the first measurement and the second measurement, and extracts the feature amount based on the amount of charge accumulated in each of the first measurement and the second measurement. do. In the first measurement, the distance image processing unit 4 determines that in the first measurement, the combination of the irradiation time and the accumulation time is the first condition, the time difference between the reference irradiation timing and the accumulation timing is the first time difference, and the first time difference is used as the reference. A plurality of measurements with different time differences between irradiation timing and accumulation timing are performed. In the second measurement, the distance image processing unit 4 determines that in the second measurement, the combination of the irradiation time and the accumulation time is the second condition, the time difference between the reference irradiation timing and the accumulation timing is the second time difference, and the second time difference is used as the reference. A plurality of measurements with different time differences between irradiation timing and accumulation timing are performed.
In the second measurement, the distance image processing unit 4 performs a measurement in which either the second condition or the second time difference is different from the first measurement. For example, in the second measurement, the distance image processing unit 4 performs a measurement in which the second condition is different from the first measurement and the second time difference is the same as the first measurement.
The distance image processing unit 4 calculates the distance to the object OB based on the tendency of the extracted feature amount. As a result, the distance image capturing device 1 of the first embodiment can perform measurements multiple times under each of the first condition and the second condition in which the combination of the irradiation time and accumulation time is changed. It becomes possible to explore multipath trends under conditions with different time combinations. Therefore, even if it is difficult to determine whether the reflected light RL has multipath characteristics in the first measurement and the distance cannot be calculated accurately, the determination can be made by changing the combination of irradiation time and accumulation time in the second measurement. This makes it possible to calculate distances with high accuracy. Therefore, it is possible to take measures according to the tendency of multipath.

Further, the distance image capturing device 1 of the first embodiment performs a multi-pass determination to determine whether the reflected light RL is received by the pixel 321 in a single pass or the reflected light RL is received by the pixel 321 in a multi-pass. conduct. The distance image processing unit 4 calculates the distance to the object OB according to the result of the multipath determination. Thereby, in the distance image imaging device 1 of the first embodiment, it becomes possible to accurately calculate the distance according to the result of the multipath determination.

Further, in the distance image imaging device 1 of the first embodiment, the distance image processing unit 4 refers to the lookup table LUT for each combination of irradiation time and accumulation time. The lookup table LUT associates the time difference between the irradiation timing and the accumulation timing with the feature amount when the reflected light RL is received by the pixel 321 in a single pass. The distance image processing unit 4 performs multipath determination based on the degree of similarity between the tendency of the lookup table LUT and the tendency of the feature amount. Thereby, in the distance image imaging device 1 of the first embodiment, it becomes possible to easily perform multipath determination using the lookup table LUT.

Further, in the distance image imaging device 1 of the first embodiment, a plurality of lookup tables LUT are created for each combination of the shape of the optical pulse PO and the irradiation time and accumulation time. The distance image processing unit 4 performs multipath determination using lookup tables corresponding to the measurement conditions of the first measurement and the second measurement among the plurality of lookup tables. Thereby, in the distance image imaging device 1 of the first embodiment, it is possible to select an appropriate lookup table LUT according to the measurement conditions, and it is possible to perform accurate determination.

Further, in the distance image capturing device 1 of the first embodiment, the feature amount is a value calculated using at least the amount of charge corresponding to the reflected light RL among the amount of charge accumulated in each of the charge storage sections CS. be. Thereby, in the distance image imaging device 1 of the first embodiment, it becomes possible to perform multipath determination based on the situation in which the reflected light RL is received.

Furthermore, in the first embodiment described above, the case where the pixel 321 includes three charge storage sections CS was explained as an example. However, it is not limited to this. The present invention can also be applied to a case where the pixel 321 includes four charge storage sections CS. In this case, the feature amount is a complex number whose variable is the amount of charge stored in each of the charge storage units CS1 to CS4. For example, the feature amount is a value expressed by a complex number whose real part is the difference between the charge amounts Q1 and Q3, and whose imaginary part is the difference between the charge amounts Q2 and Q4. Specifically, the distance image processing unit 4 calculates a complex variable CP shown in the following equation (11) based on the amount of charge accumulated in each of the charge storage units CS.

CP=(Q1-Q3)+j(Q2-Q4)...Equation (11)
However, j is an imaginary unit
Q1 is the amount of charge accumulated in charge storage section CS1
Q2 is the amount of charge accumulated in charge storage section CS2
Q3 is the amount of charge accumulated in charge storage section CS3
Q4 is the amount of charge accumulated in charge storage section CS4

Thereby, in the distance image imaging device 1 of the first embodiment, the feature amount can be calculated using the amount of charge from which the external light component is removed, that is, the amount of charge corresponding to the reflected light RL. Therefore, noise including external light components can be removed, and multipath determination can be performed with high accuracy.

Further, in the distance image imaging device 1 of the first embodiment, the distance image processing unit 4 delays the irradiation timing with respect to the accumulation timing in the first measurement and the second measurement, thereby reducing the time difference between the irradiation timing with respect to the accumulation timing. Perform multiple measurements that are different from each other. As a result, the distance image imaging device 1 of the first embodiment can easily perform multiple measurements by changing only the timing of irradiating the optical pulse PO without changing the timing of driving the pixel 321. Can be done.

In addition, in the distance image imaging device 1 of the first embodiment, the distance image processing unit 4 performs temporary measurement to calculate a provisional distance to the subject without determining whether it is single pass or multipass, and calculates the distance in the temporary measurement. At least one of the first condition and the second condition is determined according to the determined distance. Thereby, in the distance image imaging device 1 of the first embodiment, at least one of the first condition and the second condition can be determined according to the provisional distance measured in the temporary measurement, and the approximate distance to the subject OB can be determined. The first condition and the second condition can be set according to the distance, and it becomes possible to perform the first measurement or the second measurement that allows accurate multipath determination.

Further, in the distance image capturing device 1 of the first embodiment, when the distance image processing unit 4 determines that the subject OB is a short-distance object that exists relatively nearby, according to the distance calculated in the temporary measurement. , the second condition is determined such that the combination of irradiation time and accumulation time under the second condition is shorter than the first condition. As a result, in the distance image capturing device 1 of the first embodiment, when the subject OB is a short-distance object, auto-exposure that suppresses saturation of the charge storage unit CS is realized, and multipath determination is facilitated. be able to. On the other hand, when determining that the subject OB is a long-distance object that exists relatively far away, the distance image processing unit 4 sets the irradiation time and accumulation time under the second condition to be longer than the first condition. 2. Determine conditions. As a result, when the subject OB is a long-distance object, it is possible to expand the measurable range, realize HDR, and make it easier to perform multipath determination.

Furthermore, in the first embodiment described above, the case where the distance to the object OB is calculated using equation (1) when it is determined that the pixel 321 has received single-pass light has been described as an example. However, it is not limited to this. Equation (1) assumes that the irradiation timing and the accumulation timing are the same, that is, the irradiation delay time is 0 (zero). Therefore, when calculating the distance using the second and subsequent measurement results among a plurality of measurements, equation (1) cannot be applied as is. When calculating the distance using the second and subsequent measurement results, the distance image processing unit 4 performs correction according to the irradiation delay time.

That is, in the distance image imaging device 1 of the first embodiment, the distance image processing unit 4 corrects the distance based on each of the plurality of measurements according to the distance based on the time difference between the plurality of measurements, and The latter distance is the distance to the object OB. Thereby, even if the distance is calculated using the second and subsequent measurement results, it is possible to calculate the correct distance.

Further, in the distance image imaging device 1 of the first embodiment, the distance image processing unit 4 calculates the SD index. The SD index is an index value that indicates the degree of similarity between the tendency of the lookup table LUT and the tendency of each feature amount of a plurality of measurements. The SD index is expressed by equation (7). In other words, the SD index is the difference between the complex function CP(n) (first feature quantity) calculated from each of a plurality of measurements and the corresponding function GG(n) (second feature quantity) in the lookup table LUT. is the sum of the normalized difference values of a plurality of measurements with respect to the normalized difference value normalized by the absolute value of the second feature amount. The distance image processing unit 4 determines that the reflected light RL has been received by the pixel 321 in a single pass when the SD index does not exceed the threshold value. On the other hand, when the SD index exceeds the threshold value, the distance image processing unit 4 determines that the reflected light RL has been received by the pixel 321 through multipath. Thereby, in the distance image imaging device 1 of the first embodiment, multipath determination can be performed by a simple method of comparing the SD index and the threshold value.

In addition, in the distance image imaging device 1 of the first embodiment, when the distance image processing unit 4 determines that the reflected light RL is received by the pixel 321 in a multipath manner, the distance image processing unit 4 corresponds to each of the light paths included in the multipath. Calculate the distance by using the least squares method. As a result, the distance image capturing device 1 according to the first embodiment can determine the most probable route for each multipath, and can calculate the distance corresponding to each multipath. .

Furthermore, in the first embodiment described above, it is assumed that the intensity of the optical pulse PO is constant. However, it is not limited to this. The distance image processing unit 4 may control the intensity of light that irradiates the light pulse (hereinafter referred to as light intensity). For example, when measuring a short-distance object, the distance image processing unit 4 shortens the irradiation time and accumulation time and weakens the light intensity in the second measurement. Thereby, the distance image processing unit 4 can suppress saturation and power consumption. Alternatively, when measuring a distant object, the distance image processing unit 4 lengthens the irradiation time and accumulation time and increases the light intensity in the second measurement. Thereby, the distance image processing unit 4 can reduce shot noise and improve multipath separation accuracy.

Further, the distance image imaging device 1 of the first embodiment includes a drain gate transistor GD (charge discharge section). The distance image processing unit 4 controls the charge generated by the photoelectric conversion element PD so that it is discharged by the drain gate transistor GD at a timing different from the accumulation timing in one frame period. Thereby, in the distance image imaging device 1 of the first embodiment, it is avoided that charges corresponding to the external light component continue to be accumulated in a time period in which the reflected light RL of the optical pulse PO is not expected to be received. be able to.

In this manner, in the SP method, the drain gate transistor GD is turned on to discharge charges during a time period in which the reflected light RL is not expected to be received in the unit storage time UT. This prevents charges corresponding to the external light component from continuing to accumulate in a time period in which the reflected light RL of the optical pulse PO is not expected to be received.

On the other hand, in the so-called continuous wave method (hereinafter referred to as CW method) in which the optical pulse PO is continuously irradiated, it is not possible to discharge the charge every time the charge is accumulated in the charge storage part CS in the unit accumulation time UT. do not have. This is because in the CW system, since the reflected light RL is always received, there is no time period in which the reflected light RL is not expected to be received. In the CW method, during a time period in which the process of repeating the unit storage time UT multiple times in one frame is executed, a charge discharge unit such as a reset gate transistor connected to the photoelectric conversion element PD is controlled to be in an off state, Do not discharge charge. Then, when the read time RD arrives in one frame, the amount of charge accumulated in each of the charge storage sections CS is read out, and then a charge discharge section such as a reset gate transistor is controlled to be in an on state, and the charge is discharged. . Further, in the above description, the mechanism in which the charge discharge section is connected to the photoelectric conversion element PD was explained as an example, but the present invention is not limited to this. A mechanism may also be used in which the photoelectric conversion element PD does not have a charge discharge part and a reset gate transistor is used in which the charge discharge part is connected to the floating diffusion FD.

In this embodiment, since the SP method is adopted, the pixel 321 of the distance image capturing device 1 includes a drain gate transistor GD. As a result, it is possible to reduce errors compared to the case where charges are continuously accumulated in one frame using the CW method, so it is possible to increase the S/N ratio (ratio of error to signal component) of the amount of charge. . Therefore, even if the number of integrations is increased, it is difficult to accumulate errors, so the accuracy of the amount of charge stored in the charge storage section CS can be maintained, and the feature amount can be calculated with high accuracy.

Furthermore, in the first embodiment described above, it is assumed that the irradiation time To and the accumulation time Ta have the same time width, and that the equivalent time width includes a case where the irradiation time To is longer than the accumulation time Ta by a predetermined time. explained. The effect when the irradiation time To is longer than the accumulation time Ta by a predetermined time will be supplemented.

Here, as an example, consider a case where the timing at which the reflected light RL is received (hereinafter referred to as light reception timing) coincides with the timing at which the charge storage section CS2 turns on (hereinafter referred to as second accumulation timing).

In this case, if the shape of the optical pulse PO is an ideal rectangular shape, charges corresponding to the reflected light RL are accumulated only in the charge storage section CS2, and charges corresponding to the reflected light RL are accumulated in the charge accumulation sections CS1 and CS3. No corresponding charge is accumulated. However, the actual shape of the optical pulse PO has a rounded waveform and does not have an ideal rectangular shape. In this case, the irradiation time of the optical pulse PO may appear to be shorter than the accumulation time. When the irradiation time is shorter than the accumulation time, if the light reception timing and the second accumulation timing match, charges corresponding to the reflected light RL are accumulated only in the charge accumulation section CS2. However, even if the distance to the subject OB changes after that and the light reception timing is delayed from the second accumulation timing, the irradiation time is shorter than the accumulation time, so the reflected light RL only hits the charge accumulation part CS2. This results in a continued state in which charges accumulate. In such a case, the accuracy of calculating the distance may deteriorate.

On the other hand, if the irradiation time To is set longer than the accumulation time Ta, even if the light reception timing and the second accumulation timing match, the reflected light will be reflected not only in the charge storage part CS2 but also in the charge storage part CS3. Charge corresponding to RL is accumulated. Therefore, when the light reception timing is delayed from the second accumulation timing, an amount of charge corresponding to the delay can be accumulated in the charge accumulation section CS3, and deterioration of the accuracy of distance calculation can be suppressed.

FIG. 17 is a diagram showing an example of the lookup table LUT# when the irradiation time To is set longer than the accumulation time Ta. As shown in FIG. 17, when the irradiation time To is set longer than the accumulation time Ta, the lookup table changes the shape from a sharply changing shape at the point of phase x = π/2 to a continuously changing roundness. It has a tinged shape. If the phase changes sharply at the point where x=π/2, the measurement accuracy tends to decrease near the phase x=π/2. On the other hand, by setting the irradiation time To to be longer than the accumulation time Ta, since the phase changes continuously at the point where x=π/2, it is possible to suppress a decrease in measurement accuracy.

Next, a second embodiment will be described. In the second embodiment, in the second measurement, the second condition (combination of irradiation time and accumulation time) is the same as the first measurement, while the second time difference (time difference between reference irradiation timing and accumulation timing) is set. under different conditions from the first measurement,

Here, a method for measuring a long-distance object in the second embodiment will be described using FIG. 18 (FIGS. 18A and 18B). FIG. 18 is a diagram schematically showing the timing at which the distance image imaging device 1 of the second embodiment measures the object OB.

FIG. 18A shows an example in which a long-distance object is measured for the first time in the second measurement. FIG. 18B shows an example in which a distant object is measured for the Kth time in the second measurement.

The irradiation time To in FIG. 18 has the same time width as the irradiation time To in FIG. The accumulation time Ta has the same time width as the accumulation time Ta in FIG. The irradiation time To and the accumulation time Ta have approximately the same time width.

As shown in FIG. 18, in the first (initial) measurement of the second measurement, the accumulation timing is delayed by a time Tds with respect to the irradiation timing. That is, the distance image processing unit 4 sets the time Tds as the second time difference. Using time Tds, which is the second time difference, as a reference, in subsequent measurements, a plurality of measurements are performed while setting different time differences between the irradiation timing and the accumulation timing with time Tds as the reference.

In this way, by setting the time difference between the reference first irradiation timing and accumulation timing as the time Tds, in the Kth measurement of the second measurement, the irradiation timing is changed to the irradiation delay time Dtmk with respect to the first measurement. Even in the case of delay, the charges corresponding to the reflected light RL can be accumulated in the charge accumulation section CS.

Note that when performing measurements under conditions in which the accumulation timing is delayed by a time Tds from the first measurement, charges corresponding to the reflected light RL reflected from a nearby object must be accumulated in the charge accumulation section CS. This makes it difficult to measure the distance to nearby objects.

As a countermeasure for this, in this embodiment, a temporary measurement is performed separately from the first measurement and the second measurement. The provisional measurement is a measurement that is performed separately from the first measurement and the second measurement, and is a measurement that calculates the distance using equation (1) regardless of whether it is a single pass or not. In the provisional measurement, each of the irradiation time, irradiation timing, accumulation time, and accumulation timing may be set arbitrarily, but is set to the same value as the first measurement in FIG. 5, for example.

For example, when the distance image processing unit 4 determines that the subject OB is a short-distance object based on the distance calculated by provisional measurement, in the second measurement, the time difference between the irradiation timing and the accumulation timing is 0 (zero) as a reference. Perform multiple measurements.

On the other hand, when the distance image processing unit 4 determines that the subject OB is a long-distance object based on the distance calculated by the temporary measurement, in the second measurement, the distance image processing unit 4 determines that the time difference between the irradiation timing and the accumulation timing is the time Tds. Perform multiple measurements based on .

In the second measurement, when a plurality of measurements are performed based on the relationship in which the time difference between the irradiation timing and the accumulation timing is the time Tds, the distance image processing unit 4 calculates the distance calculated in the second measurement by using the second time difference. The corrected distance is corrected according to the distance based on , and the corrected distance is taken as the distance to the object OB.

As explained above, the distance image imaging device 1 of the second embodiment performs the first measurement and the second measurement. In the second measurement, the distance image processing unit 4 performs a measurement in which the second condition is the same as the first measurement, and the second time difference is different from the first measurement. As a result, in the distance image imaging device 1 of the second embodiment, multiple measurements are performed using the first time difference as a reference in the first measurement, and a second time difference different from the first time difference is used as a reference in the second measurement. The measurement can be performed a plurality of times, and the reference time difference (time difference between the irradiation timing and the accumulation timing) can be made different in each of the first measurement and the second measurement.

Therefore, even if the charge corresponding to the reflected light RL is not accumulated in the charge storage part CS in the K-th measurement in the first measurement, such as when the subject OB is a long-distance object, in the second measurement , K-th measurement also allows charges corresponding to the reflected light RL to be accumulated in the charge accumulation section CS. Therefore, it becomes possible to calculate the distance with high accuracy.

Further, in the distance image imaging device 1 of the second embodiment, the distance image processing unit 4 performs provisional measurement to calculate a provisional distance to the subject without determining whether it is single pass or multipass. The distance image processing unit 4 determines the second time difference according to the distance calculated in the temporary measurement. As a result, in the distance image capturing device 1 of the second embodiment, the second time difference can be determined according to the provisional distance measured in the temporary measurement, and the second time difference can be determined according to the approximate distance to the subject OB. Adjustment can be made so that the charge corresponding to the reflected light RL is accumulated in the charge storage section CS in all of the plurality of measurements, and it becomes possible to perform measurements with high accuracy.

Further, in the second measurement, in the distance image imaging device 1 of the second embodiment, when a plurality of measurements are performed based on the relationship in which the time difference between the irradiation timing and the accumulation timing is the time Tds, the distance image processing unit 4 , the distance calculated in the second measurement is corrected according to the distance based on the second time difference (time Tds), and the corrected distance is set as the distance to the object OB. Thereby, in the second measurement, even if the time difference between the irradiation timing and the accumulation timing is not 0 (zero), it is possible to calculate the correct distance.
Note that when performing temporary measurements, it is not necessary to perform temporary measurements every time. Specifically, it is not necessary to repeat each measurement in the order of provisional measurement, first measurement, and second measurement. For example, if the object OB exists within a certain range in the measurement area, the distance can be calculated by omitting the temporary measurement and performing a set of the first and second measurements, or only the second measurement. You can. On the other hand, if certain conditions are met, such as when a certain period of time has passed since the previous measurement, or when the subject OB has moved outside the measurement area, for example, provisional measurement, a set of provisional measurement and first measurement, or Either one of the first measurements may be performed and then the second measurement may be performed.

All or part of the distance image imaging device 1 and the distance image processing section 4 in the embodiments described above may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, a "computer-readable recording medium" refers to a storage medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include a device that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Further, the above-mentioned program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. It may also be realized using a programmable logic device such as an FPGA.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

1...Distance image imaging device 2...Light source section 3...Light receiving section 32...Distance image sensor 321...Pixel 42...Distance calculation section CS...Charge storage section PO...Light pulse RL...Reflected light Dt...Dot light L...Line light

Claims (19)

a light source unit that irradiates light pulses to the subject;
A pixel including a photoelectric conversion element that generates a charge according to incident light and a plurality of charge storage sections that accumulate charge, and each of the charge storage sections at an accumulation timing synchronized with the irradiation timing of irradiating the light pulse. a pixel drive circuit that distributes and accumulates charge,
a distance image processing unit that calculates a distance to the subject based on the amount of charge accumulated in each of the charge accumulation units;
Equipped with
The distance image processing section
A first condition is a combination of an irradiation time for irradiating the light pulse and an accumulation time for distributing and accumulating charges in each of the charge storage sections, and a time difference between the reference irradiation timing and the accumulation timing is the first time difference. and performing a first measurement consisting of a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other based on the first time difference,
The combination of the irradiation time and the accumulation time is a second condition, the time difference between the reference irradiation timing and the accumulation timing is a second time difference, and the irradiation timing and the accumulation timing are set based on the second time difference. Perform a second measurement consisting of a plurality of measurements with different time differences from each other,
In the second measurement, either the second condition or the second time difference is a measurement that is different from the first measurement,
extracting a feature amount based on the amount of charge accumulated in each of the first measurement and the second measurement, and calculating a distance to the subject based on a tendency of the feature amount;
Distance image imaging device. The distance image processing section
In the second measurement, the second time difference is the same as the first measurement, and the second condition is different from the first measurement.
The distance image imaging device according to claim 1. The distance image processing section
In the second measurement, the second time difference is different from the first measurement, and the second condition is the same as the first measurement.
The distance image imaging device according to claim 1. The distance image processing unit performs a multi-pass determination to determine whether the reflected light of the optical pulse is received by the pixel in a single pass or the reflected light of the optical pulse is received by the pixel in multiple passes. , calculating a distance to the subject according to the result of the multipath determination;
The distance image imaging device according to claim 1. The distance image processing unit calculates, for each combination of the irradiation time and the accumulation time, the time difference between the irradiation timing and the accumulation timing when the reflected light is received by the pixel in a single pass, and the feature amount. referring to the associated lookup table and performing the multipath determination based on the degree of similarity between the tendency of the lookup table and the tendency of the feature amount;
The distance image imaging device according to claim 4. A plurality of lookup tables are created for each combination of the shape of the light pulse and the irradiation time and the accumulation time,
The distance image processing unit performs the multipath determination using the lookup table corresponding to each of the measurement conditions of the first measurement and the second measurement among the plurality of lookup tables.
The distance image imaging device according to claim 5. The feature quantity is a value calculated using at least an amount of charge corresponding to the reflected light of the optical pulse, out of the amount of charge accumulated in each of the charge storage sections,
The distance image imaging device according to claim 1. The pixel is provided with a first charge storage section, a second charge storage section, a third charge storage section, and a fourth charge storage section,
The distance image processing unit is configured to charge at least one of the first charge storage unit, the second charge storage unit, the third charge storage unit, or the fourth charge storage unit, the charge corresponding to the reflected light of the optical pulse. Accumulating charges in the order of the first charge accumulation section, the second charge accumulation section, the third charge accumulation section, and the fourth charge accumulation section at the timing when the charge accumulation section is accumulated;
The feature quantity is a complex number whose variable is the amount of charge accumulated in each of the first charge accumulation section, the second charge accumulation section, the third charge accumulation section, and the fourth charge accumulation section,
The distance image imaging device according to claim 1. The feature quantity has a first variable, which is the difference between the first charge amount accumulated in the first charge accumulation section and the third charge amount accumulated in the third charge accumulation section, as a real part, and the second a value expressed as a complex number whose imaginary part is a second variable that is the difference between the second amount of charge accumulated in the charge storage section and the fourth amount of charge accumulated in the fourth charge storage section;
The distance image imaging device according to claim 8. The distance image processing unit performs a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different by delaying the irradiation timing with respect to the accumulation timing in the first measurement and the second measurement. ,
The distance image imaging device according to claim 1. The distance image processing unit performs provisional measurement to calculate the distance to the object without determining whether it is a single pass or multipass, and sets the first condition and the second condition according to the distance calculated in the provisional measurement. determining at least one of the conditions;
The distance image imaging device according to claim 2. When determining that the subject is relatively close according to the distance calculated in the provisional measurement, the distance image processing unit determines that the combination of the irradiation time and the accumulation time in the second condition is equal to the first If the second condition is determined to be shorter than the first condition and it is determined that the subject is relatively far away, the combination of the irradiation time and the accumulation time in the second condition is shorter than the first condition. determining the second condition so that the second condition is also a long time;
The distance image imaging device according to claim 11. The distance image processing unit performs a provisional measurement to calculate the distance to the object without determining whether the shot is a single pass or a multipass, and determines the second time difference according to the distance calculated in the provisional measurement.
The distance image imaging device according to claim 3. The distance image processing unit corrects the distance calculated in the second measurement according to the distance based on the second time difference, and sets the corrected distance as the distance to the subject.
The distance image imaging device according to claim 13. The distance image processing unit calculates an index value indicating the degree of similarity between the tendency of the lookup table and the tendency of the feature amount of each of the plurality of measurements,
The index value includes a first feature quantity that is the feature quantity calculated from each of the plurality of measurements, and a second feature quantity that is the feature quantity that corresponds to each of the plurality of measurements in the lookup table. is an additive value obtained by adding the normalized difference values of each of the plurality of measurements to the normalized difference value obtained by normalizing the difference of by the absolute value of the second feature amount,
The distance image processing unit determines that the reflected light is received by the pixel in a single pass when the index value does not exceed the threshold, and determines that the reflected light is received by the pixel in a single pass when the index value exceeds the threshold. determining that light has been received by the pixel,
The distance image imaging device according to claim 5. When the distance image processing unit determines that the reflected light is received by the pixel in a multipath, the distance image processing unit calculates a distance corresponding to each of the light paths included in the multipath by using a least squares method.
The distance image imaging device according to claim 4. The distance image processing unit controls the intensity of irradiating the light pulse in the first measurement and the second measurement according to the distance calculated in the temporary measurement.
The distance image imaging device according to claim 11. Further comprising a charge discharge part that discharges the charge generated by the photoelectric conversion element,
The distance image processing unit controls the charge generated by the photoelectric conversion element to be discharged by the charge discharge unit at a timing different from the accumulation timing.
The distance image imaging device according to claim 1. A light source unit that irradiates a light pulse to a subject, a pixel that includes a photoelectric conversion element that generates a charge according to the incident light, and a plurality of charge storage units that accumulate the charge, and a pixel that is synchronized with the irradiation timing that irradiates the light pulse. a pixel drive circuit that allocates and accumulates charges in each of the charge storage sections at the specified accumulation timing; and a distance image processing section that calculates a distance to the subject based on the amount of charge accumulated in each of the charge storage sections. A distance image capturing method performed by a distance image capturing device comprising:
The distance image processing section
A first condition is a combination of an irradiation time for irradiating the light pulse and an accumulation time for distributing and accumulating charges in each of the charge storage sections, and a time difference between the reference irradiation timing and the accumulation timing is the first time difference. and performing a first measurement consisting of a plurality of measurements in which the time difference between the irradiation timing and the accumulation timing is different from each other based on the first time difference,
The combination of the irradiation time and the accumulation time is a second condition, the time difference between the reference irradiation timing and the accumulation timing is a second time difference, and the irradiation timing and the accumulation timing are set based on the second time difference. Perform a second measurement consisting of a plurality of measurements with different time differences from each other,
In the second measurement, either the second condition or the second time difference is a measurement that is different from the first measurement,
extracting a feature amount based on the amount of charge accumulated in each of the first measurement and the second measurement, and calculating a distance to the subject based on a tendency of the feature amount;
Distance image imaging method.

Images