WO2020189339A1

WO2020189339A1 - Distance measuring device and distance measuring method

Info

Publication number: WO2020189339A1
Application number: PCT/JP2020/009677
Authority: WO
Inventors: 前田　俊治
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2019-03-20
Filing date: 2020-03-06
Publication date: 2020-09-24
Also published as: JP2020153799A

Abstract

The present disclosure relates to a distance measuring device and a distance measuring method which enable highly accurate distance measurement even in an environment in which a plurality of distance measuring devices are present. An interval at which a light source of a device itself emits light is increased, an exposure period is set in synchronism with the light emission timing, a pixel signal is detected by means of reflected light from the light source of the device itself and from other light sources, and integration is performed a plurality of times. Further, the exposure period is set to a period in which the light source of the device itself is extinguished, the pixel signal is detected by means of the reflected light from the other light sources, and integration is performed a plurality of times. A distance measurement calculation is then performed by subtracting the integrated value of the pixel signal for which the exposure period was set in synchronism with the timing at which the light source was extinguished, from the integrated value of the pixel signal for which the exposure period was set in synchronism with the light emission timing. This disclosure is applicable to ToF sensors.

Description

Distance measuring device and distance measuring method

The present disclosure relates to a distance measuring device and a distance measuring method, and more particularly to a distance measuring device and a distance measuring method capable of measuring a distance with high accuracy even in an environment where a plurality of distance measuring devices exist.

Conventionally, active type ranging devices such as the ToF (Time of Flight) method have been known. In such a ranging device, a laser beam that repeatedly emits light with a predetermined pulse width is irradiated, and the irradiated laser beam hits an object and receives the reflected light, so that the reciprocating time of the laser beam (laser related to the reciprocation). Distance measurement is performed based on the phase difference of light).

By the way, when performing distance measurement to the same object using a plurality of distance measuring devices, each of the plurality of distance measuring devices is affected by the reflected light of the laser beam emitted from the other distance measuring devices. Therefore, there is a risk that proper distance measurement cannot be achieved.

Therefore, when a state that is affected by other ranging devices is detected, a technique has been proposed in which the ranging results during that period are not adopted (see Patent Document 1).

Further, a technique has been proposed in which the interval between the emission pulses of the laser beam is sufficiently long so that it is not affected by the reflected light of another ranging device (see Patent Document 2).

Japanese Unexamined Patent Publication No. 2011-007805 Japanese Unexamined Patent Publication No. 2017-190978

However, in the technique described in Patent Document 1, appropriate distance measurement cannot be realized while being affected by the laser beam from another distance measuring device.

Further, in the technique described in Patent Document 2, since the interval between the emission pulses is long, the distance measurement takes time and the distance measurement at a high frame rate becomes impossible. Therefore, for example, an object moving at high speed When performing distance measurement, there is a risk that a time lag will occur in the distance measurement results and it will not be possible to achieve appropriate distance measurement in real time.

This disclosure has been made in view of such a situation, and in particular, it enables highly accurate distance measurement even in an environment where a plurality of distance measurement sensors exist.

The ranging device on one side of the present disclosure includes a light receiving portion that receives reflected light generated by reflection of irradiation light by an object in the detection region, and the object due to a phase difference between the irradiation light and the reflected light. The calculation unit that calculates the distance to the object, the required irradiation light that is the irradiation light used to calculate the distance to the object in the light receiving unit, and the unnecessary irradiation that is not used to calculate the distance to the object. It is a distance measuring device including a light receiving unit and a control unit that controls the calculation unit so as to suppress the influence of collision with light.

The distance measuring method on one aspect of the present disclosure corresponds to the distance measuring device.

In one aspect of the present disclosure, the required irradiation light, which is the irradiation light generated by the reflection of the irradiation light by the object in the detection region and used for calculating the distance to the object, and the object. It is controlled so as to suppress the influence of collision with unnecessary irradiation light, which is irradiation light that is not used for calculating the distance to.

It is a figure explaining the example in the case of using a plurality of depth sensors. It is a figure explaining the principle of distance measurement. It is a figure explaining the method of avoiding the influence when a plurality of ranging devices are used. It is a figure explaining the distance measurement calculation method of 4phase. It is a figure explaining the distance measurement calculation method of this disclosure. It is a figure explaining the structural example of the depth sensor of this disclosure. It is a figure explaining the configuration example of the TOF sensor of FIG. It is a flowchart explaining the normal mode processing. It is a flowchart explaining the correction mode processing. It is a figure explaining the determination method of the collision by the irradiation light from a plurality of light sources. It is a figure explaining the avoidance method of a collision. It is a flowchart explaining the collision avoidance mode processing. It is a flowchart explaining the collision detection process of FIG. It is a figure explaining the principle of estimating the modulation frequency of another light source. It is a figure explaining the method of estimating the modulation frequency of another light source. It is a figure explaining the correction method on IQ coordinates. It is a figure explaining the method of calculating the position and distance of another light source. It is a flowchart explaining the distance measurement processing by another light source. It is a flowchart explaining the modulation frequency estimation process. It is a block diagram which shows an example of the schematic structure of a vehicle control system. It is explanatory drawing which shows an example of the installation position of the vehicle exterior information detection unit and the image pickup unit.

The preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings below. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The explanation will be given in the following order.
1. 1. Outline of the present disclosure 2. First Embodiment 3. Second embodiment 4. Third embodiment 5. Application example to moving body

Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

<< 1. Outline of this disclosure>
<Effects of multiple ranging devices>
The present disclosure realizes distance measurement with high accuracy even when a plurality of distance measuring devices are used.

For example, as shown in FIG. 1, it is assumed that a plurality of depth sensors (including a distance measuring device) 1-1 and 1-2 are used to capture a depth image (distance image) up to the object 4 respectively. ..

The depth sensors 1-1 and 1-2 in FIG. 1 are provided with a light source 2-1 and 2-2 and a distance measuring device 3-1 and 3-2, respectively.

The light sources 2-1 and 2-2 irradiate the object 4 with a laser beam having a predetermined pulse width.

The distance measuring devices 3-1 and 3-2 receive the reflected light R1 and R2 generated by the irradiation light L1 and L2 emitted from the light sources 2-1 and 2-2 being reflected by the object 4, respectively. The distance to the object 4 is measured in pixel units based on the round-trip time of the irradiation light, and a depth image composed of the distance measurement results in pixel units is generated.

The depth sensors 1-1, 1-2, the light sources 2-1 and 2, and the distance measuring devices 3-1 and 3-2 are simply referred to as the depth sensor 1, the light source 2, and the distance measuring devices 3-1 and 3-2, respectively, unless it is necessary to distinguish them. And the distance measuring device 3, and other configurations are also referred to in the same manner.

<Principle of distance measurement>
The ranging device 3 receives the reflected light generated by reflecting the irradiation light composed of the laser light having a predetermined pulse width emitted from the light source 2 by the object 4, and based on the phase difference between the irradiation light and the reflected light. , The distance to the object 4 is measured by obtaining the round-trip time of the laser beam.

More specifically, the light source 2 irradiates, for example, a laser beam having a pulse width as shown in the uppermost stage of FIG. 2 as irradiation light.

The distance measuring device 3 receives the reflected light as shown in the second stage from the top of FIG. 2 generated by the irradiation light reflected by the object 4 as shown in the uppermost stage of FIG. 2, and the received light amount. It is provided with a light receiving element that generates a pixel signal according to the above. As for the phase of the reflected light, a delay time ΔT occurs due to the phase difference according to the distance to the object with respect to the phase of the irradiation light.

The distance measuring device 3 measures the distance to the object 4 based on the delay time ΔT caused by this phase difference. At this time, the distance measuring device 3 synchronizes the exposure time of the light receiving element with the phase of the irradiation light, and receives light, for example, by dividing it into two periods.

More specifically, the distance measuring device 3 includes exposure A, which is the exposure time when the pulse of the irradiation light is a Hi signal, and the fourth stage from the top of FIG. 2, which is shown in the third stage from the top of FIG. The light is received by being divided into two periods of the same length as the exposure B, which is the exposure time when the pulse of the irradiation light is a Low signal, which is indicated by.

In the following, among the light receiving elements of the ranging device 3, the light receiving element exposed in the exposure period shown by the exposure A in FIG. 2 is referred to as Tap A, and the light receiving element exposed in the exposure period shown in the exposure B in FIG. The element is called TapB.

The sum of the pixel signals generated by photoelectric conversion in each of the exposure periods of TapA and TapB is always the pixel signal detected in the exposure period corresponding to the light emission period of the light source 2, where the phase of the irradiation light is the Hi signal. ..

On the other hand, the pixel signal generated by TapB is a pixel signal in the exposure period that is reduced by the delay time ΔT with respect to the pixel signal generated in the period that is regarded as the Hi signal in the irradiation light.

Therefore, the distance measuring device 3 obtains the delay time ΔT from the ratio of the pixel signals of Tap B to the sum of the pixel signals of Tap A and Tap B, and uses the obtained delay time ΔT to reciprocate the light between the distance measuring device 3 and the object 4. It is regarded as time, and the distance from the distance measuring device 3 to the object 4 is measured based on the round-trip time.

<Effect of using multiple ranging devices>
The depth sensor 1 measures the distance according to the principle of distance measurement described with reference to FIG. 2, but when a plurality of depth sensors 1 measure the distance to the same object 4, the depth sensor 1 from the object 4 with respect to its own irradiation light. Appropriate distance measurement cannot be achieved unless the reflected light is received.

For example, in the case shown in FIG. 1, the irradiation lights L1 and L2 emitted by the light sources 2-1 and 2-1 in the depth sensors 1-1 and 1-2 are reflected by the object 4, respectively, and the reflected light R1 and R1 As long as the distance measuring devices 3-1 and 3-2 receive light and measure the distance as R2, appropriate distance measurement can be realized.

However, for example, when the ranging device 3-2 receives the reflected light R2 with respect to the irradiation light L2 of the light source 2-2 and the reflected light R1 corresponding to the irradiation light L1 of the light source 2-1 together, the reflected light R1 Due to the influence, proper distance measurement will not be possible. That is, for the distance measuring device 3-2, the reflected light R2 corresponding to the irradiation light L2 is the reflected light necessary for distance measurement, but the reflected light R1 corresponding to the irradiation light L1 is the reflected light unnecessary for the distance measurement. It becomes. Hereinafter, the reflected light required for distance measurement is also referred to as a necessary reflected light, and the irradiation light corresponding to the required reflected light is also referred to as a necessary irradiation light. Similarly, the irradiation light unnecessary for distance measurement is also referred to as unnecessary irradiation light, and the reflected light corresponding to the unnecessary irradiation light is also referred to as unnecessary reflected light. Therefore, for the ranging device 3-2, the irradiation light L2 is the necessary irradiation light and the reflected light R2 is the necessary reflected light, but the irradiation light L1 is the unnecessary irradiation light and the reflected light R1 is unnecessary. It is reflected light. This will be similarly affected by the distance measuring device 3-1.

As described above, when a plurality of depth sensors 1-1 and 1-2 are used, the distance measuring device 3-1 and 3-2 have mutual influences between the required reflected light and the unnecessary reflected light (or the required irradiation light and the light). There is a risk that proper distance measurement cannot be achieved due to mutual influence with unnecessary irradiation light). That is, the modulation frequency and phase of the required irradiation light and the required reflected light are known, but the modulation frequency and phase are unknown due to the influence of the unnecessary irradiation light and the unnecessary reflected light whose modulation frequency and phase are unknown. There is a risk that it will change to something and it will not be possible to measure the distance properly. Hereinafter, the effect of changing the modulation frequency or phase on the required irradiation light or required reflected light having a known modulation frequency or phase is also referred to as collision. That is, the collision between the required irradiation light or the required reflected light and the unnecessary irradiation light or the unnecessary reflected light may cause a change in the known modulation frequency or phase, making it impossible to realize appropriate distance measurement. In order to realize appropriate distance measurement even when a plurality of depth sensors 1-1 and 1-2 are used in this way, the depth sensors 1-1 and 1-2 should be able to receive the reflected light R1 and R2 separately. Therefore, it is necessary to suppress the influence of collision so that only the necessary reflected light can be used for distance measurement. Several methods can be considered in order to distinguish the reflected light R1 and R2 so that they can be received.

As a first method of distinguishing and receiving the reflected light R1 and R2, for example, as shown in the upper part of FIG. 3, the light sources 2-1 and 2-2 do not overlap with each other and emit light at equal intervals. A method of controlling the timing can be considered.

In addition, in FIG. 3, the waveform of the emission timing of the irradiation light L1 irradiated by the light source 2-1 is shown in the upper waveforms of the upper, middle, and lower stages, and in the lower waveform, the emission timing waveform is shown. The waveform of the emission timing of the irradiation light L2 emitted by the light source 2-2 is shown.

Both waveforms indicate that the timing at which the rectangular pulse waveform is generated is the timing at which the predetermined modulated light is emitted.

Further, as a second method of receiving the reflected light R1 and R2 separately, for example, as shown in the middle part of FIG. 3, light is emitted so that the emission timings of the light sources 2-1 and 2-2 do not overlap. A method of randomly controlling the timing (interval) by a random number or the like can be considered. In the middle stage of FIG. 3, the emission timing of the irradiation light L1 is set to the cycle T1, and the emission timing of the irradiation light L2 is set to the cycle T2.

Further, as a third method of distinguishing and receiving the reflected light R1 and R2, for example, as shown in the lower part of FIG. 3, the modulation method related to the light emission of the light sources 2-1 and 2-2 is randomly changed. A method of controlling such as this can be considered. In the case of this method, the distance measuring devices 3-1 and 3-2 know the modulation method related to the light emission of the light sources 2-1 and 2, so that even if the light emission timings of the irradiation light overlap. It can be separated as background light.

However, in order to realize the first method, the depth sensors 1-1, 1-2 are configured to control the light emission timings of the light sources 2-1 and 2-2 of the depth sensors 1-1 and 1-2 so as not to overlap with each other. It is necessary to separately provide the device or one of the depth sensors 1-1 and 1-2 to control the timing, which increases the cost related to the device configuration and complicates the control.

Further, in the second method, since the light emission timing of the light sources 2-1 and 2-2 changes randomly, the light may be emitted at the timing when the irradiation lights L1 and L2 overlap. The ranging devices 3-1, 3-2 cannot determine whether or not the timings overlap, and cannot separate the reflected lights R1 and R2 corresponding to the irradiation lights L1 and L2, so that they are not always appropriate. Distance measurement may not be possible.

Further, in the third method, even if the modulation method related to the light emission of the light sources 2-1 and 2-2 changes randomly, the possibility of matching cannot be denied, so that it may not always be possible to realize appropriate distance measurement. is there.

Therefore, in the present disclosure, the interval between the emission timings of the own light source 2 is set wide by a predetermined time, and the pixel signal obtained by the reflected light with respect to the irradiation light from the own light source 2 in the distance measuring device 3 and others. The pixel signal obtained by the reflected light with respect to the irradiation light from the light source 2 of the above is integrated multiple times, and the irradiation light and the reflected light by the own light source 2 are integrated by the distance measurement calculation of the 4-phase method from the difference of the integration result. The distance to the object 4 is measured by obtaining the phase difference with.

The interval of the light emission timing referred to here is not the interval of each pulse waveform, but the light source in which the light source blinks and emits light in a state of being modulated at a predetermined modulation frequency, and the light source is completely. Indicates the interval from the non-lighting state.

Here, the distance measurement calculation method of the 4-phase method will be described.

The ranging calculation method described with reference to FIG. 2 is a 2-phase ranging calculation method as opposed to the 4-phase ranging calculation method, and the irradiation light is emitted in synchronization with the phase of the irradiation light. Two-phase pixel signals are detected in one frame (during one cycle related to modulation) by TapA and TapB in synchronization with the timing and the timing when the lights are turned off, and the pixels of TapB with respect to the sum of the pixel signals of TapA and TapB. This is a method of performing distance measurement calculation based on the phase difference expressed by the ratio of signals.

On the other hand, the 4-phase method is the following distance measurement calculation method.

Here, in explaining the 4-phase ranging calculation method, as shown in the uppermost stage (Emitted Light) of FIG. 4, modulation is performed so that irradiation is repeatedly turned on / off at the irradiation time T (1 cycle = 2T). The emitted irradiation light is output. Further, in the light receiving element of the distance measuring device 3, as shown in the second stage (Reflected Light) of FIG. 4, the reflected light generated by the reflection of the irradiation light by the object has a delay time ΔT according to the distance to the object. It is assumed that the light is received with a delay.

In the 4Phase method, as shown by φ0 to φ3 in the third to seventh stages from the top of FIG. 4, the exposure timings of TapA and TapB are controlled so that the phase is the same as that of the irradiation light (that is, Phase0). The exposure is performed at four timings of a phase shifted by 90 degrees (Phase90), a phase shifted by 180 degrees (Phase180), and a phase shifted by 270 degrees (Phase270), and a pixel signal is detected in each exposure period.

The timings of the same phase (Phase0) and the phase (Phase180) can be set continuously in the same frame, and the phase (Phase90) and the phase (Phase270) can also be set continuously in the same frame. can do. Therefore, for phase (Phase0) and phase (Phase180), TapA receives light at continuous timing, and for phase (Phase90) and phase (Phase270), TapB receives light at continuous timing, so that in one frame. At the same time, the exposure time for 4 phases can be set.

The signal values detected in Phase 0, Phase 90, Phase 180, and Phase 270 of Tap A and Tap B in the 4 Phase method are also referred to as q _0A , q _1A , q _2A , and q _3A , respectively.

The phase shift amount θ corresponding to the delay time ΔT can be detected by the distribution ratio of the signal values q _0A , q _1A , q _2A , and q _3A . That is, since the delay time ΔT is obtained based on the phase shift amount θ, the distance to the object can be obtained from the delay time ΔT.

The distance to the object 4 is calculated by, for example, the following equation (1).

... (1)

Here, C is the speed of light, ΔT is the delay time, f _mod is the modulation frequency of light, and φ0 to φ3 are shown in the third to seventh stages of FIG. 4, respectively. The signal values q _0A , q _1A , q _2A , and q _3A detected in Phase 90, Phase 180, and Phase 270, respectively, and Distance is the distance from the distance measuring device 3 to the object 4.

In the present disclosure, if it is not affected by a plurality of depth sensors 1, the distance measurement calculation is normally performed by the 4-phase method, and hereinafter, the operation mode by this distance measurement calculation is referred to as a normal mode.

However, when a plurality of depth sensors 1 are provided, even in the distance measurement calculation of the 4-phase method, it is affected by the irradiation light emitted by the light source 2 of the other depth sensor 1, so the measurement as it is is appropriate. Distance cannot be achieved.

Therefore, in the present disclosure, when the same object 4 is measured by a plurality of depth sensors 1 and the plurality of depth sensors 1 are affected by other irradiation light, as shown in FIG. 5, the own irradiation light is used. The light emission interval of is set longer than the time when the reflected light is sufficiently attenuated and the amount of light is reduced. In other words, during the light emission period in which the light source 2 irradiates the irradiation light modulated at the modulation frequency, a turn-off period is set in which the light source 2 having a sufficient length for sufficiently diminishing the amount of the irradiation light is turned off. To.

In FIG. 5, for each of the N frames to N + 3 frames from the top of the figure, the waveform WL of its own irradiation light, the waveform of the reflected light WR from the object 4 due to its own irradiation light, and the object 4 due to other irradiation light. The waveform of the reflected light WRo from is shown, and the exposure timings of TapA and TapB are shown. Further, at each exposure timing by Tap A and Tap B, which of the reflected light WR and WRo is received during each exposure period is color-coded. That is, in Tap A and Tap B of FIG. 5, the pointillistic region represents the timing at which the reflected light WRo is received, and the grid-like region represents the timing at which the reflected light WR is received. ..

In each of the N frames to the N + 3 frames, the waveform WL of the irradiation light emits light with a period _TL as shown by the times t0 to t1, t2 to t3, and t4 to t5. The period _TL is set to be sufficiently longer than the time Td at which the brightness is sufficiently attenuated after the emission period of the reflected light ends.

Further, in each of the N frame to the N + 3 frame, the waveform WR of the reflected light corresponding to the own irradiation light is only the delay time ΔT according to the distance to the object 4 with respect to the waveform WL of the own irradiation light. The light is received in a state of light emission at the delayed timings t11 to t12, t13 to t14, and t15 to t16.

TapA has the same phase (that is, Phase 0) and a phase shifted by 180 degrees (Phase180) from the upper part of FIG. 5 with respect to the period during which the irradiation light is emitted for each of the N frame to the N + 3 frame. Both the reflected light that is exposed in the phase shifted by 90 degrees (Phase90) and the phase shifted by 270 degrees (Phase270) and that its own irradiation light is reflected by the object 4 and the reflected light that other irradiation light is reflected by the object 4. Is received and the pixel signal is detected.

On the other hand, in TapB, for each of the N frames to N + 3 frames, the time Td at which the brightness sufficiently attenuates after the emission period of the reflected light ends in synchronization with the period during which the irradiation light is emitted. Is exposed in the same phase (that is, Phase 0), 180 degree shifted phase (Phase 180), 90 degree shifted phase (Phase 90), and 270 degree shifted phase (Phase 270), and other irradiation light is an object. Only the reflected light reflected in 4 is received to detect the pixel signal.

In the present disclosure, a pixel signal that combines the reflected light of its own light source 2 for four phases and the reflected light of another light source 2 thus obtained in units of four frames, and another light source for four phases. The pixel signal of only the reflected light of 2 is repeatedly integrated a predetermined number of times.

Then, the integrated value of the pixel signals for four phases, which is the sum of the reflected light corresponding to the own irradiation light detected by TapA and the reflected light of the other light source 2, and the other ones detected by TapB. Using the integrated value of the pixel signals for four phases consisting only of the reflected light corresponding to the irradiation light, φ0 to φ3 in the above equation (1) are expressed as shown by the following equation (2), and the distance measurement is performed. To realize.

φ0 ＝ ΣTapA _N －ΣTapB _N
φ1 ＝ ΣTapA _{N + 2} －ΣTapB _{N + 2}
φ2 ＝ ΣTapA _{N + 1} －ΣTapB _{N + 1}
φ3 ＝ ΣTapA _{N + 3} －ΣTapB _{N + 3}

‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥‥

ΣTapA _N , ΣTapA _{N + 1} , ΣTapA _{N + 2} , and ΣTapA _{N + 3} are the integrated values of the pixel signals detected in each of the four frames N to N + 3 in TapA, respectively. ..

In addition, ΣTapB _N , ΣTapB _{N + 1} , ΣTapB _{N + 2} , and ΣTapB _{N + 3} are the integrated values of the predetermined number of pixel signals detected in each of the four frames N to N + 3 in TapB, respectively. Is.

In the case of the equation (2), φ0 to φ3 correspond to the signal values q _0A , q _1A , q _2A , and q _3A detected in each of the phases Phase 0, Phase 90, Phase 180, and Phase 270, respectively.

The delay time ΔT is represented by α / (2πf _mod ).

That is, from the integrated value of the pixel signal detected by the reflected light of the reflected light WR and the reflected light WRo in each frame unit of N to N + 3 frames in TapA, N to N + 3 frames in TapB. The integrated value of the pixel signal detected only by the reflected light WRo in each frame unit is subtracted.

From the integrated value (ΣTapA _{N to} ΣTapA _{N + 3} ) of the average value of the pixel signals detected by the reflected light WR and the reflected light WRo together by such calculation, the pixel signal detected only by the reflected light WRo By subtracting the integrated value (ΣTapB _{N to} ΣTapB _{N + 3} ) of the average value, the component of the reflected light corresponding to the irradiation light from the other light source 2 is substantially removed.

As a result, in the calculation of (φ1-φ3) / (φ0-φ2), a value in which the influence of the reflected light WRo based on the other light source 2 is substantially removed is used, so that the other light source is used. It is possible to make corrections so as to suppress the influence of 2, and it is possible to realize an appropriate distance measurement calculation.

As described with reference to FIG. 5, the process of performing the distance measurement calculation by correcting so as to suppress the influence of the reflected light by the other light sources 2 by the plurality of depth sensors 1-1 and 1-2 is shown in FIG. The normal mode processing described with reference to 4 is referred to as a correction mode processing.

<< 2. First Embodiment >>
<Configuration example of depth sensor>
FIG. 6 is a block diagram showing a configuration example of an embodiment of a depth sensor to which the present disclosure is applied.

In FIG. 6, the depth sensor 11 includes an optical modulation unit 21, a light emitting diode 22, a light projecting lens 23, a light receiving lens 24, a filter 25, a TOF sensor 26, an image storage unit 27, a synchronization processing unit 28, a calculation unit 29, and a depth image generation. It is configured to include a unit 30 and a control unit 31.

The optical modulation unit 21 is controlled by the control unit 31 and supplies a modulation signal for modulating the light output from the light emitting diode 22 at a high frequency of, for example, about 10 MHz to the light emitting diode 22. Further, the optical modulation unit 21 is controlled by the control unit 31 and supplies a timing signal indicating the timing at which the light of the light emitting diode 22 is modulated to the TOF sensor 26 and the synchronization processing unit 28.

The light emitting diode 22 emits light while modulating light in an invisible region such as infrared light at high speed according to a modulation signal supplied from the light modulation unit 21, and the depth sensor 11 acquires a depth image of the light. Then, the light is emitted toward the detection area, which is the area for detecting the distance. In the present embodiment, the light source that irradiates the light toward the detection region is described as the light emitting diode 22, but another light source such as a laser diode may be used.

Further, the light emitting diode 22 may be arranged adjacent to the TOF sensor 26 described later. With this configuration, when the emitted light is reflected by the object and returned to the depth sensor 11, the path difference between the going and returning is minimized, and the distance measurement error can be reduced. Further, the light emitting diode 22 and the TOF sensor 26 may be integrally formed in one housing. With this configuration, it is possible to suppress variations in the going and returning paths when the emitted light is reflected by the object and returned to the depth sensor 11, and it is possible to reduce the distance measurement error. Further, the light emitting diode 22 and the TOF sensor 26 may be formed from different housings as long as the mutual positional relationship can be grasped.

The light projecting lens 23 is composed of a lens that adjusts the light distribution so that the light emitted from the light emitting diode 22 has a desired irradiation angle.

The light receiving lens 24 is composed of a lens that captures the detection region where the light is emitted from the light emitting diode 22 in the field of view, and the light reflected by the object in the detection region is imaged on the sensor surface of the TOF sensor 26.

The filter 25 is a BPF (Band Pass Filter) that allows only light in a predetermined band to pass through, and is of the light in a predetermined pass band among the light reflected by an object in the detection region and incident on the TOF sensor 26. Only light passes through. For example, the filter 25 has a central wavelength of the pass band set between 920 nm and 960 nm, and allows more light in the pass band to pass than light having a wavelength other than the pass band. Specifically, in the filter 25, light having a wavelength in the pass band is transmitted by 60% or more, and light having a wavelength other than the pass band is transmitted by less than 30%.

Further, the pass band through which the filter 25 passes light is narrower than the pass band of the filter used in the conventional TOF method, and is limited to a narrow band corresponding to the wavelength of the light emitted from the light emitting diode 22. For example, when the wavelength of the light emitted from the light emitting diode 22 is 940 nm, a band of 10 nm before and after (930 to 950 nm) centered on 940 nm is adopted as a pass band of the filter 25 in conjunction with the wavelength. ..

With such a pass band of the filter 25, the TOF sensor 26 can detect the irradiation light while suppressing the influence of the disturbance light of the sun. The pass band of the filter 25 is not limited to this, and may be a band of 15 nm or less before and after the predetermined wavelength. Further, as the pass band of the filter 25, a band (840 to 860 nm) of 10 nm before and after centering on 850 nm, which is the wavelength band having the best characteristics of the TOF sensor 26, may be adopted. As a result, the TOF sensor 26 can effectively detect the irradiation light.

The TOF sensor 26 is composed of an image pickup element having sensitivity in the wavelength range of the light emitted from the light emitting diode 22, and the light collected by the light receiving lens 24 and passed through the filter 25 is arranged in an array on the sensor surface. Light is received by a plurality of pixels. As shown in the figure, the TOF sensor 26 is arranged in the vicinity of the light emitting diode 22 and can receive the light reflected by the object in the detection region where the light is irradiated by the light emitting diode 22. Then, the TOF sensor 26 outputs a pixel signal in which the amount of light received by each pixel is used as a pixel value for generating a depth image. As a specific configuration of the TOF sensor 26, for example, various sensors such as SPAD (single photon avalanche diode), APD (Avalanche Photo Diode) and CAPD (Current Assisted Photonic Demodulator) can be applied.

The image storage unit 27 stores an image constructed by the pixel signal output from the TOF sensor 26. For example, the image storage unit 27 can store the latest image when there is a change in the detection area, or can store an image in a state where no object exists in the detection area as a background image. ..

The synchronization processing unit 28 receives the reflected light corresponding to the modulated light emitted by the light emitting diode 22 among the pixel signals supplied from the TOF sensor 26 in synchronization with the timing signal supplied from the optical modulation unit 21. Performs the process of extracting the pixel signal of. As a result, the synchronization processing unit 28 can supply only the pixel signal of the timing whose main component is the light emitted from the light emitting diode 22 and reflected by the object in the detection region to the calculation unit 29. Further, the synchronization processing unit 28 moves in the detection region by, for example, reading the background image stored in the image storage unit 27 and obtaining the difference from the image constructed by the pixel signal supplied from the TOF sensor 26. It is possible to generate a pixel signal consisting only of an object with a certain object. The synchronization processing unit 28 is not an essential configuration, and the pixel signal supplied from the TOF sensor 26 may be directly supplied to the calculation unit 29.

The calculation unit 29 calculates the distance to the object in the detection area for each pixel based on the pixel signal supplied from the synchronization processing unit 28 or the TOF sensor 26, and calculates the distance obtained by the calculation. The indicated depth signal is supplied to the depth image generation unit 30.

Specifically, the calculation unit 29 has a phase difference between the phase of the light emitted by the light emitting diode 22 and the phase of the light emitted from the light emitting diode 22 reflected by an object and incident on the pixels of the TOF sensor 26. Based on, the calculation for finding the distance to the object in the detection area is performed.

The depth image generation unit 30 generates a depth image in which the distances to the subject are arranged according to the arrangement of pixels from the depth signal supplied from the calculation unit 29, and the depth image is sent to a subsequent processing device (not shown). Output.

The control unit 31 is composed of a processor and a memory, and controls the entire operation of the depth sensor 11. The control unit 31 controls the optical modulation unit 21 according to the operation mode, and controls the light emission timing of the light emitting diode 22 by controlling the modulation method. The control unit 31 controls the TOF sensor 26 according to the operation mode to control, for example, the operation timings of TapA and TapB. The control unit 31 controls the calculation content of the calculation unit 29 according to the operation mode.

The operation mode referred to here is, for example, the above-mentioned normal mode and correction mode.

That is, when the depth sensor 11 itself generates a depth image of a region including a predetermined object, the control unit 31 performs the normal mode processing described with reference to FIG. Various configurations are controlled so as to.

Further, when the plurality of depth sensors 11 generate a depth image of a region including the same predetermined object, the control unit 31 sets the operation mode to the correction mode processing described with reference to FIG. Control various configurations.

The light emitting diode 22 and the light projecting lens 23 in FIG. 6 constitute a light source 32 corresponding to the light source 2 in FIG. Further, the light receiving lens 24, the filter 25, and the TOF sensor 26 constitute a distance measuring device 33 corresponding to the distance measuring device 3 of FIG.

As described above, the depth sensor 11 configured in this way employs a narrow band filter 25 in which the pass band is limited to a narrow band corresponding to the wavelength of the light emitted from the light emitting diode 22. As a result, the filter 25 can remove noise components due to ambient light and allow a large amount of signal components required for measurement to pass through. That is, in the depth sensor 11, the TOF sensor 26 can receive more light, which is a signal component necessary for measurement, as compared with ambient light, which is a noise component, and the SN ratio (Signal to) of the acquired depth image. Noise Ratio) can be improved.

Therefore, the depth sensor 11 can increase the acquisition distance capable of acquiring a more accurate depth image even in an environment affected by ambient light, and can improve the depth image acquisition performance. Can be improved.

<Example of TOF sensor configuration>
Next, a configuration example of the TOF sensor 26 of FIG. 6 will be described with reference to FIG. 7.

The TOF sensor 26 shown in FIG. 7 is composed of, for example, a back-illuminated type sensor, but may be a front-illuminated type sensor.

The TOF sensor 26 has a configuration including a pixel array unit 40 formed on a semiconductor substrate (not shown) and a peripheral circuit unit integrated on the same semiconductor substrate as the pixel array unit 40. The peripheral circuit unit is composed of, for example, a tap drive unit 41, a vertical drive unit 42, a column processing unit 43, a horizontal drive unit 44, and a system control unit 45.

The TOF sensor 26 is also provided with a signal processing unit 49 and a data storage unit 50. The signal processing unit 49 and the data storage unit 50 may be mounted on the same substrate as the TOF sensor 26, or may be arranged on a substrate different from the TOF sensor 26 in the imaging device.

The pixel array unit 40 has a configuration in which pixels 51 that generate an electric charge according to the amount of received light and output a signal corresponding to the electric charge are two-dimensionally arranged in a matrix in the row direction and the column direction. That is, the pixel array unit 40 has a plurality of pixels 51 that photoelectrically convert the incident light and output a signal corresponding to the electric charge obtained as a result. Here, the row direction refers to the arrangement direction of the pixels 51 in the horizontal direction, and the column direction refers to the arrangement direction of the pixels 51 in the vertical direction. The row direction is the horizontal direction in the figure, and the column direction is the vertical direction in the figure.

The pixel 51 receives light incident from the outside, particularly infrared light, and performs photoelectric conversion, and outputs a pixel signal corresponding to the electric charge obtained as a result. The pixel 51 applies a predetermined voltage MIX0 (first voltage) to detect the photoelectrically converted charge, and has a first tap TA corresponding to the above-mentioned TapA and a predetermined voltage MIX1 (second voltage). ) Is applied to detect the photoelectrically converted charge, and has a second tap TB corresponding to the above-mentioned Tap B.

The tap drive unit 41 supplies a predetermined voltage MIX0 to the first tap TA (TapA) of each pixel 51 of the pixel array unit 40 via a predetermined voltage supply line 48, and supplies a predetermined voltage MIX0 to the second tap TB (TapB). Is supplied with a predetermined voltage MIX1 via a predetermined voltage supply line 48. Therefore, two voltage supply lines 48, a voltage supply line 48 for transmitting the voltage MIX0 and a voltage supply line 48 for transmitting the voltage MIX1, are wired in one pixel array of the pixel array unit 40.

In the pixel array unit 40, a pixel drive line 46 is wired along the row direction for each pixel row with respect to the matrix-like pixel array, and two vertical signal lines 47 are wired along the column direction in each pixel row. ing. For example, the pixel drive line 46 transmits a drive signal for driving when reading a signal from a pixel. In FIG. 1, the pixel drive line 46 is shown as one wiring, but the wiring is not limited to one. One end of the pixel drive line 46 is connected to the output end corresponding to each line of the vertical drive unit 42.

The vertical drive unit 42 is composed of a shift register, an address decoder, and the like, and drives each pixel of the pixel array unit 40 at the same time for all pixels or in line units. That is, the vertical drive unit 42 constitutes a drive unit that controls the operation of each pixel of the pixel array unit 40 together with the system control unit 45 that controls the vertical drive unit 42.

The signal output from each pixel 51 of the pixel row according to the drive control by the vertical drive unit 42 is input to the column processing unit 43 through the vertical signal line 47. The column processing unit 43 performs predetermined signal processing on the pixel signal output from each pixel 51 through the vertical signal line 47, and temporarily holds the pixel signal after the signal processing.

Specifically, the column processing unit 43 performs noise removal processing, AD (Analog to Digital) conversion processing, and the like as signal processing.

The horizontal drive unit 44 is composed of a shift register, an address decoder, and the like, and sequentially selects unit circuits corresponding to the pixel strings of the column processing unit 43. By the selective scanning by the horizontal drive unit 44, the pixel signals processed by the column processing unit 43 for each unit circuit are sequentially output.

The system control unit 45 is composed of a timing generator or the like that generates various timing signals, and based on the various timing signals generated by the timing generator, the tap drive unit 41, the vertical drive unit 42, the column processing unit 43, And the drive control of the horizontal drive unit 44 and the like is performed.

The signal processing unit 49 has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing based on the pixel signal output from the column processing unit 43. The data storage unit 50 temporarily stores data necessary for signal processing by the signal processing unit 49.

<Normal mode processing>
Next, the normal mode processing described with reference to FIG. 4 will be described with reference to FIG. In the normal mode processing, the distance is measured by a single depth sensor 11 and the depth image is generated based on the distance measurement result, so that the influence of other light sources is not considered.

In step S11, the control unit 31 controls the optical modulation unit 21 to cause the light emitting diode 22 to emit light at a predetermined modulation frequency to irradiate the irradiation light. By this process, the irradiation area is irradiated with the irradiation light composed of the modulated light modulated at a predetermined modulation frequency from the light source 32, and the reflected light reflected by the object in the irradiation area is incident on the TOF sensor 26. ..

In step S12, the control unit 31 controls the first tap TA (TapA) of each of the pixels 51 of the TOF sensor 26 to expose the light emitting diode 22 in synchronization with the light emitting timing, and has the same phase (that is, that is). Phase0) and 180-degree shifted phase (Phase180) pixel signals are continuously detected and output.

That is, by this processing, the signal values q _0A and q _2A detected in the phases Phase 0 and Phase 180, which are φ0 and φ2 in the equation (1), which are described with reference to FIG. 4 of each pixel 51, are continuous. Will be required.

In step S13, the control unit 31 controls the second tap TB (TapB) of each of the pixels 51 of the TOF sensor 26 to expose the light emitting diode 22 in a phase shifted by 90 degrees from the light emitting timing of the light emitting diode 22 to 90 degrees. Pixel signals having a shifted phase (Phase90) and a shifted phase (Phase270) of 270 degrees are continuously detected and output to the calculation unit 29 via the synchronization processing unit 28.

That is, by this processing, the signal values q _1A and q _3A detected in the phases Phase 90 and Phase 270, which are φ1 and φ3 in the equation (1), which are described with reference to FIG. 4 of each pixel 51, are continuous. Will be required.

In step S14, the calculation unit 29 calculates the distance measurement calculation for each pixel 51 by calculating the equation (1), and outputs the depth signal as the calculation result to the depth image generation unit 30.

In step S15, the depth image generation unit 30 generates and outputs a depth image based on the depth signal of each pixel 51.

In step S16, the control unit 31 determines whether or not the end of the process is instructed, and if the end is not instructed, the process returns to step S11. That is, the processes of steps S11 to S16 are repeated until the end of the process is instructed, and the depth image is generated and continues to be output.

Then, in step S16, when the end of the process is instructed, the process ends.

By the above series of processing, distance measurement processing is performed by a single depth sensor 11, and a depth image is generated and output based on the distance measurement result. In the normal mode, signal values for four phases are obtained in one frame and a depth image is generated, so that high-speed processing can be realized.

<Correction mode processing>
Next, the correction mode processing will be described with reference to the flowchart of FIG. The correction mode processing is a processing assuming that distance measurement processing is performed on the same object by a plurality of depth sensors 11.

In step S31, the control unit 31 controls the optical modulation unit 21 to emit light from the light emitting diode 22 at a predetermined modulation frequency and at a predetermined interval to irradiate the irradiation light.

That is, in the correction mode processing, as described with reference to FIG. 5, the light emitting diode 22 emits light at a period TL sufficiently longer than the length Td at which the light emitted at a predetermined modulation frequency is sufficiently attenuated. It is controlled to repeat turning off.

Further, by this processing, the irradiation light composed of the modulated light modulated at a predetermined modulation frequency from the light source 32 is irradiated to the irradiation region in the periodic TL, and the reflected light reflected by the object in the irradiation region is applied to the TOF sensor 26. It will be in a state of being incident.

In step S32, the control unit 31 initializes the counter θ for counting the phase shift to 0. This counter θ is a counter that counts four types of phases of 0, 90, 180, and 270 degrees in 90 degree increments.

In step S33, the control unit 31 controls the first tap TA (TapA) of each of the pixels 51 of the TOF sensor 26, and exposes the light at a timing that is out of phase by θ degrees with respect to the light emission timing of the light emitting diode 22. The pixel signal is detected and output to the calculation unit 29 via the synchronization processing unit 28.

In step S34, the calculation unit 29 integrates and stores the pixel signal which is the exposure result of TapA in association with the counter θ.

In step S35, the control unit 31 controls the first tap TB (TapB) of each of the pixels 51 of the TOF sensor 26, and θ with respect to the timing when the light emitting diode 22 is turned off and the light is sufficiently attenuated. The pixels are exposed at a timing that is out of phase by a degree, and the pixel signal is detected and output to the calculation unit 29 via the synchronization processing unit 28.

In step S36, the calculation unit 29 integrates and stores the pixel signal which is the exposure result of TapB in association with the counter θ.

In step S37, the control unit 31 determines whether or not the counter θ is 270, and if it is not 270, the process proceeds to step S38.

In step S38, the control unit 31 increments the counter θ by 90, and the process returns to step S33.

That is, in step S37, the processes of steps S33 to S38 are repeated until it is determined to be 270, and pixel signals corresponding to four types of phase shifts of 0, 90, 180, and 270 degrees are sequentially integrated.

Then, if it is determined in step S37 that the counter θ is 270, the process proceeds to step S39.

In step S39, the control unit 31 determines whether or not the processes of steps S32 to S38 have been repeated a predetermined number of times, and if the processes have not been repeated a predetermined number of times, the process returns to step S32.

That is, in the processing in steps S33 and S34, the pixel signals of the reflected light WR with respect to the own irradiation light in frames N to N + 3 of FIG. This is the process of integrating.

In the processing in steps S35 and S36, the pixel signals of the reflected light WRo with respect to the other irradiation light in the frames N to N + 3 of FIG. 5 corresponding to the counters θ corresponding to 0, 90, 180, and 270 are repeatedly detected and integrated into each. It is a process.

Then, by repeating the processes of steps S32 to S39, the pixel signals of the reflected light WR and the reflected light WRo are repeatedly integrated a predetermined number of times in association with the four types of phase shifts of 0,90,180,270.

That is, by repeating the processes of steps S32 to 39, ΣTapA _N , ΣTapA _{N + 2} , ΣTapA _{N + 1} , ΣTapA _{N + 3} , and ΣTapB _N , ΣTapB _{N + 2} , ΣTapB _{N + 1 in the} equation (2). ， ΣTapB _{N + 3} is required.

If it is determined in step S39 that the process has been repeated a predetermined number of times, the process proceeds to step S40.

In step S40, the calculation unit 29 performs a distance measurement calculation for each pixel 51 by calculating the formula (1) using φ0 to φ3 obtained by the above formula (2), and the depth signal which is the calculation result. Is output to the depth image generation unit 30.

In step S41, the depth image generation unit 30 generates and outputs a depth image based on the depth signal of each pixel 51.

In step S42, the control unit 31 determines whether or not the end of the process is instructed, and if the end is not instructed, the process returns to step S32. That is, the processes of steps S32 to S42 are repeated until the end of the process is instructed, and the depth image is generated and continues to be output.

Then, in step S42, when the end of the process is instructed, the process ends.

Through the above series of processing, the subtraction results of φ0 to φ3 represented by the equation (2) are substantially the reflected light WR (necessary reflected light) based on the own light source 2 in each phase and the other. From the integrated value of the average value of the pixel signals detected by the reflected light including the reflected light WRo (unnecessary reflected light) based on the light source 2, the reflected light WRo (unwanted reflected light) based only on the other light source 2 in each phase. ), The integrated value of the average value of the pixel signals is removed. That is, φ0 to φ3 represented by the equation (2) are detected by the reflected light WR based on the own light source 2 in each phase, which does not include the pixel signal detected by the reflected light WRo based on the other light source 2. It is the integrated value of the average value of the pixel signals to be generated.

As a result, in the calculation of (φ1-φ3) / (φ0-φ2), a value in which the influence of the reflected light WRo based on the other light source 2 is substantially removed is used, so that the other light source is used. It is possible to make corrections so as to reduce the influence of the collision caused by 2, and it is possible to realize an appropriate distance measurement calculation.

<< 3. Second Embodiment >>
<Collision detection>
In the above, when a single depth sensor 11 independently generates a depth image of a predetermined region, a depth image is generated by normal mode processing, and a plurality of depth sensors 11 generate a depth image of the same predetermined region. In the case of generation, an example of generating a depth image by correction mode processing has been described.

However, when it is not possible to recognize whether or not a plurality of depth sensors 11 are used, it is necessary to generate a depth image by the correction mode processing, but the frame rate is lower than that of the normal mode processing, or the irradiation light Depth images will continue to be generated in situations where the Duty goes down.

Therefore, for example, as an operation mode for detecting interference or collision from another light source, TapB among the pixels 51 detects a pixel signal of four phases synchronized with the light emission timing and measures the distance with only TapA. A plurality of different integration times (IntegTime) are set in synchronization with the timing when the light is turned off, and the integrated value of the pixel signal is detected.

When modulated light from other light sources is not included, TapB detects only a certain level of disturbing light such as sunlight, so it detects an integrated value that changes in proportion to the length of the integrated time. To.

On the other hand, when modulated light from another light source is included, TapB detects a pixel signal of reflected light consisting of modulated light in addition to a certain level of interfering light such as sunlight. , An integrated value that changes regardless of the length of the integrated time is detected.

That is, in FIG. 10, the integrated value of the pixel signal when TapB is exposed at different integrated times Te1 to Te4 is detected in each of the Nth frame to the N + 3rd frame continuous from the top. It is shown.

The upper waveform in each frame is the waveform Wm of the modulated light, and the lower waveform is the waveform Ws of sunlight.

As shown in FIG. 10, since the waveform Wm of the modulated light changes according to the modulation, pixel signals are detected in the Nth frame to the N + 3rd frame at different lengths of integration times Te1 to Te4. However, an integrated value that does not correlate with the length of time is detected.

On the other hand, since the waveform Ws of sunlight is constant to some extent, when pixel signals are detected at the integration times Te1 to Te4 having different lengths in the Nth frame to the N + 3rd frame, the length of the integration time is determined. Correlated integrated values are detected.

Therefore, for example, the standard deviation of the integrated value considered to be background light such as sunlight at different integrated times is obtained in advance as a threshold value, and the standard deviation when the integrated value is detected a predetermined number of times at different integrated times and the standard deviation in advance. The presence or absence of collision may be determined based on the comparison with the obtained threshold value.

For example, when the standard deviation when the integrated values are detected a predetermined number of times at different integrated times is smaller than the preset threshold value, that is, when the variance of the integrated values is small and it can be considered that there is no variation, the influence of the modulated light is It can be determined that there is no collision with other light sources.

On the contrary, when the standard deviation of the integrated value is larger than the threshold value obtained in advance, the variance of the integrated value is large, and it can be considered that there is a variation, it is determined that there is an influence of the modulated light and there is a collision with another light source. be able to.

Note that TapA will perform distance measurement processing while detecting a collision. In this case, the distance measuring process using both TapA and TapB cannot be performed, but the distance measuring process may be performed although the frame rate is reduced while the collision is detected by TapB.

<Avoid collision>
In addition, when a collision with another light source is detected, the length of the period for turning off the light source, that is, the blank period, which is set during the period for emitting the light source, is randomly changed by using a random number. Therefore, the influence of the collision may be reduced by avoiding the collision with another light source.

That is, as shown in the upper part of FIG. 11, when the light emission period TL in the waveform WL of the own light source and the light emission period TL in the waveform WLo of another light source overlap, the collision is detected by the above-mentioned processing. It will be.

In the upper part of FIG. 11, when the light emission period TL is set after the blank Tb1 having a predetermined length, the exposure period Te of TapA is set synchronously, and then the read period Tr is set.

Then, after the blank period Tb1 is set again, when the light emission period TL is set, the exposure period Te of TapA is set in synchronization, and then the read period Tr is set, and the same process is repeated. ..

In the case of the upper part of FIG. 11, a collision will be detected. Therefore, by changing the blank period Tb1 to a length based on a random number, for example, as shown in the lower part of FIG. 11, a blank period is used. The period Tb2 may be changed.

As a result, the light emission period TLo in the waveform WLo of another light source and the light emission period TL do not overlap, so that collision is avoided.

Since the length of the blank period is randomly changed by a random number, there is a risk that collision will not be avoided even if the length of the blank period is changed. In this case, the length of the repeating blank period may be randomly changed until a collision is avoided.

Further, although the lengths of the blank period Tb1 in the upper part of FIG. 11 and the blank period Tb2 in the lower part of FIG. 11 are randomly changed, the lengths of the exposure period Te and the read period Tr remain the same. ..

Furthermore, when detecting a collision, the frame rate is reduced by measuring the distance only with TapA. Therefore, normally, the operation is performed in the normal mode processing, and the collision is detected every time a predetermined time elapses. Therefore, when a collision is detected, the length of the blank period may be changed at random.

The operation mode in which the collision is detected as described above and the collision is avoided when the collision is detected is hereinafter referred to as the collision avoidance mode processing.

<Collision avoidance mode processing>
Next, the collision avoidance mode processing will be described with reference to the flowchart of FIG.

In step S51, the control unit 31 performs distance measurement processing in pixel units in the normal mode described with reference to the flowchart of FIG. 8, and generates and outputs a depth image based on the depth signal which is the distance measurement result. To do. In the normal mode processing of step S51, the processing is only the processing of steps S11 to S15 in the flowchart of FIG.

In step S52, the control unit 31 determines whether or not a predetermined time has elapsed since the normal mode processing was performed, and the processing of steps S51 and S52 is repeated until the predetermined time elapses, and the depth image is obtained by the normal mode processing. Continues to be generated. Then, if it is determined in step S52 that the predetermined time has elapsed, the process proceeds to step S53.

In step S53, the control unit 31 executes the collision detection process to obtain the integrated value of the predetermined number of times of the TapB pixel signal in the four types of integrated times. The details of the collision detection process will be described later with reference to FIG.

In step S54, the control unit 31 determines whether or not the standard deviation of the predetermined number of the integrated values of the TapB pixel signals at the four types of integration times is less than the predetermined threshold value, and the variation of the integrated values varies. It is determined whether or not modulated light from another light source is included.

In step S54, at least one of the standard deviations of a predetermined number of the integrated values of the TapB pixel signals at the four types of integration times is not less than a predetermined threshold value, and a collision due to modulated light from some other light source occurs. If it is determined, the process proceeds to step S55.

In step S55, the control unit 31 controls the optical modulation unit 21 to change the length of the blank period of the light emitting period of the light emitting diode 22 based on a random number. That is, this process changes the length of the blank period of the light emitting period of the light emitting diode 22 in the normal mode process.

In step S54, it is determined that the standard deviations of the predetermined numbers of the integrated values of the TapB pixel signals at the four types of integration times are all less than the predetermined threshold values, and there is no collision due to the modulated light from other light sources. If so, the process of step S55 is skipped.

In step S56, the control unit 31 determines whether or not the end of the process is instructed, and if the end is not instructed, the process returns to step S51, and the subsequent processes are repeated.

Then, in step S56, when the end of the process is instructed, the process ends.

By the above processing, the presence or absence of collision due to the modulated light from another light source is detected, and when the collision is detected, the length of the blank period between the light emitting periods of the light emitting diode 22 is randomly changed. , Collisions will be avoided.

<Collision detection process>
Next, the collision detection process will be described with reference to the flowchart of FIG.

In step S71, the control unit 31 initializes the counter N for counting the phase shift to 0. This counter N is a counter for distinguishing four types of integrated times.

In step S72, the control unit 31 controls the second tap TB (TapB) of each of the pixels 51 of the TOF sensor 26 to expose within the integration time of the length set corresponding to the counter N. , The pixel signal is detected and output to the calculation unit 29 via the synchronization processing unit 28.

In step S73, the calculation unit 29 integrates the pixel signals, which are the exposure results of TapB, for the integration time set in association with the counter N, obtains and stores the integrated value. Therefore, the processes of steps S72 and S73 are repeated within the integration time.

In step S74, the control unit 31 determines whether or not the counter N is 3, and if N is not 3, the process proceeds to step S75.

In step S75, the control unit 31 increments the counter N by 1, and the process returns to step S72.

That is, in step S74, the processes of steps S72 to S75 are repeated until the counter N is determined to be 3.

Then, when it is determined in step S74 that the counter N is 3, that is, when the integrated values of the pixel signals in the four types of integration periods are obtained, the process proceeds to step S76.

In step S76, the control unit 31 determines whether or not the process has been repeated a predetermined number of times, and if the process has not been repeated a predetermined number of times, the process returns to step S71.

That is, the process in steps S72 and S73 is a process in which the integrated value of the pixel signal detected in the four types of integrated times of different lengths distinguished by the counter N is obtained and stored.

In step S76, when it is determined that a predetermined number of integrated values corresponding to the four types of integrated times have been obtained after being repeated a predetermined number of times, the process proceeds to step S77.

In step S77, the calculation unit 29 obtains the standard deviation of each of the four types of integrated values in the integrated period having different lengths, and outputs the standard deviation to the control unit 31.

Through the above series of processing, TapB obtains the integrated value of the pixel signals at multiple integrated times of different lengths and their standard deviations.

The integrated values of the four types of integrated times detected by TapB have small variations when only background light such as sunlight is included, so that the standard deviation is smaller than a predetermined threshold value obtained in advance. It can be considered that the collision due to the modulated light from another light source has not occurred.

On the other hand, on the other hand, the integrated values of the four types of integrated times detected by TapB have a large variation when modulated light from another light source is included, so that the standard deviation is determined in advance. Since it is larger than the threshold value, it can be considered that a collision due to the modulated light from another light source has occurred.

That is, it is possible to obtain the integrated value of the pixel signals at a plurality of integration times of different lengths for determining the presence or absence of collision and the standard deviation thereof by a series of processes.

As a result, it is possible to determine the collision and avoid the collision by the above-mentioned processing.

In addition, although the explanation is omitted, since the distance measurement process and the depth image generation can be repeated while reducing the frame rate by TapA, the distance measurement can be performed in parallel in TapA even during the collision detection process. This can be achieved, and collision detection can be performed while continuously generating depth images.

<< 4. Third Embodiment >>
<Estimation of modulation frequency of modulated light from other light sources>
In the above, an example of detecting the presence or absence of a collision due to modulated light from another light source and avoiding the collision when the collision is detected has been described, but a collision due to the modulated light from another light source is detected. In this case, the modulation frequency of the modulated light of another light source may be estimated, and the distance may be measured by using another light source based on the estimated modulation frequency of the other modulated light. In this case, by measuring the distance using another light source, the collision between the irradiation light of the own light source and the irradiation light of the other light source is substantially avoided, and the influence of the collision is reduced.

The following relationship holds between the modulation frequency of one's own light source and the modulation frequency of another light source different from its own modulation frequency.

That is, when the Hi signal and the Low signal of the own modulation frequency are set in the same period and each is exposed by TapA and TapB, the period of the difference frequency of the modulation frequency between the own light source and the other light source is integrated. When set to, the relationship that the pixel signals of TapA and TapB are equal is established.

For example, if the modulation frequency of one's own light source is 100MHz and the modulation frequency of another light source is 83.3MHz, the difference frequency is 16.7MHz, and the period of the difference frequency is 60ns (= 1 / (100MHz-). 83.3MHz)).

When 60 ns, which is this cycle, is set as the integrated time (Integ Time), as shown in the upper part of FIG. 14, the period during which light is emitted in the waveform WLo of another light source in the figure is the period H1 to H5.

When the irradiation light from another light source with a modulation frequency of 83.3MHz is received by TapA and TapB so as to correspond to the modulation frequency of 100MHz of its own light source in synchronization with the emission timing of the other light source. The ratio of the pixel signals of TapA and TapB is TapA: TapB = 7: 5, 3: 9, 3: 9, 7: 5, 10: 2 in each of the periods H1 to H5.

Therefore, the pixel signals of TapA and TapB for the entire period of 60ns in the integrated time (IntegTime) are TapA: TapB = 30:30 (= (7 + 3 + 3 + 7 + 10) :( 5 + 9 + 9 +). It becomes 5 + 2)).

Even if the exposure periods of TapA and TapB are shifted by 90 degrees, the ratio of the pixel signals of TapA and TapB is TapA: TapB = 10: 2, 8: 4, 4: 8, in each of the periods H1 to H5. It becomes 2:10 and 6: 6.

Therefore, the pixel signals of TapA and TapB for the entire period of 60ns in the integrated time (IntegTime) are again TapA: TapB = 30: 30 (= (10 + 8 + 4 + 2 + 6) :( 2+). It becomes 4 + 8 + 10 + 6)).

Further, for example, when the modulation frequency of one's own light source is 100 MHz and the modulation frequency of another light source is 90.9 MHz, the difference frequency is 9.1 MHz, and the period of the difference frequency is 110 ns (= 1 / (= 1 / ( It becomes 100MHz-90.9MHz)).

When 110 ns, which is this cycle, is set as the integrated time (Integ Time), as shown in the middle part of FIG. 14, the period during which the waveform WLo of the other light source in the figure emits light is set to the period H11 to H20.

When the irradiation light from another light source with a modulation frequency of 90.9MHz is received by TapA and TapB so as to correspond to the modulation frequency of 100MHz of its own light source in synchronization with the emission timing of the other light source. The ratio of the pixel signals of TapA and TapB is TapA: TapB = 7: 4, 5: 6, 3: 8, 1:10, 2: 9, 4: 7, 6: 5, respectively, in the periods H11 to H20. It becomes 8: 3, 10: 1, 9: 2.

Therefore, the pixel signals of TapA and TapB for the entire period of 110ns in the integrated time (IntegTime) are TapA: TapB = 55:55 (= (7 + 5 + 3 + 1 + 2 + 4 + 6 + 8 + 10+). 9) :( 4 + 6 + 8 + 10 + 9 + 7 + 5 + 3 + 1 + 2)).

However, since measurement in the ns order is not realistic, for example, the measurement results of multiple cycles such as 1000 cycles are integrated and used. As a result, the measurement results in the us order will be used, but the ratio of the pixel signals of TapA and TapB is the same even if the integration time is increased from 60ns or 110ns, for example, 60us or 110us multiplied by 1000.

If the modulation frequency of another light source is 100 MHz, which is the same as the modulation frequency of its own light source, the difference frequency becomes 0 and the period of the difference frequency becomes infinite. Therefore, the ratio of the pixel signals of TapA and TapB in the period of light emission in the waveform WLo of the light source becomes a constant value, and for example, when TapA: TapB = 7: 3 as shown in the lower part of FIG. , The ratio is the same for the entire period.

By using the relationship described with reference to the upper and middle stages of FIG. 14, if the length of the integration time at which the integrated values of the pixel signals of Tap A and Tap B become the same is obtained, the difference frequency from the own modulation frequency can be obtained. It becomes possible to obtain it, and it is possible to obtain the approximate modulation frequency of another light source from its own modulation frequency and the difference frequency.

Also, if the approximate modulation frequency of another light source is obtained, it is possible to estimate the modulation frequency of the other light source.

That is, the difference I (= TapA-TapB) (integrated value) detected by TapA and TapB, where the modulation frequency of itself is 100 MHz and the reflected light of another light source is detected, depends on the difference frequency, for example, in FIG. It is represented as waveforms c1 to c3 consisting of sine waves as shown.

That is, when the difference frequency between the own light source and another light source is 100 kHz, for example, the difference I detected by Tap A and Tap B is expressed as a waveform c1 having a period of 10 us.

Further, when the difference frequency is 90 kHz, the difference I of the difference detected by TapA and TapB is expressed as a waveform c2 having a period of 11.1us.

Further, when the difference frequency is 80 kHz, the difference I of the integrated values detected by TapA and TapB is expressed as a waveform c3 having a period of 12.5us.

Note that the detection results of TapA and TapB are 1000 cycles here, so the actual order is ns order, but here it is expressed in us order.

For example, if the modulation frequency of one's own light source is 100 MHz, the modulation cycle is 10 ns, the modulation frequency of another light source is 99.9 MHz, and the modulation cycle is 10.01 ns, the difference I detected by Tap A and Tap B. Will change at 100kHz.

When the difference frequency is 100 kHz in this way, the difference I of the pixel signals detected by Tap A and Tap B changes periodically in a cycle of 10 us.

Therefore, the difference frequency of 100 kHz or less is specified by sampling the difference I detected by Tap A and Tap B for which the integration time consisting of multiples of 2.5 us divided into four equal parts is set for 10 us, which is the period. ..

For example, as shown by the circle in FIG. 15, the difference between the integrated values detected by TapA and TapB at the integrated times of 2.5us, 5.0us, 7.5us, and 10us, which is a multiple of 2.5us obtained by dividing the period into four equal parts. If the sample of I is plotted on the waveform c1, it is specified as the waveform c1 having a difference frequency of 100 kHz.

Similarly, if the difference I of the integrated values detected by TapA and TapB for the integrated times 2.5us, 5.0us, 7.5us, and 10us consisting of multiples of 2.5us divided into four equal parts is the value on the waveform c2, The difference frequency is specified as the waveform c2 of 90 kHz.

Further, if the difference I of the integrated values detected by TapA and TapB of the integrated times 2.5us, 5.0us, 7.5us, and 10us consisting of multiples of 2.5us divided into four equal parts is the value on the waveform c3, the difference. It is specified as a waveform c3 having a frequency of 80 kHz.

Note that FIG. 15 describes an example of plotting on the waveforms c1 to c3 of the difference frequency, but the integrated values detected by TapA and TapB at the integrated times of 2.5us, 5.0us, 7.5us, and 10us. By specifying the waveform of the corresponding sine wave from the difference I of, it is possible to obtain a difference frequency of 100 kHz or less.

The modulation frequency of another light source is estimated from the modulation frequency and difference frequency of its own light source.

If the modulation frequencies of other light sources are known, it is possible to set the exposure periods of Tap A and Tap B corresponding to the modulation frequencies of other light sources by calibration, and it is possible to realize distance measurement using other light sources. It becomes.

In the case of general phase modulation, when a carrier frequency shift occurs, the IQ coordinates rotate at the frequency that is the difference. Therefore, if the viewpoint is also rotated at the same frequency, it will appear to stop.

In other words, the fact that there is a frequency difference of 100kHz, becomes (100k rotates for one second) that the frequency difference Δθ = θ _{_N + 1} -θ _N corresponds to 100kHz on IQ coordinates shown in Figure 16. The correction of the frequency difference Δθ is realized by an operation for performing coordinate transformation that rotates in the opposite direction.

<Use of other light sources>
When the modulation frequency of another light source is estimated as described above, the exposure period of TapA and TapB is controlled so as to correspond to the modulation frequency of the other light source, so that the other light source is not used. It is possible to realize distance measurement using only the light source of.

That is, the distance and position to the other light source are specified from the depth image obtained by the distance measurement result using the own light source and the depth image obtained by the distance measurement result using the other light source.

If the distance and position to another light source are specified, the distance to the object can be measured by receiving the reflected light generated by the irradiation light from the other light source being reflected by the object.

If the other light source is, for example, a fast blinking ceiling light installed on the ceiling, the modulation frequency of the other light source is estimated, and the own light source and the other light source are used to obtain the other light source. The distance and position can be specified. Then, as long as the positional relationship between its own light source and another light source composed of ceiling lighting does not change, it is possible to continue to realize distance measurement using only the other light source.

For example, consider the case where the depth sensor 11, the objects 71 to 73, and the ceiling lighting 81 are provided in the same space as shown in FIG.

The ceiling illumination 81 emits light with a waveform pattern WLp as shown in the upper part of FIG. Here, with respect to the waveform pattern WLp, the timing at which light is emitted at a modulation frequency composed of a rectangular waveform and the timing at which the light is turned off without a rectangular waveform are alternately set, but even if the timing without a rectangular waveform is straddled. , It is assumed that there is no phase shift in the modulation frequency pattern.

In the case of FIG. 17, the depth sensor 11 measures the distance to the objects 71 to 73 by causing the light emitting diode 22 which is its own light source to emit light at its own modulation frequency. Here, for example, it is assumed that the distance from the depth sensor 11 to the object 71 is 1 m, the distance to the object 72 is 1.5 m, and the distance to the object 73 is 0.5 m.

Since the objects 71 to 73 are imaged by the depth image, the positional relationship between the depth sensor 11 and the objects 71 to 73 is also specified.

Next, the depth sensor 11 estimates the modulation frequency of the ceiling illumination 81, stops the light emission of the light emitting diode 22 which is its own light source, and tapA at a modulation frequency as close as possible to the modulation frequency of the ceiling illumination 81 which is the estimation result. And, by adjusting the exposure period of Tap B, calibration is performed using the distance measurement result when the light emitting diode 22 emits light at its own modulation frequency, and the distance measurement by the ceiling illumination 81 is enabled. Then, the depth sensor 11 adjusts the exposure periods of TapA and TapB at the modulation frequency of the ceiling illumination 81, and the ceiling illumination 81 via the objects 71 to 73 is stopped while the light emitting diode 22 which is its own light source is stopped. Measure the distance to and generate the corresponding depth image.

Here, the distance from the depth sensor 11 to the ceiling illumination 81 via the object 71 is 1.5 m, the distance from the ceiling illumination 81 via the object 72 is 2.0 m, and the distance to the ceiling illumination 81 via the object 73 is It is assumed that the distance is 1.0 m.

The depth sensor 11 obtains the position and distance of the ceiling illumination 81 from the correspondence between the depth image using its own light source and the depth image using the ceiling illumination 81 as the light source.

That is, it can be assumed that the ceiling illumination 81 exists in the spherical surface K1 having a radius of 2 m (= 1.5 m × 2-1.0 m) centered on the object 71.

Further, it can be assumed that the ceiling illumination 81 exists in the spherical surface K2 having a radius of 2.5 m (= 2.0 m × 2-1.5 m) centered on the object 72.

Further, it can be assumed that the ceiling illumination 81 exists in the spherical surface K3 having a radius of 1.5 m (= 1.0 m × 2-0.5 m) centered on the object 73.

Therefore, the ceiling lighting 81 exists at the intersection of the spherical surfaces K1 to K3. Therefore, the depth sensor 11 determines the position of the ceiling illumination 81 by obtaining the intersection of the spherical surfaces K1 to K3. When the light emitting diode 22 which is the self-light source is used, the depth sensor 11 measures the distance from the light emitting diode 22 to the objects 71 to 73 by using 1/2 of the round trip time of the optical path. When the illumination 81 is used, the distance is measured in the time required for the return route from the ceiling illumination 81 to the objects 71 to 73. However, since the depth sensor 11 is calibrated based on the distance measurement result when the light emitting diode 22 which is the self-light source is used, the distance measurement result when the ceiling illumination 81 is used is 1/2 of the actual measurement result. turn into. Therefore, when the ceiling illumination 81 is used as the light source, the depth sensor 11 needs to double the distance measurement result (make it × 2).

If the positional relationship between the depth sensor 11 and the ceiling illumination 81 is required, the depth sensor 11 can realize distance measurement using the ceiling illumination 81 as a light source, and the light emitting diode 22 which is its own light source does not need to emit light. Therefore, it is possible to reduce the power consumption of the light emitting diode 22.

<Distance measurement processing with other light sources>
Next, the distance measuring process using another light source will be described with reference to the flowchart of FIG.

In step S111, the control unit 31 measures the distance to the surrounding objects only by the irradiation light of its own light source by the correction mode processing, and generates a depth image. That is, by this processing, for example, the distance from the objects 71 to 73 in FIG. 17 is measured by the correction mode processing (FIG. 9) when the irradiation light of the own light source is used, and a depth image is generated. To.

In step S112, the control unit 31 executes the modulation frequency estimation process and estimates the modulation frequency of another light source. By this process, for example, the modulation frequency of the irradiation light of another light source such as the ceiling illumination 81 described with reference to FIG. 17 is estimated. The details of the modulation frequency estimation process will be described later with reference to the flowchart of FIG.

In step S113, the control unit 31 irradiates the distance of the same object as the object measured by its own light source by the normal mode processing (FIG. 8) by another light source with another light source such as the ceiling illumination 81 in FIG. The distance is measured by light and a depth image is generated. At this time, the exposure periods of TapA and TapB are set based on the estimation results of the modulation frequencies of other light sources.

In step S114, the calculation unit 29 calculates the position and distance of another light source from the depth signal obtained by distance measurement based on its own light source and the depth signal obtained by distance measurement based on another light source. ..

In step S115, the control unit 31 measures the distance to the object based on the other light source by the normal mode processing by the other light source in consideration of the position and distance of the other light source, and generates a depth image. At this time, the calculation unit 29 considers the distance and position from the depth sensor 11 to another light source, measures the distance to the object by the irradiation light of the other light source, and generates a depth image. Further, since the depth sensor 11 does not require the irradiation light of its own light source, it is possible to turn off the light source of its own, and as a result, it is possible to reduce the power consumption.

In step S116, the control unit 31 determines whether or not the end of the distance measurement process by another light source is instructed, and if the end of the process is not instructed, the process returns to step S115.

That is, in step S116, the same process is repeated until the end of the process is instructed and the end of the normal mode process by another light source is instructed.

Then, in step S116, when the end is instructed, the process ends.

By the above series of processing, the modulation frequency of the other light source is estimated, and the position and distance of the other light source are specified, so that the irradiation light of the other light source is not used without using the irradiation light of the other light source. It is possible to realize distance measurement and generate a depth image only by itself.

As a result, it is possible to realize distance measurement processing and generation of a depth image while turning off its own light source, so that it is possible to reduce the power consumption of the depth sensor 11.

Further, in the above, an example in which the position and distance between the depth sensor 11 and the ceiling illumination 81, which is another light source, is required has been described. However, if the positional relationship between the depth sensor 11 and the ceiling illumination 81 is known in advance, the depth sensor No. 11 can realize distance measurement by using the irradiation light of the ceiling illumination 81 without providing its own light source.

<Modulation frequency estimation process>
Next, the modulation frequency estimation process will be described with reference to the flowchart of FIG. In this process, it is premised that the difference frequency between the modulation frequency of the other light source and the modulation frequency of the own light source is roughly known based on the relationship described with reference to FIG. Therefore, in this process, it is assumed that the period of the difference frequency is roughly obtained as shown in FIG.

In step S151, the control unit 31 initializes the counter N for identifying each integrated time (IntegTime) when the period of the difference frequency is divided into four equal parts.

For example, as described with reference to FIG. 15, when the period of the difference frequency is 10us, it becomes 2.5us when divided into four equal parts, so four types of exposure consisting of 2.5us, 5.0us, 7.5us, and 10us. The time (IntegTime) is set. The counter N is a counter for identifying these four types of integration time. Counter N = 0 indicates the integration time of 2.5us, counter N = 1 indicates the integration time of 5.0us, and counter N = It is assumed that 2 indicates the integration time of 7.5us, and the counter N = 3 indicates the integration time of 10us.

In step S152, the control unit 31 controls the optical modulation unit 21 to stop the light emission of the light emitting diode 22.

In step S153, the control unit 31 controls the first tap TA (TapA) of each of the pixels 51 of the TOF sensor 26 to expose the light emitting diode 22 which is its own light source at a timing synchronized with the light emitting timing. , The pixel signal is detected and output to the calculation unit 29 via the synchronization processing unit 28.

In step S154, the calculation unit 29 integrates and stores the pixel signal which is the exposure result of TapA.

In step S155, the control unit 31 controls the second tap TB (TapB) of each of the pixels 51 of the TOF sensor 26 and exposes the light emitting diode 22 as its own light source at the timing when the light emitting diode 22 is turned off to obtain a pixel signal. Is detected and output to the calculation unit 29 via the synchronization processing unit 28.

In step S156, the calculation unit 29 integrates and stores the pixel signal which is the exposure result of TapB.

In step S157, the calculation unit 29 determines whether or not the integration time corresponding to the counter N has elapsed, and if the integration time corresponding to the counter N has not elapsed, the process returns to step S153. That is, the pixel signals, which are the exposure results by Tap A and Tap B, are sequentially integrated until the integration time corresponding to the counter N elapses.

If it is determined in step S157 that the exposure time corresponding to the counter N has elapsed, the process proceeds to step S158.

In step S158, the calculation unit 29 calculates the difference I between the pixel signal integration result, which is the stored exposure result of TapA, and the pixel signal integration result, which is the exposure result of TapB, and integrates corresponding to the counter N. Store as the time sampling result.

In step S159, the control unit 31 determines whether or not the counter N is 3, that is, whether or not the difference I of all exposure times has been sampled.

If the counter N is not 3 in step S159, the process proceeds to step S160.

In step S160, the control unit 31 increments the counter N by 1, and the process returns to step S153. That is, by repeating the processes of steps S153 to S159, the difference I between the pixel signal integration result, which is the exposure result of TapA, and the pixel signal integration result, which is the exposure result of TapB, is associated with the counter N. The exposure time is sequentially calculated and repeated until the difference I of the four types of integration time is sampled.

Then, in step S159, it is determined that the difference between the pixel signal integration result, which is the exposure result of TapA, and the pixel signal integration result, which is the exposure result of TapB, has been calculated and sampled for all four types of integration times. If, that is, if the counter N is 3, the process proceeds to step S160.

In step S160, the calculation unit 29 uses the difference I between the pixel signal integration result, which is the exposure result of TapA, and the pixel signal integration result, which is the exposure result of TapB, sampled at four types of integration times. , Estimate the modulation frequencies of other light sources.

That is, as described with reference to FIG. 15, the calculation unit 29 uses the integration result of the pixel signal, which is the exposure result of TapA, and the exposure result of TapB, which are sampled at four types of integration times (IntegTime). The modulation frequency of another light source is estimated using the difference I from the integration result of a certain pixel signal.

That is, the calculation unit 29 estimates the waveform of the difference frequency between the modulation frequency of its own light source and the modulation frequency of another light source by using the sampling result of the difference I.

Further, the calculation unit 29 estimates the modulation frequency of another light source based on the estimated difference frequency and the modulation frequency of its own light source.

By the above processing, it is possible to estimate the modulation frequency of the irradiation light of another light source.

As a result, by controlling the integration time of TapA and TapB based on the estimated modulation frequency of the other light source, the measurement using only the irradiation light of the other light source without using the irradiation light of its own light source. It is possible to realize the generation of distance and depth images.

In the above, an example of sampling the difference I by the integration time (IntegTime) obtained by dividing the period of the difference frequency from the modulation frequency of another light source into four equal parts has been described, but the period of the difference frequency is set to 4 or more. It may be divided and the number of samples of the difference I may be increased. By increasing the number of samplings in this way, it is possible to estimate the modulation frequencies of other light sources with high accuracy.

Further, the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

<< 5. Application example to moving body >>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on a moving body of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 20 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving body control system to which the technique according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 20, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating braking force of the vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives inputs of these radio waves or signals and controls a vehicle door lock device, a power window device, a lamp, and the like.

The vehicle outside information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or characters on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects the in-vehicle information. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (Advanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040. It is possible to perform coordinated control for the purpose of automatic driving that runs autonomously without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio image output unit 12052 transmits the output signal of at least one of the audio and the image to the output device capable of visually or audibly notifying the passenger of the vehicle or the outside of the vehicle. In the example of FIG. 20, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 21 is a diagram showing an example of the installation position of the imaging unit 12031.

In FIG. 21, the vehicle 12100 has

imaging units

12101, 12102, 12103, 12104, 12105 as imaging units 12031.

The

imaging units

12101, 12102, 12103, 12104, 12105 are provided at positions such as, for example, the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The images in front acquired by the

imaging units

12101 and 12105 are mainly used for detecting the preceding vehicle, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like.

Note that FIG. 21 shows an example of the photographing range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the

imaging units

12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining, it is possible to extract as the preceding vehicle a three-dimensional object that is the closest three-dimensional object on the traveling path of the vehicle 12100 and that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, 0 km / h or more). it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform coordinated control for the purpose of automatic driving or the like in which the vehicle runs autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that can be seen by the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

The above is an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to, for example, the imaging unit 12031 among the configurations described above. Specifically, the TOF sensor 26 of FIG. 6 can be applied to the imaging unit 12031. By applying the technique according to the present disclosure to the imaging unit 12031, it is possible to measure the distance with high accuracy even in an environment where a plurality of distance measuring devices exist.

The present disclosure may also have the following configuration.
<1> A light receiving unit that receives the reflected light generated by the reflection of the irradiation light by an object in the detection area, and a light receiving portion.
An arithmetic unit that calculates the distance to the object based on the phase difference between the irradiation light and the reflected light.
Suppresses the influence of collision between the required irradiation light, which is the irradiation light used for calculating the distance to the object, and the unnecessary irradiation light, which is the irradiation light not used for calculating the distance to the object, in the light receiving unit. As described above, a distance measuring device including the light receiving unit and the control unit that controls the calculation unit.
<2> The distance measuring device according to <1>, wherein the required irradiation light used for calculating the distance to the object is the irradiation light for which the information required for calculating the distance to the object is known.
<3> The distance measuring device according to <2>, wherein the information required for calculating the distance to the object is information on the modulation frequency and phase of the irradiation light.
<4> An irradiation unit that irradiates the detection region with the pulsed irradiation light as the necessary irradiation light is further included.
The control unit
The light receiving unit is made to receive the required reflected light, which is the reflected light generated when the required irradiation light irradiated by the irradiation unit is reflected by an object in the detection region.
The distance measuring device according to <1>, wherein the calculation unit is controlled to calculate the distance to the object based on the phase difference between the required irradiation light and the required reflected light irradiated by the irradiation unit.
<5> The control unit controls the light receiving unit to receive the necessary reflected light required for distance measurement with a part of the pixels and to receive the unnecessary irradiation light with the pixels other than the part. 4> The ranging device.
<6> The control unit
The irradiation unit is operated by setting a period for irradiating the required irradiation light and a period for turning off the light.
A part of the pixel is received by the light receiving unit at a phase corresponding to the timing when the required irradiation light is irradiated by the irradiation unit, and the irradiation unit is turned off by pixels other than the part. Receive light in the phase corresponding to the timing of the light
The pixel signal generated by receiving the required reflected light in the phase corresponding to the timing when the irradiation unit irradiates the required irradiation light to the calculation unit corresponds to the timing when the irradiation unit is turned off. The method according to <5>, wherein the distance to the object is calculated by the phase difference between the required irradiation light and the required reflected light based on the difference from the pixel signal generated by receiving the light in phase. Distance measuring device.
<7> The control unit
The integrated value of the pixel signal generated by receiving the required reflected light in the phase corresponding to the timing when the irradiation unit irradiates the calculation unit with the required irradiation light, and the timing at which the irradiation unit is turned off. The distance measuring device according to <6>, which controls to calculate the distance to the object based on the difference from the integrated value of the pixel signal generated by receiving light in the phase corresponding to.
<8> The control unit
A part of the pixel receives the required reflected light on the light receiving unit at a phase corresponding to the timing when the required irradiation light is irradiated by the irradiation unit, and the phase corresponds to the timing when the irradiation unit is turned off. So, control so that the light is received at multiple integration times,
The calculation unit is made to determine the presence or absence of the collision based on the integrated value of the pixel signals generated by receiving light at the plurality of integrated times.
A period for irradiating the required irradiation light to the irradiation unit and a period for turning off the light are set, and when it is determined that the collision has occurred, the length of the period for turning off the light is changed. The ranging device according to <4>.
<9> The measurement according to <8>, wherein the control unit controls the irradiation unit to randomly change the length of the extinguishing period when it is determined that the collision has occurred. Distance device.
<10> The calculation unit determines the presence or absence of the collision based on a comparison between the standard deviation of the integrated value of the pixel signal generated by receiving light at the plurality of integrated times and a predetermined threshold value. 8> The distance measuring device.
<11> The control unit
A part of the pixel and a part other than the part of the pixel are alternately exposed to the light receiving unit for half a cycle of the modulation cycle of the modulation frequency of the required irradiation light in the irradiation unit.
The calculation unit is made to estimate the modulation frequency of another light source different from the irradiation unit based on the integrated values of the pixel signals detected by the light receiving unit at a plurality of integration times.
A part of the pixel and a part other than the part of the pixel are alternately exposed to the light receiving portion for half a cycle of the modulation cycle of the modulation frequency of the other light source estimated.
The distance measuring device according to <4>, wherein the calculation unit is controlled to calculate the distance to an object by the phase difference between the irradiation light of the other light source and the reflected light of the irradiation light of the other light source.
<12> The control unit detects the integrated value of the pixel signal detected by a part of the pixels in the light receiving unit and the integrated value of the pixel signal other than a part of the pixels in the light receiving unit during the plurality of integration times. The distance measuring device according to <11>, which controls to estimate the modulation frequency of the other light source based on the difference from the integrated value of the pixel signal.
<13> The control unit detects the integrated value of the pixel signal detected by a part of the pixels in the light receiving unit and the integrated value of the pixel signal other than a part of the pixels in the light receiving unit during the plurality of integration times. Based on the difference from the integrated value of the pixel signal, the difference frequency between the modulation frequency of the other light source and the modulation frequency of the required irradiation light irradiated by the irradiation unit is estimated, and based on the difference frequency. The distance measuring device according to <12>, which controls the modulation frequency of the other light source to be estimated.
<14> The distance measuring device according to <11>, wherein the plurality of integrated times are multiples of a time obtained by dividing the modulation period of the modulation frequency of the other light source into a plurality of modulation frequencies.
<15> Further includes a depth image generation unit that generates a depth image based on the information of the distance to the object calculated by the calculation unit.
The control unit
In the depth image generation unit,
The required reflected light corresponding to the required irradiation light of the irradiation unit, which is controlled based on the modulation frequency of the required irradiation light emitted by the irradiation unit, is received by the light receiving unit and is detected by the pixel signal. A first depth image based on the calculated distance information to the object,
The irradiation light of the other light source, which is controlled based on the modulation frequency of the other light source, is calculated by the pixel signal detected when the required reflected light corresponding to the required irradiation light is received by the light receiving unit. A second depth image based on the information on the distance to the object is generated.
Turn off the irradiation part
In the calculation unit
Based on the first depth image and the second depth image, the position and distance of the other light source are calculated.
The phase difference between the irradiation light emitted as the necessary irradiation light from the other light source and the reflected light in which the irradiation light emitted as the necessary irradiation light from the other light source is reflected by the object as the necessary reflected light, and The distance measuring device according to <11>, wherein the distance to the object is calculated based on the position and distance of the other light source.
<16> The distance measuring device according to any one of <1> to <15>, wherein the light receiving unit is a TOF (Time of Flight) sensor.
<17> A light receiving process that receives the reflected light generated by the reflection of the irradiation light by an object in the detection area.
Arithmetic processing that calculates the distance to the object by the phase difference between the irradiation light and the reflected light,
In the light receiving process, the influence of collision between the required irradiation light, which is the irradiation light used for calculating the distance to the object, and the unnecessary irradiation light, which is the irradiation light not used for calculating the distance to the object, is suppressed. As described above, a distance measuring method including a control process for controlling the light receiving process and the arithmetic process.

11 depth sensor, 21 optical modulator, 22 light emitting diode, 23 floodlight lens, 24 light receiving lens, 25 filter, 26 TOF sensor, 27 image storage unit, 28 synchronization processing unit, 29 calculation unit, 30 depth image generation unit, 31 control Unit, 40 pixel array unit, 41 tap drive unit, 42 vertical drive unit, 47 vertical signal line, 48 voltage supply line, 51 pixels

Claims

A light receiving part that receives the reflected light generated by the reflection of the irradiation light by an object in the detection area,
An arithmetic unit that calculates the distance to the object based on the phase difference between the irradiation light and the reflected light.
Suppresses the influence of collision between the required irradiation light, which is the irradiation light used for calculating the distance to the object, and the unnecessary irradiation light, which is the irradiation light not used for calculating the distance to the object, in the light receiving unit. As described above, a distance measuring device including the light receiving unit and the control unit that controls the calculation unit.
The distance measuring device according to claim 1, wherein the required irradiation light used for calculating the distance to the object is the irradiation light for which the information required for calculating the distance to the object is known.
The distance measuring device according to claim 2, wherein the information required for calculating the distance to the object is information on the modulation frequency and phase of the irradiation light.
An irradiation unit that irradiates the detection region with the pulsed irradiation light as the necessary irradiation light is further included.
The control unit
The light receiving unit is made to receive the required reflected light, which is the reflected light generated when the required irradiation light irradiated by the irradiation unit is reflected by an object in the detection region.
The distance measuring device according to claim 1, wherein the calculation unit is controlled to calculate the distance to the object by the phase difference between the required irradiation light irradiated by the irradiation unit and the required reflected light.
According to claim 4, the control unit controls the light receiving unit to receive the necessary reflected light required for distance measurement with a part of the pixels and to receive the unnecessary irradiation light with the pixels other than the part. The distance measuring device described.
The control unit
The irradiation unit is operated by setting a period for irradiating the required irradiation light and a period for turning off the light.
A part of the pixel is received by the light receiving unit at a phase corresponding to the timing when the required irradiation light is irradiated by the irradiation unit, and the irradiation unit is turned off by pixels other than the part. Receive light in the phase corresponding to the timing of the light
The pixel signal generated by receiving the required reflected light in the phase corresponding to the timing when the irradiation unit irradiates the required irradiation light to the calculation unit corresponds to the timing when the irradiation unit is turned off. The fifth aspect of claim 5, wherein the distance to the object is controlled to be calculated by the phase difference between the required irradiation light and the required reflected light based on the difference from the pixel signal generated by receiving light in phase. Distance measuring device.
The control unit
The integrated value of the pixel signal generated by receiving the required reflected light in the phase corresponding to the timing at which the irradiation unit irradiates the calculation unit with the required irradiation light, and the timing at which the irradiation unit is turned off. The distance measuring device according to claim 6, wherein the distance to the object is controlled to be calculated based on the difference from the integrated value of the pixel signal generated by receiving light in the phase corresponding to the above.
The control unit
A part of the pixel receives the required reflected light on the light receiving unit at a phase corresponding to the timing when the required irradiation light is irradiated by the irradiation unit, and the phase corresponds to the timing when the irradiation unit is turned off. So, control so that the light is received at multiple integration times,
The calculation unit is made to determine the presence or absence of the collision based on the integrated value of the pixel signals generated by receiving light at the plurality of integrated times.
A period for irradiating the required irradiation light to the irradiation unit and a period for turning off the light are set, and when it is determined that the collision has occurred, the length of the period for turning off the light is changed. The distance measuring device according to claim 4.
The distance measuring device according to claim 8, wherein the control unit controls the irradiation unit to randomly change the length of the extinguishing period when it is determined that the collision has occurred.
According to claim 8, the calculation unit determines the presence or absence of the collision based on a comparison between the standard deviation of the integrated value of the pixel signal generated by receiving light at the plurality of integrated times and a predetermined threshold value. The described ranging device.
The control unit
A part of the pixel and a part other than the part of the pixel are alternately exposed to the light receiving unit for half a cycle of the modulation cycle of the modulation frequency of the required irradiation light in the irradiation unit.
The calculation unit is made to estimate the modulation frequency of another light source different from the irradiation unit based on the integrated values of the pixel signals detected by the light receiving unit at a plurality of integration times.
A part of the pixel and a part other than the part of the pixel are alternately exposed to the light receiving portion for half a cycle of the modulation cycle of the modulation frequency of the other light source estimated.
The distance measuring device according to claim 4, wherein the calculation unit is controlled to calculate the distance to an object by the phase difference between the irradiation light of the other light source and the reflected light of the irradiation light of the other light source.
The control unit tells the calculation unit that the integrated value of the pixel signal detected by a part of the pixels in the light receiving unit and the pixels detected by other than a part of the pixels in the light receiving unit during the plurality of integration times. The distance measuring device according to claim 11, wherein the modulation frequency of the other light source is controlled to be estimated based on the difference from the integrated value of the signal.
The control unit tells the calculation unit that the integrated value of the pixel signal detected by a part of the pixels in the light receiving unit and the pixels detected by other than a part of the pixels in the light receiving unit during the plurality of integration times. The difference frequency between the modulation frequency of the other light source and the modulation frequency of the required irradiation light irradiated by the irradiation unit is estimated based on the difference from the integrated value of the signal, and the difference frequency is estimated based on the difference frequency. The distance measuring device according to claim 12, wherein the modulation frequency of another light source is controlled to be estimated.
The distance measuring device according to claim 11, wherein the plurality of integrated times are multiples of a time obtained by dividing the modulation period of the modulation frequency of the other light sources into a plurality of modulation cycles.
Further including a depth image generation unit that generates a depth image based on the information of the distance to the object calculated by the calculation unit.
The control unit
In the depth image generation unit,
The required reflected light corresponding to the required irradiation light of the irradiation unit, which is controlled based on the modulation frequency of the required irradiation light emitted by the irradiation unit, is received by the light receiving unit and is detected by the pixel signal. A first depth image based on the calculated distance information to the object,
The irradiation light of the other light source, which is controlled based on the modulation frequency of the other light source, is calculated by the pixel signal detected when the required reflected light corresponding to the required irradiation light is received by the light receiving unit. A second depth image based on the information on the distance to the object is generated.
Turn off the irradiation part
In the calculation unit
Based on the first depth image and the second depth image, the position and distance of the other light source are calculated.
The phase difference between the irradiation light emitted as the necessary irradiation light from the other light source and the reflected light in which the irradiation light emitted as the necessary irradiation light from the other light source is reflected by the object as the necessary reflected light, and The distance measuring device according to claim 11, wherein the distance to the object is calculated based on the position and distance of the other light source.
The distance measuring device according to claim 1, wherein the light receiving unit is a TOF (Time of Flight) sensor.
Light receiving processing that receives the reflected light generated by the reflection of the irradiation light by the object in the detection area,
Arithmetic processing that calculates the distance to the object by the phase difference between the irradiation light and the reflected light,
In the light receiving process, the influence of collision between the required irradiation light, which is the irradiation light used for calculating the distance to the object, and the unnecessary irradiation light, which is the irradiation light not used for calculating the distance to the object, is suppressed. As described above, a distance measuring method including a control process for controlling the light receiving process and the arithmetic process.