WO2021006048A1 - Signal processing device and signal processing method - Google Patents

Abstract

The present invention relates to a signal processing device and signal processing method, which make it possible to obtain impulse response data regarding the temporal direction of reflected light in a simpler manner. This signal processing device comprises: a training data generation unit in which, on the basis of received-light data outputted in a second drive mode by a distance measurement sensor which, in either a first drive mode or the second drive mode, receives reflected light, which is radiated light that has reflected off a subject and returned, impulse response data regarding the temporal direction of the light reflected off the subject is generated as training data; and a learning unit which calculates a model parameter for a learning model that learns a relationship between received-light data in the first drive mode and impulse response data regarding the temporal direction of the reflected light. The present invention can be applied to a distance measurement system, for example.

Description

Signal processing device and signal processing method

The present technology relates to a signal processing device and a signal processing method, and more particularly to a signal processing device and a signal processing method that enable more easily to obtain impulse response data in the time direction of reflected light.

In recent years, some mobile terminals such as smartphones are equipped with a ranging module. As a distance measuring method in the distance measuring module, for example, there is an Indirect ToF (Time of Flight) method. In the IndirectToF method, the reflected light that is reflected on the surface of the object by irradiating the light toward the object is detected. Then, the distance to the object is calculated based on the measured value obtained by measuring the flight time of the reflected light.

Exposure control in the Indirect ToF sensor is realized, for example, by using an image sensor provided with two charge storage units per pixel and distributing the photoelectrically converted charges to the two charge storage units at high speed.

On the other hand, Patent Document 1 and Non-Patent Document 1 propose a sensor having four or eight charge storage units per pixel and acquiring an impulse response of reflected light in the time direction of several tens of ns.

Non-Patent Documents 2 to 4 propose a technique of inputting the output of the IndirectToF sensor to the neural network, reducing the depth estimation error mainly due to multiple reflections, and outputting accurate depth information.

JP-A-2018-33008

In the techniques of Patent Document 1 and Non-Patent Document 1, in order to obtain an impulse response in the time direction of reflected light, information is acquired in time division while changing many parameters, so it takes a long time just to take a picture. , The user cannot perform simple measurement. Moreover, since it can be applied only to a completely stationary object, it is not suitable for use in a general environment or consumer use.

The techniques of Non-Patent Documents 2 to 4 have the effect of reducing the depth estimation error due to multiple reflections, but since the depth map is an output, it is not possible to obtain an impulse response in the time direction of the reflected light.

This technology was made in view of such a situation, and makes it possible to more easily obtain the impulse response data in the time direction of the reflected light.

In the signal processing device on one aspect of the present technology, the ranging sensor that receives the reflected light reflected by the subject in the first drive mode or the second drive mode is in the second drive mode. A teacher data generation unit that generates impulse response data of the reflected light with respect to the subject in the time direction as teacher data from the output received data, and an impulse response of the received data in the first drive mode and the reflected light in the time direction. It is provided with a learning unit that calculates model parameters of a learning model that learns the relationship with data.

In the signal processing method of one aspect of the present technology, the distance measuring sensor in which the signal processing device receives the reflected light reflected by the subject in the first drive mode or the second drive mode is the first. From the received light data output in the second drive mode, impulse response data in the time direction of the reflected light to the subject is generated as teacher data, and the received light data in the first drive mode and the impulse response in the time direction of the reflected light are generated. Calculate the model parameters of the learning model that learns the relationship with the data.

In one aspect of the present technology, the light receiving received by the distance measuring sensor that receives the reflected light reflected by the subject in the first drive mode or the second drive mode is output in the second drive mode. From the data, the impulse response data of the reflected light with respect to the subject in the time direction is generated as teacher data, and learning to learn the relationship between the received light data in the first drive mode and the impulse response data of the reflected light in the time direction. The model parameters of the model are calculated.

The signal processing device of one aspect of the present technology can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting through a transmission medium or by recording on a recording medium.

The signal processing device may be an independent device or an internal block constituting one device.

It is a block diagram which shows the structural example of one Embodiment of the distance measuring system to which this technique is applied. It is a figure explaining the impulse response data in the time direction. It is a figure explaining the impulse response data in the time direction. It is a block diagram which shows the detailed configuration example of a distance measurement module. It is a figure explaining the operation of a distance measuring sensor. It is a figure explaining the operation of a distance measuring sensor. It is a figure explaining the operation of a distance measuring sensor. It is a figure explaining the operation of a distance measuring sensor. It is a figure explaining the operation of a distance measuring sensor. It is a figure explaining the operation of a distance measuring sensor. It is a block diagram which shows the detailed structure example of a teacher data generation part. It is a block diagram which shows the detailed structure example of a learning part. It is a flowchart explaining the learning process which learns the prediction model which predicts the impulse response data in the time direction. It is a flowchart explaining the prediction process of a distance measurement system. It is a figure explaining the modification of the distance measurement system. It is a block diagram which shows the 1st application example of a distance measuring system. It is a figure explaining the 1st application example of a distance measuring system. It is a block diagram which shows the 2nd application example of the distance measuring system. It is a figure explaining the 2nd application example of the distance measuring system. It is a block diagram which shows the 3rd application example of a distance measuring system. It is a figure explaining the 3rd application example of the distance measuring system. It is a figure explaining the 3rd application example of the distance measuring system. It is a flowchart explaining the AR display processing of the 3rd application example. It is a figure explaining the application example of the movement control of an autonomous mobile robot. It is a block diagram which shows the structural example of one Embodiment of the computer to which this technique is applied.

Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described. The explanation will be given in the following order.
1. 1. Configuration example of distance measurement system 2. Outline of processing of distance measurement system 3. Detailed configuration example of the ranging module 4. Detailed configuration example of the teacher data generation unit 5. Detailed configuration example of the learning unit 6. Distance measurement system processing 7. Modification example 8. Application example 9. Computer application example

<1. Configuration example of distance measurement system>
FIG. 1 is a block diagram showing a configuration example of an embodiment of a distance measuring system to which the present technology is applied.

The distance measuring system 1 of FIG. 1 includes a distance measuring module 11, a teacher data generation unit 12, a learning unit 13, an estimation unit 14, and a control unit 15.

The distance measuring module 11 is an Indirect ToF type distance measuring module that outputs light receiving data corresponding to distance information to a subject by receiving the reflected light that is reflected by a predetermined object as a subject and returned. Is. The received light data output by the ranging module 11 is also referred to as raw data.

The ranging module 11 has a first drive mode and a second drive mode as drive modes, operates in the drive mode specified by the control unit 15, and outputs Raw data. The first drive mode is the light receiving timing in which the phase is shifted by 0 °, 90 °, 180 °, or 270 ° based on the irradiation timing of the irradiation light, as in the case of the general Indirect ToF type ranging module. This is a drive mode for receiving reflected light, and is also referred to as a normal drive mode below. The second drive mode is a drive mode in which the reflected light is received at a light reception timing in which the phase is shifted by 1 ° from 0 ° to 179 °, for example, with reference to the irradiation timing of the irradiation light. Refer to.

When the distance measuring module 11 operates in the first drive mode (normal drive mode), the distance measuring module 11 receives the reflected light and supplies the raw data generated to the learning unit 13 or the estimation unit 14. On the other hand, when the distance measuring module 11 operates in the second drive mode (special drive mode), the distance measuring module 11 receives the reflected light and supplies the raw data generated to the teacher data generation unit 12.

The teacher data generation unit 12 acquires the raw data of the second drive mode from the distance measuring module 11, and from the acquired raw data of the second drive mode, the teacher data is the impulse response data of the reflected light to the subject in the time direction. Is generated and supplied to the learning unit 13.

The learning unit 13 acquires the light receiving data in the first drive mode supplied from the distance measuring module 11 and the impulse response data in the time direction of the reflected light supplied from the teacher data generation unit 12, and the first unit. Learn the relationship between the received light data in the drive mode and the impulse response data in the time direction of the reflected light. More specifically, the learning unit 13 has, for example, a learner using a convolutional neural network (hereinafter referred to as CNN (Convolutional Neural Network)) as a learning model, and the teacher data is stored in the CNN as a learning model. Is the time-direction impulse response data of the reflected light obtained in the second drive mode, the student data is the light-receiving data in the first drive mode, and the light-receiving data in the first drive mode is input. Calculate the CNN parameters (model parameters) that predict the data. The calculated model parameters are supplied to the estimation unit 14.

The estimation unit 14 includes a predictor that executes prediction processing using the model parameters obtained by the learning device by the learning unit 13. That is, the estimation unit 14 uses the learning model (CNN) using the model parameters calculated by the learning unit 13 as a prediction model, and predicts the light receiving data of the first drive mode supplied from the distance measuring module 11. Input to the model and output the impulse response data in the time direction of the reflected light as the prediction result.

In this embodiment, CNN is adopted as a model used for the learner and the predictor of machine learning, but other neural networks may be adopted, or other algorithms may be adopted. .. That is, machine learning models and algorithms are not limited.

The control unit 15 is, for example, an operation control signal based on the user's operation from the operation unit of the device in which the distance measurement system 1 is incorporated, or an operation control signal from a higher control unit of the device in which the distance measurement system 1 is incorporated. The distance measurement module 11, the teacher data generation unit 12, the learning unit 13, and the estimation unit 14 are controlled based on the above.

Specifically, in the learning mode in which the learning unit 13 learns the model parameters, the control unit 15 sequentially drives the distance measuring module 11 in the first drive mode and the second drive mode, and sequentially drives the raw data into the teacher data. Output to the generation unit 12 or the learning unit 13. Further, in the prediction mode after the model parameters are learned, the control unit 15 drives the ranging module 11 in the first drive mode and outputs the raw data to the estimation unit 14.

The teacher data generation unit 12, the learning unit 13, the estimation unit 14, and the control unit 15 in the latter stage of processing the raw data output from the distance measuring module 11 are DSP (Digital Signal Processor) and ISP (Image Signal Processor). It can be configured by one signal processing device 16 such as. Of course, each of the teacher data generation unit 12, the learning unit 13, the estimation unit 14, and the control unit 15 may be configured by individual DSPs or ISPs.

<2. Outline of processing of distance measurement system ＞
The impulse response data in the time direction of the reflected light output by the ranging system 1 will be described with reference to FIGS. 2 and 3.

In the case of a general Indirect ToF type ToF sensor, for example, in the case of the 4Phase method, the reflected light is shifted by 0 °, 90 °, 180 °, and 270 ° based on the irradiation timing of the irradiation light. It receives light and outputs the resulting Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ . Using the raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ , the distance to the object can be calculated by performing a predetermined operation described later.

By the way, the reflected light received by the ToF sensor includes the reflected light that is directly irradiated and reflected on the object to be measured as shown by the solid line in FIG. 2, and the reflected light that is reflected by the object as shown in FIG. This includes the reflected light that is once irradiated to the object to be measured and then irradiated to the object to be measured, the reflected light that is irradiated to the object to be measured, and the light that is further reflected by another object. Therefore, if the resolution in the time direction is increased, the received light data of the profile that momentarily receives light at a predetermined timing as shown in the waveform of FIG. 2 can be obtained.

The received light data of the profile in FIG. 2 is impulse response data in the time direction of the reflected light obtained by using impulse light emission with a very short emission time and shortening the accumulation time of the ToF sensor. Such impulse response data in the time direction of the reflected light can be obtained by using impulse light emission with a very short emission time and shortening the accumulation time of the ToF sensor, but in an actual usage environment, the impulse response data can be obtained. It is practically difficult to perform such a drive.

Therefore, as shown in FIG. 3, the estimation unit 14 of the distance measuring system 1 inputs the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ output by the ToF sensor, and the time of the reflected light is reflected. Predicts the impulse response data in the direction and outputs it as the prediction result. The impulse response data in the time direction of the reflected light is data represented in a total of three dimensions, two dimensions in the spatial direction corresponding to the pixel array of the ToF sensor and one dimension in the time direction.

Of course, the estimation unit 14 outputs not only the impulse response data of the reflected light in the time direction but also the four-phase raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ like a general ToF sensor. can do.

In the learning unit 13, the estimation unit 14 learns the model parameters of the CNN as a predictor for predicting the impulse response data in the time direction of the reflected light.

The teacher data generation unit 12 generates impulse response data in the time direction of the reflected light, which is the teacher data for the learning unit 13 to learn.

<3. Detailed configuration example of ranging module>
FIG. 4 is a block diagram showing a detailed configuration example of the ranging module 11.

The distance measuring module 11 includes a light emitting source 21, a light emitting control unit 22, a delay generation unit 23, and a distance measuring sensor 24.

The light emitting source 21 has, for example, an infrared laser diode as a light source, emits light at a timing corresponding to a light emitting control signal supplied from the light emitting control unit 22, and irradiates an object with irradiation light. The light emission control signal is composed of, for example, a rectangular wave pulse signal that becomes high in a predetermined period, an impulse signal that momentarily becomes high for a short time in a predetermined period, and the like.

The light emission control unit 22 controls the light emission source 21 by supplying a light emission control signal of a predetermined frequency (for example, 20 MHz or the like) to the light emission source 21. Further, the light emission control unit 22 also supplies a light emission control signal to the delay generation unit 23 in order to drive the distance measuring sensor 24 in accordance with the timing of light emission in the light emission source 21.

The delay generation unit 23 generates an exposure control signal delayed by a predetermined delay amount with respect to the timing (irradiation timing) indicated by the light emission control signal from the light emission control unit 22, and supplies the exposure control signal to the ranging sensor 24. The delay amount is supplied from the storage unit 61 (FIG. 11) of the teacher data generation unit 12.

The distance measuring sensor 24 receives the reflected light from the object based on the exposure control signal supplied from the delay generation unit 23. When the exposure control signal supplied from the delay generation unit 23 is delayed by a predetermined phase difference (delay amount) from the light emission control signal supplied to the light emission source 21, the distance measuring sensor 24 uses the light emission source 21. The light is received at a timing shifted by a predetermined phase difference from the irradiation timing.

The distance measuring sensor 24 is an Indirect ToF type ToF sensor. The distance measuring sensor 24 has a normal drive mode (first drive mode) and a special drive mode (second drive mode) as operation modes, and is in the drive mode designated by the control unit 15 (FIG. 1). It works and outputs Raw data. The normal drive mode and the special drive mode drive the distance measuring sensor 24 in the same manner, but differ in the exposure control signal. Specifically, the exposure control signal supplied from the delay generation unit 23 in the normal drive mode has a phase of 0 °, 90 °, 180 °, or a phase with respect to the light emission control signal output by the light emission control unit 22. It is a signal shifted by 270 °. On the other hand, the exposure control signal supplied from the delay generation unit 23 in the special drive mode has a phase of only one of 1 ° units from 0 ° to 179 ° with respect to the light emission control signal output by the light emission control unit 22. It is a shifted signal.

When the normal drive mode is specified as the drive mode, the ranging sensor 24 emits the reflected light at a light receiving timing that is out of phase with the irradiation timing of the irradiation light by 0 °, 90 °, 180 °, or 270 °. The light is received, and the raw data of the resulting phase differences of 0 °, 90 °, 180 °, and 270 ° are supplied to the learning unit 13 or the estimation unit 14.

When the special drive mode is specified as the drive mode, the distance measuring sensor 24 receives the reflected light at the light receiving timing in which the phase is changed in 1 ° units from 0 ° to 179 ° with respect to the irradiation timing of the irradiation light. The raw data having a phase difference of 0 ° to 179 ° as a result is supplied to the storage unit 61 of the teacher data generation unit 12.

Considering that the raw data of the predetermined phase difference output by the distance measuring sensor 24 is an image in which the brightness value of the reflected light is stored as a pixel value, the distance measuring sensor 24 has a phase of 0 ° in the normal drive mode. It outputs 90 °, 180 °, and 270 ° 4-phase (4 types) raw images. On the other hand, in the special drive mode, the ranging sensor 24 outputs 180 phases (180 types) of raw images having a phase of 0 ° to 179 °.

<Details of ranging sensor>
Next, the details of the distance measuring sensor 24 will be described with reference to FIGS. 5 to 10.

As shown in FIG. 5, the distance measuring sensor 24 has a pixel array unit 32 in which pixels 31 are two-dimensionally arranged in a matrix in the row direction and a column direction, and a drive control in which the pixels 31 are arranged in a peripheral region of the pixel array unit 32. It has a circuit 33. The pixel 31 generates an electric charge according to the amount of reflected light received, and outputs a signal corresponding to the electric charge.

The drive control circuit 33 is, for example, a control signal for controlling the drive of the pixel 31 based on the exposure control signal supplied from the delay generation unit 23 (for example, the distribution signal DIMIX described later, the selection signal ADDRESS DECODE, and the reset). (Signal RST, etc.) is output.

As shown in FIG. 5, the pixel 31 has a photodiode 51 and a first tap 52A and a second tap 52B that detect the charge photoelectrically converted by the photodiode 51. In the pixel 31, the electric charge generated by one photodiode 51 is distributed to the first tap 52A or the second tap 52B. Then, of the charges generated by the photodiode 51, the charges distributed to the first tap 52A are output as a detection signal A from the signal line 53A, and the charges distributed to the second tap 52B are detected signals B from the signal line 53B. Is output as.

The first tap 52A is composed of a transfer transistor 41A, an FD (Floating Diffusion) unit 42A, a selection transistor 43A, and a reset transistor 44A. Similarly, the second tap 52B is composed of a transfer transistor 41B, an FD unit 42B, a selection transistor 43B, and a reset transistor 44B.

In the present embodiment, as shown in FIG. 5, the pixel 31 adopts a 2-tap pixel structure having two charge distribution portions of the first tap 52A and the second tap 52B. Can be realized with 1 tap or 4 taps.

The operation of the pixel 31 will be described.

As shown in FIG. 6, for example, irradiation light modulated (1 cycle = 2T) so as to repeat irradiation on / off at the irradiation time T is output from the light emitting source 21 according to the distance to the object. The reflected light is received by the photodiode 51 with a delay time ΔT. Further, the distribution signal DIMIX_A controls the on / off of the transfer transistor 41A, and the distribution signal DIMIX_B controls the on / off of the transfer transistor 41B. The distribution signal DIMIX_A is a signal having the same phase as the irradiation light, and the distribution signal DIMIX_B has a phase in which the distribution signal DIMIX_A is inverted.

Therefore, in FIG. 6, the electric charge generated by the photodiode 51 receiving the reflected light is transferred to the FD unit 42A according to the distribution signal DIMIX_A while the transfer transistor 41A is on, and is transferred to the transfer transistor according to the distribution signal DIMIX_B. While 41B is on, it is transferred to the FD unit 42B. As a result, the charges transferred via the transfer transistor 41A are sequentially accumulated in the FD unit 42A and transferred via the transfer transistor 41B during a predetermined period in which the irradiation light of the irradiation time T is periodically irradiated. The electric charge is sequentially accumulated in the FD unit 42B.

Then, when the selection transistor 43A is turned on according to the selection signal ADDRESS DECODE_A after the period for accumulating the electric charge, the electric charge accumulated in the FD unit 42A is read out via the signal line 53A and corresponds to the amount of the electric charge. The detection signal A is output from the ranging sensor 24. Similarly, when the selection transistor 43B is turned on according to the selection signal ADDRESS DECODE_B, the electric charge accumulated in the FD unit 42B is read out via the signal line 53B, and the detection signal B according to the amount of the electric charge is the distance measuring sensor 24. Is output from. Further, the electric charge stored in the FD section 42A is discharged when the reset transistor 44A is turned on according to the reset signal RST_A, and the electric charge stored in the FD section 42B is discharged when the reset transistor 44B is turned on according to the reset signal RST_B. Will be done.

In this way, the pixel 31 distributes the electric charge generated by the reflected light received by the photodiode 51 to the first tap 52A or the second tap 52B according to the delay time ΔT, and outputs the detection signal A and the detection signal B. To do. The data obtained by AD-converting the detection signal A and the detection signal B into digital signals corresponds to the raw data described above. The delay time ΔT corresponds to the time during which the light emitted from the light emitting source 21 flies to the object, is reflected by the object, and then flies to the distance measuring sensor 24, that is, the delay time according to the distance to the object. Therefore, the distance measuring module 11 can obtain the distance (depth value) to the object according to the delay time ΔT based on the detection signal A and the detection signal B.

In the normal drive mode, the distance measuring sensor 24 reflects at the light receiving timing shifted by 0 °, 90 °, 180 °, and 270 ° with respect to the irradiation timing of the irradiation light, as shown in FIG. It receives light and outputs 4 phases (4 types) of raw data.

As shown in FIG. 8, in the first tap 52A, the detection signal A obtained by receiving light in the same phase (phase 0 °) as the irradiation light is shifted by 90 degrees from the detection signal A ₀ and the irradiation light (phase). The detection signal A obtained by receiving light at 90 °) is the detection signal A ₉₀ , and the detection signal A obtained by receiving light at a phase (phase 180 °) shifted by 180 degrees from the irradiation light is detected signal A ₁₈₀ , the irradiation light and 270. The detection signal A obtained by receiving light in a shifted phase (phase 270 °) is referred to as a detection signal A ₂₇₀ .

Further, in the second tap 52B, the detection signal B obtained by receiving the light in the same phase (phase 0 °) as the irradiation light is received in the detection signal B ₀ and the phase (phase 90 °) shifted by 90 degrees from the irradiation light. The detection signal B obtained by receiving the detection signal B with the detection signal B ₉₀ and the phase shifted by 180 degrees from the irradiation light (phase 180 °) is received and the detection signal B obtained by receiving the detection signal B with the detection signal B ₁₈₀ and the phase shifted by 270 degrees from the irradiation light (phase). The detection signal B obtained by receiving light at 270 °) will be referred to as the detection signal B ₂₇₀ .

FIG. 8 is a diagram illustrating a method of calculating the depth value and the reliability by the 4 Phase method.

In the Indirect ToF method, the depth value d corresponding to the distance to the object can be obtained by the following equation (1).

In equation (1), c is the speed of light, ΔT is the delay time, and f is the modulation frequency of light. Further, φ in the equation (1) represents the phase shift amount [rad] of the reflected light, and is represented by the following equation (2).

In the 4-Phase method, I and Q of the equation (2) set the phases to 0 °, 90 °, 180 ° and 270 ° to obtain the detection signals A _{0 to} A ₂₇₀ and the detection signals B _{0 to} B ₂₇₀ . It is calculated by the following equation (3). I and Q are signals obtained by converting the phase of the cos wave from polar coordinates to a Cartesian coordinate system (IQ plane), assuming that the change in brightness of the irradiation light is a cos wave.
I = c _0- c ₁₈₀ = (A _0- B ₀ )-(A _180- B ₁₈₀ )
Q ＝ c ₉₀ －c ₂₇₀ ＝ (A ₉₀ －B ₉₀ ) － (A ₂₇₀ －B ₂₇₀ ) ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (3)

Therefore, the detection signals A ₀ and B ₀ (digital values) obtained by receiving light at phase 0 ° are the raw data of phase 0 °, and the detection signals A ₉₀ and B ₉₀ (digital value) obtained by receiving light at phase 90 ° are digital. The value) is received in 90 ° phase Raw data, and the detection signals A ₁₈₀ and B ₁₈₀ (digital values) obtained by receiving light in phase 180 ° are received in phase 180 ° Raw data and in phase 270 °. By outputting the detection signals A ₂₇₀ and B ₂₇₀ (digital values) as raw data with a phase of 270 °, the distance to the object can be calculated.

Further, the reliability cnf can be obtained by the following equation (4).

FIG. 9 is a conceptual diagram of the light emission control signal and the exposure control signal when the drive mode is the normal drive mode, and the raw data.

In the normal drive mode, the light emission control signal is, for example, a rectangular wave pulse signal that becomes high at a predetermined period, as shown in FIG. Further, the exposure control signal is a rectangular wave pulse signal whose phase is shifted by 0 °, 90 °, 180 °, or 270 ° with respect to the irradiation timing indicated by the light emission control signal.

Then, in the normal drive mode, the distance measuring sensor 24 outputs the detection signals A ₀ and B ₀ having a phase of 0 ° as raw data P ₀ having a phase of 0 °, and the detection signals A ₉₀ and B ₉₀ having a phase of 90 ° have a phase. The 90 ° raw data P ₉₀ is output, the 180 ° phase detection signals A ₁₈₀ and B ₁₈₀ are output as 180 ° phase Raw data P ₁₈₀ , and the 270 ° phase detection signals A ₂₇₀ and B ₂₇₀ are in phase. It is outputted as Raw data P ₂₇₀ of 270 °.

FIG. 10 is a conceptual diagram of a light emission control signal and an exposure control signal when the drive mode is a special drive mode, and raw data.

In the special drive mode, as shown in FIG. 10, the light emission control signal is, for example, an impulse function-like signal that momentarily becomes high for a short time in a predetermined cycle. The light emission control signal can be expressed as, for example, a periodic function δ (t) such as the following equation (5) that causes an impulse only when the phase is 0.

Here, f in the equation (5) represents the emission frequency, t represents the time, and n represents an integer of 0 or more. Hereinafter, the periodic function δ (t) is referred to as a light emitting function δ (t).

On the other hand, the exposure control signal is a square wave pulse signal whose phase is shifted by 1 ° from 0 ° to 179 ° with respect to the irradiation timing indicated by the light emission control signal. The exposure period T is set to be the same as the normal drive mode.

Then, in the special drive mode, the distance measuring sensor 24 outputs the detection signals A ₀ and B ₀ having a phase of 0 ° as the raw data P ₀ having a phase of 0 °, and the detection signals A ₁ and B ₁ having a phase of 1 ° have a phase. Output as 1 ° raw data P ₁ and phase 2 ° detection signals A ₂ and B ₂ are output as phase 2 ° raw data P ₂ ..., phase 179 ° detection signals A ₁₇₉ and B _179. Is output as Raw data P ₁₇₉ with a phase of 179 °.

<4. Detailed configuration example of the teacher data generation unit>
FIG. 11 is a block diagram showing a detailed configuration example of the teacher data generation unit 12.

The teacher data generation unit 12 is composed of a storage unit 61 and a calculation unit 62, and operates when the drive mode is a special drive mode.

The storage unit 61 sets a delay amount with respect to the irradiation timing indicated by the light emission control signal, that is, a phase difference, and supplies the delay amount to the delay generation unit 23 of the distance measuring module 11. The storage unit 61 sets the phase difference from 0 ° to 179 ° in 1 ° units. Then, when the temperature is set from 0 ° to 179 ° in 1 ° units, the raw data P _{1 to} P ₁₇₉ having a total of 180 phases, which are sequentially supplied from the distance measuring sensor 24, are stored. When the raw data P _{1 to} P ₁₇₉ having a total of 180 phases are accumulated, the storage unit 61 supplies them to the calculation unit 62.

The calculation unit 62 generates impulse response data in the time direction of the reflected light with respect to the subject as teacher data from the raw data P _{1 to} P _{179 having} a total of 180 phases supplied from the storage unit 61.

Here, the function representing the known emission control signal (emission function) is δ (t), the function representing the known exposure control signal (exposure function) is E (t-τ), and the unknown reflected light to the object to be measured. When the impulse response data in the time direction of is r (t), the brightness value I _τ (x, y) of the predetermined pixel (x, y) acquired as raw data is expressed by the following equation (6). Can be done.

In equation (6), T is the exposure time, * is the operator representing the convolution integral, and τ is the phase difference (delay amount) ΔT between the irradiation timing of the emission control signal and the exposure timing of the exposure control signal.

The displacement of the luminance values of the raw data P _{1 to} P ₁₇₉ of a total of 180 phases obtained by receiving light while changing the phase difference from 0 ° to 179 ° for a predetermined pixel (x, y) is a function F _xy (τ). Then, the function F _xy (τ) can be expressed by Eq. (7).

If this function F _xy (τ) is called a correlation function, the correlation function F _xy (τ) is an exposure function E in which the impulse response data r (t) of unknown reflected light in the time direction has a finite width. It can be said that it is in a "blurred" state by (t-τ).

Therefore, the impulse response of the reflected light of a predetermined pixel (x, y) in the time direction is performed by performing a deconvolution operation on the correlation function F _xy (τ) with the blur function as the exposure function E (t-τ). It can be converted to data r _xy (t).

That is, the arithmetic unit 62 is based on the convolution by the inverse function of the correlation function F _xy (τ) and the exposure function E (t-τ) and the inverse function of the emission function δ (t) represented by the following equation (8). By convolution, the impulse response data r _xy (t) in the time direction of a predetermined pixel (x, y) is calculated.

In equation (8), * is an operator that represents the convolution integral.

If the SN ratio Γ of the brightness value I _τ (x, y) of a predetermined pixel (x, y) acquired as raw data can be estimated, a Wiener filter M (t) for noise suppression is used. It can be used to calculate the impulse response data r _xy (t) of the reflected light of a predetermined pixel (x, y) in the time direction. The impulse response data r _xy (t) in the time direction of the reflected light of a predetermined pixel (x, y) when the Wiener filter M (t) is used is represented by the following equation (9).

In equation (9), * is an operator representing a convolution integral, and M (t) is a Wiener filter. The Wiener filter M (t) is represented by the equation (10).

FT ^{-1 in} equation (10) is the operator that performs the inverse Fourier transform (that is, inverse Fourier transform) of the Fourier transform. F hat (∧ on F) _xy is FT [F _xy ], F hat ^* _xy is , F represents the complex conjugate of _xy .

Since the impulse response data r (t) in the time direction of the reflected light is calculated for each pixel 31 of the pixel array unit 32, it is a total of two dimensions in the spatial direction and one dimension in the time direction corresponding to the pixel array of the ToF sensor. It becomes three-dimensional data.

As described above, the calculation unit 62 calculates the impulse response data r (t) in the time direction of the reflected light with respect to the subject from the Raw data P _{1 to} P _{179 having} a total of 180 phases supplied from the storage unit 61. , Supply to the learning unit 13.

<5. Detailed configuration example of the learning department>
FIG. 12 is a block diagram showing a detailed configuration example of the learning unit 13.

The learning unit 13 is composed of a model calculation unit 81, an error calculation unit 82, and a parameter update unit 83.

The learning section 13, the ranging module 11 is generated and is driven in the normal drive mode, Raw data P ₀ of the phase 0 °, the phase 90 ° of Raw data P _90, a phase 180 ° of Raw Data P ₁₈₀ and, Raw data P ₂₇₀ with a phase of 270 ° is supplied from the ranging module 11.

Further, in the learning unit 13, teacher data is generated based on 180 phase raw data P _{0 to} P ₁₇₉ in 1 ° units from phase 0 ° to 179 °, which is generated by driving the distance measuring module 11 in the special drive mode. The impulse response data r (t) in the time direction of the reflected light generated by the generation unit 12 is supplied from the teacher data generation unit 12.

Here, the four-phase raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ generated by the ranging module 11 in the normal drive mode and the 180-phase raw data generated by the ranging module 11 in the special drive mode. P _{0 to} P ₁₇₉ are acquired in a static scene and corresponded to each other. In other words, the raw data used by the learning unit 13 is data obtained by the distance measuring module 11 measuring the same stationary subject in each of the normal drive mode and the special drive mode.

Raw data P ₀ of the phase 0 °, the phase 90 ° of Raw data P _90, a phase 180 ° of Raw Data P ₁₈₀ and,, Raw data P ₂₇₀ phase 270 °, the model learning unit 13 as student data for learning model The impulse response data r (t) supplied to the calculation unit 81 in the time direction of the reflected light is supplied to the error calculation unit 82 of the learning unit 13 as training data of the learning model.

The model calculation unit 81 has a phase 0 ° raw data P ₀ , a phase 90 ° raw data P ₉₀ , and a phase 180 ° supplied as student data to the learning model using the model parameters set by the parameter update unit 83. The raw data P ₁₈₀ and the raw data P ₂₇₀ having a phase of 270 ° are substituted to calculate the impulse response data r (t) in the time direction of the reflected light. The impulse response data r (t) in the time direction of the reflected light calculated by the learning model is supplied to the error calculation unit 82.

The error calculation unit 82 includes the impulse response data r (t) in the time direction of the reflected light supplied from the model calculation unit 81 and the impulse response data in the time direction of the reflected light supplied as teacher data from the teacher data generation unit 12. The error with r (t) is calculated and supplied to the parameter update unit 83.

When the error supplied from the error calculation unit 82 is equal to or greater than a predetermined value, the parameter update unit 83 updates the model parameters so that the error is small and supplies the model parameters to the model calculation unit 81. On the other hand, when the error supplied from the error calculation unit 82 is smaller than the predetermined value and it is determined that the learning model has been sufficiently learned, the current model parameters are supplied to the estimation unit 14 (FIG. 1). In machine learning such as a convolutional neural network, in general, the learning model is sufficiently learned by repeating the update of the model parameters a predetermined number of times.

The learned model parameters are supplied from the learning unit 13 to the estimation unit 14 and set as model parameters of the CNN which is a predictor. As a result, as described in FIG. 3, the estimation unit 14 receives the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ output by the ranging sensor 24 as inputs, and impulses the reflected light in the time direction. The response data r (t) can be predicted and output.

<6. Distance measurement system processing>
The learning process in which the ranging system 1 learns the prediction model (learning model) for predicting the impulse response data in the time direction of the reflected light will be described with reference to the flowchart of FIG. This process is started, for example, when an operation unit (not shown) instructs the start of the learning process.

First, in step S1, the control unit 15 sets the ranging module 11 to the normal drive mode, which is the first drive mode. The light emission control unit 22 supplies a light emission control signal to the light emission source 21 that repeats irradiation on / off at the irradiation time T. The delay generation unit 23 sequentially generates an exposure control signal whose phase is shifted by 0 °, 90 °, 180 °, and 270 ° with respect to the light emission control signal, and supplies the exposure control signal to the distance measuring sensor 24.

In step S2, the distance measuring sensor 24 receives the reflected light that is reflected by the object and returned from the irradiation light emitted from the light emitting source 21. More specifically, the ranging sensor 24 sequentially receives the reflected light with a phase difference of 0 °, 90 °, 180 °, and 270 ° with respect to the irradiation timing of the irradiation light according to the exposure control signal. Then, the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ obtained as a result are output to the learning unit 13.

In step S3, the control unit 15 sets the ranging module 11 to the special drive mode, which is the second drive mode. The light emission control unit 22 supplies the light emission source 21 with an impulse function-like light emission control signal that becomes high for a short time in a predetermined cycle. The delay generation unit 23 sequentially generates an exposure control signal whose phase is shifted by 1 ° from 0 ° to 179 ° with respect to the irradiation timing indicated by the light emission control signal, and supplies the exposure control signal to the distance measuring sensor 24.

In step S4, the distance measuring sensor 24 receives the reflected light that is reflected by the object and returned by the irradiation light emitted from the light emitting source 21. More specifically, the ranging sensor 24 receives the reflected light by changing the phase difference in 1 ° increments from 0 ° to 179 ° with respect to the irradiation timing of the irradiation light according to the exposure control signal. The obtained 180-phase Raw data P _{0 to} P ₁₇₉ are output to the teacher data generation unit 12.

In step S5, the teacher data generation unit 12 sequentially supplies each pixel (x _,,) of the pixel array unit 32 from the 180-phase raw data P _{0 to} P ₁₇₉ from 0 ° to 179 °, which are sequentially supplied from the distance measuring sensor 24. The correlation function F _xy (τ) of _y ) is calculated. Specifically, the storage unit 61 stores raw data P _{0 to} P ₁₇₉ of 180 phases from 0 ° to 179 ° sequentially supplied from the distance measuring sensor 24. Using the 180-phase Raw data P _{0 to} P ₁₇₉ stored in the storage unit 61, the calculation unit 62 uses the above-mentioned equation (7) to perform a correlation function F of each pixel (x, y) of the pixel array unit 32. Calculate _xy (τ).

In step S6, the calculation unit 62 calculates the impulse response data r _xy (t) of the reflected light in the time direction using the calculated correlation function F _xy (τ). More specifically, the arithmetic unit 62 convolves by the inverse function of the correlation function F _xy (τ) and the exposure function E (t-τ) and the emission function δ (t) represented by the above equation (8). The impulse response data r _xy (t) in the time direction of the reflected light of each pixel (x, y) of the pixel array unit 32 is calculated by convolution by the inverse function of. The calculated impulse response data r _xy (t) of the reflected light of each pixel (x, y) in the time direction is supplied to the learning unit 13.

In step S7, the learning unit 13 uses the four-phase raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ generated in the normal drive mode as student data, and the reflected light generated by the teacher data generation unit 12. The prediction model is _{trained using} the impulse response data r _xy (t) in the time direction of.

The parameters (model parameters) of the prediction model obtained by the learning in step S7 are supplied to the estimation unit 14, and the learning process is completed.

Next, with reference to the flowchart of FIG. 14, the prediction process of the distance measuring system 1 that predicts and outputs the impulse response data in the time direction of the reflected light in the normal drive mode will be described. This process is started, for example, when an operation unit (not shown) instructs the start of the prediction process.

First, in step S11, the control unit 15 sets the ranging module 11 to the normal drive mode, which is the first drive mode. The light emission control unit 22 supplies a light emission control signal to the light emission source 21 that repeats irradiation on / off at the irradiation time T. The delay generation unit 23 sequentially generates an exposure control signal whose phase is shifted by 0 °, 90 °, 180 °, and 270 ° with respect to the light emission control signal, and supplies the exposure control signal to the distance measuring sensor 24.

In step S12, the distance measuring sensor 24 receives the reflected light that is reflected by the object and returned by the irradiation light emitted from the light emitting source 21. More specifically, the ranging sensor 24 sequentially receives the reflected light with a phase difference of 0 °, 90 °, 180 °, and 270 ° with respect to the irradiation timing of the irradiation light according to the exposure control signal. Then, the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ obtained as a result are output to the estimation unit 14.

In step S13, the estimation unit 14 predicts that the four-phase raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ supplied from the distance measuring sensor 24 are set with the model parameters learned by the learning unit 13. Input to the device (CNN) to generate (predict) impulse response data in the time direction of the reflected light. Then, the estimation unit 14 outputs the impulse response data of the reflected light obtained as the prediction result in the time direction to the outside, and ends the prediction process.

In step S13, the estimation unit 14 includes not only the impulse response data of the generated reflected light in the time direction, but also the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ supplied from the ranging sensor 24. At the same time, it may be output.

According to the above learning process by the distance measuring system 1, when the distance measuring sensor 24, which is an Indirect ToF type ToF sensor, outputs light receiving data corresponding to the distance information to the subject in the normal drive mode, the light receiving data. Based on the above, it is possible to generate a model parameter of a predictor that generates impulse response data in the time direction of reflected light, which is light receiving data with improved resolution in the time direction.

Then, according to the prediction processing by the distance measuring system 1, the four-phase Raw data P ₀ , P ₉₀ , and P ₁₈₀ output by the distance measuring sensor 24 in the normal drive mode using the model parameters learned in the learning process _. , And P ₂₇₀ , it is possible to generate impulse response data of reflected light in the time direction, which is received data with improved resolution in the time direction.

Therefore, according to the ranging system 1, it is possible to more easily obtain impulse response data in the time direction of the reflected light.

<7. Modification example>
<Use of 4-phase raw data with multiple frequencies>
When the irradiation light is emitted at the emission frequency f and the raw data of four phases is acquired in the normal drive mode to calculate the distance to the object, the calculated depth value d is compared with the _true depth value d _true. An error Δd occurs due to the influence of multipath or the like. The error Δd with respect to the _true depth value d _{true at} the emission frequency f can be expressed by the following equation (11).

In equation (11), c is the speed of light, ∠ is the operator that extracts the phase in complex space, and R (f) is the complex Fourier transform of the impulse response data r (t) in the time direction of the reflected light at the emission frequency f. Represent.

Therefore, as shown in FIG. 15, the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ obtained by setting a plurality of emission frequencies f (emission frequencies f ₁ and f ₂ ) are obtained. By giving the learning device of the learning unit 13 as student data for learning, it is possible to perform learning by excluding error factors due to multi-pass or the like, and it is possible to learn model parameters with high prediction accuracy.

When the emission frequencies f ₁ and f ₂ are set and the raw data of four phases is acquired as learning data in the normal drive mode, the special drive mode corresponds to either the emission frequencies f ₁ or f ₂ 180. The phase raw data P _{0 to} P ₁₇₉ may be used. In other words, even when phase data is acquired at a plurality of emission frequencies in the normal drive mode, the raw data acquired in the special drive mode is the same as in the case of one emission frequency.

<8. Application example>
<First application example>
Next, an application example using the ranging system 1 of FIG. 1 will be described.

Note that, in FIGS. 16 to 24, the parts corresponding to those in FIG. 1 and the like described above are the same, and the description thereof will be omitted as appropriate.

FIG. 16 is a block diagram showing a first application example using the ranging system 1.

The distance measuring system 1 of FIG. 16 is composed of a distance measuring module 11, a signal processing device 101, and a peak position calculation unit 102.

The signal processing device 101 of FIG. 16 corresponds to the signal processing device 16 shown by the broken line in FIG. 1, and the learned model parameters are set in the estimation unit 14. Alternatively, the signal processing device 101 may be composed only of the estimation unit 14 in which the trained model parameters are set.

The ranging module 11 is set to the normal drive mode, which is the first drive mode, and receives the reflected light that is reflected by the object and the irradiation light emitted from the light emitting source 21 is received by the object, and the four-phase raw data P Outputs ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ .

The signal processing device 101 (estimating unit 14) inputs the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ supplied from the ranging sensor 24 of the ranging module 11 into the predictor for prediction. As a result, impulse response data in the time direction of the reflected light is generated and output to the peak position calculation unit 102.

The peak position calculation unit 102 detects the first peak from the impulse response data in the time direction output from the signal processing device 101 (estimating unit 14), and sets the distance corresponding to the first peak to the object. Generates and outputs a corrected depth image as the depth value of.

As described with reference to FIG. 2 and the like, when the irradiation light is multiple-reflected and received by the ranging module 11, the impulse response data in the time direction of the reflected light has two or more peaks with respect to the time axis. It becomes the received light data of the profile having. As shown in FIG. 17, of the plurality of peaks of the impulse response data in the time direction of the reflected light, the light directly reflected from the object becomes the first peak, and is indirectly reflected and incident on the same pixel. The emitted light becomes the second peak, the third peak, and so on. The first peak, the second peak, the third peak, ... Are counted in order from the peak having the smaller time axis.

The peak position calculation unit 102 corrects the distance corresponding to the first peak as a true depth value, so that a more accurate depth image can be generated.

<Second application example>
FIG. 18 is a block diagram showing a second application example using the ranging system 1.

The distance measuring system 1 of FIG. 18 is composed of a distance measuring module 11, a signal processing device 101, and a material estimation unit 103.

The impulse response data of the reflected light in the time direction differs depending on the material, for example, S. Su, F. Heide, R. Swanson, J. Klein, C. Callenberg, M. Hullin, W. Heidrich, Material Classification Using Raw. It is disclosed in Time-of-Flight Measurements, in CVPR 2016., etc.

The material estimation unit 103 estimates and outputs the material of the object as the subject from the impulse response data in the time direction of the reflected light output from the signal processing device 101 (estimation unit 14). The material estimation unit 103 classifies the material of the object into, for example, cloth, wood, metal, and the like, and outputs the material.

As shown in FIG. 19, the material estimation unit 103 includes, for example, a feature extraction unit 111 by a convolutional neural network having a three-dimensional (x, y, t) convolutional layer, and a classification unit using a fully connected layer. It can be configured with 112. Of course, other configurations may be adopted as the estimator for estimating the material.

<Third application example>
FIG. 20 is a block diagram showing a third application example using the ranging system 1.

The distance measuring system 1 of FIG. 20 is composed of a distance measuring module 11, a signal processing device 101, a material estimation unit 103, an AR display control unit 104, and an AR glass 105.

The distance measuring system 1 of FIG. 20 has a configuration in which an AR display control unit 104 and an AR glass 105 are added to the distance measuring system 1 of FIG.

The ranging system 1 of FIG. 20 is a system that uses the material information estimated by the material estimation unit 103 for the superimposed display of the AR glass 105 that superimposes characters and images on the landscape in the real world.

FIG. 21 shows a display example of the AR glass 105 in the third application example.

The materials of the floor 121 and the floor 122 in the real world reflected in the user's view through the AR glass 105 are estimated by the material estimation unit 103 to be polyethylene and polypropylene, respectively.

The AR display control unit 104 superimposes and displays the estimated material name on the areas of the floor 121 and the floor 122 of the AR glass 105.

FIG. 22 shows another display example of the AR glass 105 in the third application example.

The AR display control unit 104 superimposes and displays the robots (images) 131 and 132 on the landscape in the real world. When the materials of the floor 121 and the floor 122 in the real world are estimated to be polyethylene and polypropylene by the material estimation unit 103, respectively, the AR display control unit 104 uses the area of the floor 121 where the robots 131 and 132 are estimated to be polyethylene. The robots 131 and 132 are displayed on the AR glass 105 so as to move only on the top.

The AR display control unit 104 of FIG. 20 generates an image to be displayed (superimposed display) on the AR glass 105 based on the material supplied from the material estimation unit 103, and supplies the image to the AR glass 105. In the example of FIG. 21, the image displayed on the AR glass 105 by the AR display control unit 104 is a character representing a material name such as polyethylene or polypropylene. In the example of FIG. 22, the images displayed on the AR glass 105 by the AR display control unit 104 are the robots 131 and 132.

The AR glass 105 displays an image supplied from the AR display control unit 104 on a predetermined display unit.

FIG. 23 is a flowchart of the AR display process for performing the AR display described with reference to FIG. 22 in the third application example.

First, in step S31, the distance measuring module 11 measures the distance to the subject in the normal driving mode, which is the first driving mode. The four-phase raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ obtained by the measurement are supplied to the signal processing device 101.

In step S32, the signal processing device 101 inputs the four-phase Raw data P ₀ , P ₉₀ , P ₁₈₀ , and P ₂₇₀ supplied from the ranging sensor 24 into the predictor, and as a prediction result, the time of the reflected light. The impulse response data of the direction is generated and output to the material estimation unit 103.

In step S33, the material estimation unit 103 estimates the material in the real world reflected in the user's field of view from the impulse response data in the time direction of the reflected light supplied from the signal processing device 101, and outputs the material to the AR display control unit 104. To do. The correspondence between the user's field of view through the AR glass 105 and the measurement area of the distance measuring module 11 is taken by calibration or the like.

In step S34, the AR display control unit 104 determines the moving directions of the robots 131 and 132, and in step S35, determines whether the determined moving directions are the regions in which the robots 131 and 132 can move. For example, when moving only on the region of the floor 121 presumed to be polyethylene as in the above-mentioned example, it is determined whether the determined moving direction is on the region of the floor 121.

If it is determined in step S35 that the determined moving direction is not the area in which the robots 131 and 132 can move, the process returns to step S34, and the moving directions of the robots 131 and 132 are determined again.

On the other hand, if it is determined in step S35 that the determined moving direction is the area in which the robots 131 and 132 can move, the process proceeds to step S36, and the AR display control unit 104 determines the robots 131 and 132. Move in the direction of movement.

After step S36, the process returns to step S31, and the processes of steps S31 to S36 described above are repeated. For example, when the end of the AR display is instructed by an operation unit (not shown) or the like, the AR display process of FIG. 23 is ended.

In the third application example described above, the material estimation result by the material estimation unit 103 was used for the superimposed display of the AR glass 105, but it can also be used for the movement control of the autonomously moving robot.

FIG. 24 shows an application example in which the material estimation result by the material estimation unit 103 is used for the movement control of the autonomously moving robot.

The robot 141 of FIG. 24 has an image sensor and a distance measuring module 11 as visual sensors, and includes captured images and information such as a distance to an object existing in the traveling direction measured by the distance measuring module 11. Move autonomously based on.

The robot 141 includes the distance measuring module 11, the signal processing device 101, the material estimation unit 103, and the movement control unit that controls the movement, as described above.

The movement control unit determines the movement direction based on the materials of the floor 142 and the floor 143 supplied from the material estimation unit 103. For example, it is estimated that the floor 142 is a soft material such as a carpet that becomes unstable when the robot 141 touches the ground, and the floor 143 is a hard material such as a tile that is stable when the robot 141 touches the ground. In this case, the movement control unit controls the movement so as to move the area of the floor 143 without moving the area of the floor 142.

In step S33 of the AR display process described with reference to FIG. 23, the material estimation unit 103 estimates the material of the robot's field of view instead of the user's field of view, and in steps S34 to S36, instead of the AR display control unit 104. In addition, when the movement control unit executes the same process, the control as described with reference to FIG. 24 becomes possible. As a result, the robot 141 can select and move a stable ground on which the posture can be easily maintained.

<9. Computer application example>
The series of processes described above can be executed by hardware or by software. When a series of processes are executed by software, the programs constituting the software are installed on the computer. Here, the computer includes a microcomputer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 25 is a block diagram showing a configuration example of computer hardware that programmatically executes a series of processes by the signal processing device 16 in the subsequent stage of the ranging module 11.

In a computer, the CPU (Central Processing Unit) 201, ROM (ReadOnly Memory) 202, and RAM (RandomAccessMemory) 203 are connected to each other by a bus 204.

An input / output interface 205 is further connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

The input unit 206 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 207 includes a display, a speaker, an output terminal, and the like. The storage unit 208 includes a hard disk, a RAM disk, a non-volatile memory, and the like. The communication unit 209 includes a network interface and the like. The drive 210 drives a removable recording medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 201 loads the program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executes the above-described series. Is processed. The RAM 203 also appropriately stores data and the like necessary for the CPU 201 to execute various processes.

The program executed by the computer (CPU201) can be recorded and provided on the removable recording medium 211 as a package medium or the like, for example. Programs can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 208 via the input / output interface 205 by mounting the removable recording medium 211 in the drive 210. Further, the program can be received by the communication unit 209 and installed in the storage unit 208 via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM 202 or the storage unit 208.

In addition, in this specification, the steps described in the flowchart are not necessarily processed in chronological order as well as in chronological order in the order described, but are called in parallel or are called. It may be executed at a necessary timing such as when.

In the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

For example, a form in which all or a part of the plurality of embodiments described above can be combined can be adopted.

For example, this technology can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and processed jointly.

In addition, each step described in the above flowchart can be executed by one device or shared by a plurality of devices.

Further, when one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

Note that the effects described in the present specification are merely examples and are not limited, and effects other than those described in the present specification may be obtained.

The present technology can have the following configurations.
(1)
The reflection on the subject from the received data output in the second drive mode by the distance measuring sensor that receives the reflected light reflected by the subject in the first drive mode or the second drive mode. A teacher data generator that generates impulse response data in the time direction of light as teacher data,
A signal processing device including a learning unit that calculates model parameters of a learning model that learns the relationship between the received light data in the first drive mode and the impulse response data in the time direction of the reflected light.
(2)
An estimation unit that inputs the received data of the distance measuring sensor in the first drive mode to the learning model using the model parameters calculated by the learning unit and outputs the impulse response data in the time direction of the reflected light. The signal processing device according to (1) above.
(3)
The teacher data generation unit refers to the subject from a plurality of received data when the distance measuring sensor in the second drive mode receives the reflected light a plurality of times while changing the phase difference with the irradiation light. The signal processing device according to (1) or (2) above, which generates impulse response data in the time direction of the reflected light as training data.
(4)
The teacher data generation unit exposes a plurality of received data when the distance measuring sensor in the second drive mode receives the reflected light a plurality of times while changing the phase difference with the irradiation light. The signal processing device according to (3) above, which generates impulse response data in the time direction of the reflected light as the teacher data by performing convolution by the inverse function of the exposure function indicating the timing.
(5)
The signal processing device according to any one of (1) to (4), wherein in the second drive mode, the irradiation light is emitted based on an impulse function-shaped light emission control signal.
(6)
The learning unit uses the teacher data as impulse response data in the time direction of the reflected light obtained in the second drive mode, and the student data as light receiving data in the first drive mode, as model parameters of the learning model. The signal processing apparatus according to any one of (1) to (5) above.
(7)
The signal processing device according to any one of (1) to (6) above, wherein the learning model of the learning unit is composed of a convolutional neural network.
(8)
The learning unit calculates model parameters of the learning model as the light receiving data obtained by the distance measuring sensor by setting a plurality of emission frequencies in the first drive mode from the student data (1). The signal processing apparatus according to any one of (7) to (7).
(9)
With the distance measuring sensor
A light emitting source that emits the irradiation light and
A light emission control unit that supplies a light emission control signal that controls the light emission source,
The signal processing according to any one of (1) to (8) above, further comprising a distance measuring module including a delay generating unit for generating an exposure control signal for controlling the exposure timing of the distance measuring sensor from the light emission control signal. apparatus.
(10)
The signal processing device
The reflection on the subject from the received light data output by the distance measuring sensor in the second drive mode, which receives the reflected light reflected by the subject in the first drive mode or the second drive mode. Generate impulse response data in the time direction of light as teacher data,
A signal processing method for calculating model parameters of a learning model for learning the relationship between the received light data in the first drive mode and the impulse response data in the time direction of the reflected light.

1 distance measurement system, 11 distance measurement module, 12 teacher data generation unit, 13 learning unit, 14 estimation unit, 15 control unit, 16 signal processing device, 21 light emission source, 22 light emission control unit, 23 delay generation unit, 24 distance measurement Sensor, 101 signal processing device, 102 peak position calculation unit, 103 material estimation unit, 104 AR display control unit, 105 AR glass, 201 CPU, 202 ROM, 203 RAM, 206 input unit, 207 output unit, 208 storage unit, 209 Communication unit, 210 drive

Claims (10)

1. The reflection on the subject from the received data output in the second drive mode by the distance measuring sensor that receives the reflected light reflected by the subject in the first drive mode or the second drive mode. A teacher data generator that generates impulse response data in the time direction of light as teacher data,
   A signal processing device including a learning unit that calculates model parameters of a learning model that learns the relationship between the received light data in the first drive mode and the impulse response data in the time direction of the reflected light.
2. An estimation unit that inputs the received data of the distance measuring sensor in the first drive mode to the learning model using the model parameters calculated by the learning unit and outputs the impulse response data in the time direction of the reflected light. The signal processing device according to claim 1, further comprising.
3. The teacher data generation unit refers to the subject from a plurality of received data when the distance measuring sensor in the second drive mode receives the reflected light a plurality of times while changing the phase difference with the irradiation light. The signal processing device according to claim 1, wherein the impulse response data in the time direction of the reflected light is generated as training data.
4. The teacher data generation unit exposes a plurality of received data when the distance measuring sensor in the second drive mode receives the reflected light a plurality of times while changing the phase difference with the irradiation light. The signal processing apparatus according to claim 3, wherein the impulse response data in the time direction of the reflected light as the teacher data is generated by convolution by the inverse function of the exposure function indicating the timing.
5. The signal processing device according to claim 1, wherein in the second drive mode, the irradiation light is emitted based on an impulse function-shaped emission control signal.
6. The learning unit uses the teacher data as impulse response data in the time direction of the reflected light obtained in the second drive mode, and the student data as light receiving data in the first drive mode, as model parameters of the learning model. The signal processing device according to claim 1.
7. The signal processing device according to claim 1, wherein the learning model of the learning unit is composed of a convolutional neural network.
8. The learning unit calculates the model parameter of the learning model as the light receiving data obtained by the distance measuring sensor by setting a plurality of emission frequencies in the first drive mode. The signal processing device described.
9. With the distance measuring sensor
   A light emitting source that emits the irradiation light and
   A light emission control unit that supplies a light emission control signal that controls the light emission source,
   The signal processing device according to claim 1, further comprising a distance measuring module including a delay generation unit for generating an exposure control signal for controlling the exposure timing of the distance measuring sensor from the light emission control signal.
10. The signal processing device
    The reflection on the subject from the received light data output in the second drive mode by the distance measuring sensor that receives the reflected light reflected by the subject in the first drive mode or the second drive mode. Generate impulse response data in the time direction of light as teacher data,
    A signal processing method for calculating model parameters of a learning model for learning the relationship between the received light data in the first drive mode and the impulse response data in the time direction of the reflected light.

Images