WO2021177045A1

WO2021177045A1 - Signal processing device, signal processing method, and range-finding module

Info

Publication number: WO2021177045A1
Application number: PCT/JP2021/006075
Authority: WO
Inventors: 基三原; 俊海津; 優介森内
Original assignee: ソニーグループ株式会社
Priority date: 2020-03-04
Filing date: 2021-02-18
Publication date: 2021-09-10

Abstract

The present technology relates to a signal processing device, a signal processing method, and a range-finding module which make it possible to more appropriately correct variations in the characteristics of taps. This signal processing device comprises a signal processing unit that performs a process for computing the distance to an object on the basis of pixel data obtained from a pixel receiving reflected light, which is illumination light that has been emitted from a prescribed light source and has reflected off of the object and returned. The signal processing unit has a characteristics calculation unit that computes a correction parameter for correcting the characteristics of a first charge detection unit and a second charge detection unit of the pixel on the basis of a luminance model corresponding to the shape of the light emission waveform of the illumination light and the shape of the exposure waveform of the pixel. The present technology can be applied to, for example, a range-finding module that measures the distance to a subject.

Description

Signal processing device, signal processing method, and ranging module

The present technology relates to a signal processing device, a signal processing method, and a ranging module, and more particularly to a signal processing device, a signal processing method, and a ranging module capable of more appropriately correcting characteristic variations between taps.

In recent years, advances in semiconductor technology have led to the miniaturization of distance measuring modules that measure the distance to an object. As a result, for example, it has been realized that a distance measuring module is mounted on a mobile terminal such as a so-called smartphone.

For example, smartphones are equipped with a distance measurement module that measures distance by the Indirect ToF (Time of Flight) method. In the IndirectToF method, the irradiation light is emitted toward the object, the reflected light is reflected on the surface of the object and returned, and the flight from the emission of the irradiation light to the reception of the reflected light is detected. This is a method of calculating the distance to an object based on time.

In the Indirect ToF method, the light receiving sensor side receives the reflected light at the light receiving timings shifted by 0 °, 90 °, 180 °, and 270 ° based on the irradiation timing of the irradiation light. A method of calculating the distance to an object by using four phase images detected in four different phases with respect to the irradiation timing of the irradiation light is called a 4 Phase method or the like.

On the other hand, one pixel has two charge storage portions, a first tap and a second tap, and by alternately distributing the received charge to the first tap and the second tap, the phase is inverted in one phase image. Since two phases can be detected, there is also a method called a 2-phase method in which the distance to an object is calculated using two phase images.

The 2Phase method has the advantage that the distance can be calculated with half the number of phase images of the 4Phase method, but the characteristics of the first tap and the second tap, which are the two charge storage units of each pixel, vary between taps (tap). Since there is a sensitivity difference between taps), it is necessary to correct the characteristic variation between taps.

For example, Patent Document 1 proposes a method of calculating the offset of each tap by pre-measurement to reduce the error of distance calculation.

Japanese Unexamined Patent Publication No. 2019-191118

However, in the method disclosed in Patent Document 1, the correction formula for correcting the characteristic variation between taps is a formula for dividing by a value close to 0 when the amplitude of the received light waveform is small, which is sufficient for calculation stability and noise immunity. I could not say.

This technology was made in view of such a situation, and makes it possible to more appropriately correct the characteristic variation between taps in a light receiving sensor having two charge storage units in one pixel.

The signal processing device on the first aspect of the present technology is based on the pixel data obtained by the pixels that receive the reflected light that is reflected by the object and returned by the irradiation light emitted from the predetermined light emitting source. A signal processing unit that performs processing for calculating the distance to the It has a characteristic calculation unit that calculates correction parameters that correct the characteristics of the first charge detection unit and the second charge detection unit.

The signal processing method of the second aspect of the present technology is based on the pixel data obtained by the pixel that receives the reflected light that is reflected by the object and returned by the irradiation light emitted from the predetermined light emitting source. The signal processing device that performs the process of calculating the distance to 2 Calculate the correction parameter that corrects the characteristics with the charge detection unit.

The distance measuring module on the third side of the present technology comprises a predetermined light emitting source and a distance measuring sensor having pixels that receive the reflected light that is reflected by an object and returned from the irradiation light emitted from the light emitting source. The distance measuring sensor includes a first charge detection unit and a second charge detection unit of the pixel based on a luminance model corresponding to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel. It has a characteristic calculation unit that calculates correction parameters for correcting characteristics.

In the first to third aspects of the present technology, the first charge detection unit and the second charge detection unit of the pixel are based on the luminance model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel. A correction parameter that corrects the characteristics of and is calculated.

The signal processing device and the ranging module may be independent devices or may be modules incorporated in other devices.

It is a block diagram which shows the schematic structure example of the distance measurement module to which this technology is applied. It is a block diagram which shows the detailed configuration example of a light receiving part. It is a figure explaining the operation of a pixel. It is a figure explaining 2Phase method and 4Phase method. It is a figure explaining 2Phase method and 4Phase method. It is a figure explaining 2Phase method and 4Phase method. It is a block diagram which shows the 1st structural example of a signal processing part. It is a figure which shows the example of the luminance model which a model determination part determines. It is a conceptual diagram of the luminance function when it is assumed that the luminance model is a triangular wave. It is a conceptual diagram of the correction processing performed by the signal correction unit when it is assumed that the luminance model is a triangular wave. It is a conceptual diagram of the luminance function when it is assumed that the luminance model is a sine wave. It is a conceptual diagram of the correction processing performed by the signal correction unit when it is assumed that the luminance model is a sine wave. It is a figure explaining the learning process which learns the luminance model of a harmonic. It is a conceptual diagram of the operation which obtains the 2nd evaluation function L _2. It is a flowchart explaining the measurement process by a distance measurement module. It is a detailed flowchart of the correction parameter estimation process executed in step S2 of FIG. It is a detailed flowchart of the distance measurement process by the 2 Phase method executed in step S3 of FIG. It is a flowchart of the process of calculating offset and gain using machine learning. It is a block diagram which shows the 2nd structural example of a signal processing part. It is a flowchart of the measurement process of the distance measurement module in the case of adopting the 2nd configuration example of the signal processing unit. It is a block diagram which shows the 3rd structural example of a signal processing part. It is a figure explaining the fixed pattern noise evaluation process by an evaluation part. It is a flowchart of the measurement process of the distance measurement module in the case of adopting the 3rd configuration example of the signal processing unit. It is a detailed flowchart of the fixed pattern noise evaluation process executed in step S105 of FIG. It is a perspective view which shows the chip structure example of the distance measuring sensor. It is a block diagram which shows the configuration example of a smartphone as an electronic device equipped with a distance measurement module. It is a block diagram which shows an example of the schematic structure of a vehicle control system. It is explanatory drawing which shows an example of the installation position of the vehicle exterior information detection unit and the image pickup unit.

Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described. The explanation will be given in the following order.
1. 1. Schematic configuration example of the ranging module 2. Indirect To F method distance calculation principle 3. First configuration example of signal processing unit 4. 4. Correction parameter calculation process when the luminance waveform is a triangular wave. Correction parameter calculation process when the luminance waveform is a sine wave 6. Correction parameter calculation process when the luminance waveform is a harmonic 7. Flowchart of measurement process by distance measurement module 8. Second configuration example of the signal processing unit 9. Flowchart of measurement processing using the second configuration example 10. Third configuration example of the signal processing unit 11. Flowchart of measurement processing using the third configuration example 12. Example of chip configuration of distance measuring sensor 13. Configuration example of electronic device 14. Application example to mobile

<1. Schematic configuration example of ranging module>
FIG. 1 is a block diagram showing a schematic configuration example of a distance measuring module to which the present technology is applied.

The distance measurement module 11 shown in FIG. 1 is a distance measurement module that performs distance measurement by the Indirect ToF method, and has a light emitting unit 12 and a distance measurement sensor 13. The ranging module 11 irradiates an object with light, and the light (irradiation light) receives the light (reflected light) reflected by the object to generate a depth map as distance information to the object. And output. The distance measuring sensor 13 includes a control unit 14, a light receiving unit 15, and a signal processing unit 16.

The light emitting unit 12 includes, for example, a VCSEL array in which a plurality of VCSELs (Vertical Cavity Surface Emitting Lasers) are arranged in a plane in a plane as a light emitting source, and responds to a light emitting control signal supplied from the control unit 14. It emits light while being modulated at the timing, and irradiates the object with irradiation light.

The control unit 14 controls the operation of the entire ranging module 11. For example, the control unit 14 controls the light emitting unit 12 by supplying a light emitting control signal of a predetermined frequency (for example, 200 MHz or the like) to the light emitting unit 12. Further, the control unit 14 also supplies a light emission control signal to the light receiving unit 15 in order to drive the light receiving unit 15 in accordance with the timing of light emission in the light emitting unit 12.

The light receiving unit 15 receives the reflected light from the object in the pixel array 32 in which a plurality of pixels 31 are two-dimensionally arranged, which will be described in detail later with reference to FIG. Then, the light receiving unit 15 supplies the pixel data composed of the detection signals according to the received amount of the received reflected light to the signal processing unit 16 in units of pixels 31 of the pixel array 32.

The signal processing unit 16 calculates a depth value, which is the distance from the distance measuring module 11 to the object, based on the pixel data supplied from the light receiving unit 15 for each pixel 31 of the pixel array 32, and the pixel value of each pixel 31. Generates a depth map in which the depth value is stored as, and outputs it to the outside of the module.

<2. Indirect To F method distance calculation principle>
Next, the distance calculation principle of the Indirect ToF method will be described while explaining a detailed configuration example of the light receiving unit 15.

FIG. 2 is a block diagram showing a detailed configuration example of the light receiving unit 15.

The light receiving unit 15 generates a charge corresponding to the amount of received light, and a pixel array 32 and a pixel array in which pixels 31 that output a detection signal corresponding to the charge are two-dimensionally arranged in a matrix in the row direction and the column direction. It has a drive control circuit 33 arranged in a peripheral region of 32.

The drive control circuit 33 is, for example, a control signal for controlling the drive of the pixel 31 based on a light emission control signal supplied from the control unit 14, (for example, a distribution signal DIMIX, a selection signal ADDRESS DECODE, or a reset, which will be described later). (Signal RST, etc.) is output.

The pixel 31 includes a photodiode 51 as a photoelectric conversion unit that generates an electric charge according to the amount of received light, a first tap 52A (first charge detecting unit) that detects the electric charge generated by the photodiode 51, and a second. It has a tap 52B (second charge detection unit). In the pixel 31, the electric charge generated by one photodiode 51 is distributed to the first tap 52A or the second tap 52B. Then, of the charges generated by the photodiode 51, the charges distributed to the first tap 52A are output as a detection signal A from the signal line 53A, and the charges distributed to the second tap 52B are detected signals B from the signal line 53B. Is output as.

The first tap 52A is composed of a transfer transistor 41A, an FD (Floating Diffusion) unit 42A, a selection transistor 43A, and a reset transistor 44A. Similarly, the second tap 52B is composed of a transfer transistor 41B, an FD unit 42B, a selection transistor 43B, and a reset transistor 44B.

Irradiation light modulated (1 cycle = 2T) so as to repeat irradiation on / off at irradiation time T is output from the light emitting unit 12, for example, as shown in FIG. 3, depending on the distance to the object. The reflected light is received by the photodiode 51 with a delay time ΔT. It is assumed that the waveform of the reflected light is the same as the emission waveform of the irradiation light except for the delay of the phase (delay time ΔT) according to the distance to the object.

The distribution signal DIMIX_A controls the on / off of the transfer transistor 41A, and the distribution signal DIMIX_B controls the on / off of the transfer transistor 41B. The distribution signal DIMIX_A in FIG. 3 is a signal having the same phase as the irradiation light, and the distribution signal DIMIX_B has a phase in which the distribution signal DIMIX_A is inverted.

Therefore, in FIG. 2, the electric charge generated by the photodiode 51 receiving the reflected light is transferred to the FD unit 42A according to the distribution signal DIMIX_A while the transfer transistor 41A is on, and is transferred to the transfer transistor according to the distribution signal DIMIX_B. While 41B is on, it is transferred to the FD unit 42B. As a result, the electric charges transferred via the transfer transistor 41A are sequentially accumulated in the FD section 42A and transferred via the transfer transistor 41B during a predetermined period in which the irradiation light of the irradiation time T is periodically irradiated. The electric charge is sequentially accumulated in the FD unit 42B.

Then, when the selection transistor 43A is turned on according to the selection signal ADDRESS DECODE_A after the period for accumulating the electric charge, the electric charge accumulated in the FD unit 42A is read out via the signal line 53A and corresponds to the amount of the electric charge. The detection signal A is output from the light receiving unit 15. Similarly, when the selection transistor 43B is turned on according to the selection signal ADDRESS DECODE_B, the electric charge accumulated in the FD unit 42B is read out via the signal line 53B, and the detection signal B corresponding to the amount of the electric charge is transmitted from the light receiving unit 15. It is output. Further, the electric charge stored in the FD section 42A is discharged when the reset transistor 44A is turned on according to the reset signal RST_A, and the electric charge stored in the FD section 42B is discharged when the reset transistor 44B is turned on according to the reset signal RST_B. Will be done.

In this way, the pixel 31 distributes the electric charge generated by the reflected light received by the photodiode 51 to the first tap 52A or the second tap 52B according to the delay time ΔT, and outputs the detection signal A and the detection signal B to the pixels. Output as data.

The signal processing unit 16 calculates the depth value based on the detection signal A and the detection signal B supplied as pixel data from each pixel 31. As a method for calculating the depth value, there are a 2Phase method using two types of phase detection signals and a 4Phase method using four types of phase detection signals.

The 2Phase method and the 4Phase method will be explained.

In the 4Phase method, as shown in FIG. 4, the light receiving unit 15 transmits the reflected light at the light receiving timings shifted by 0 °, 90 °, 180 °, and 270 ° with respect to the irradiation timing of the irradiation light. Receive light (exposure). More specifically, the light receiving unit 15 receives light with the phase set to 0 ° with respect to the irradiation timing of the irradiation light in a certain frame period, and receives light with the phase set to 90 ° in the next frame period. In the frame period, the phase is set to 180 ° to receive light, and in the next frame period, the phase is set to 270 ° to receive light.

In this way, the phase of the light receiving timing that is preset with reference to the irradiation timing of the irradiation light is referred to as the set phase. The set phase of 0 °, 90 °, 180 °, or 270 ° represents the set phase of the pixel 31 at the first tap 52A unless otherwise specified. Since the second tap 52B has a phase inverted from that of the first tap 52A, when the first tap 52A has a set phase of 0 °, 90 °, 180 °, or 270 °, the second tap 52B has a phase opposite to that of the first tap 52A, respectively. , 180 °, 270 °, 0 °, or 90 °.

FIG. 5 is a diagram showing the exposure periods of the first tap 52A of the pixel 31 in each set phase of 0 °, 90 °, 180 °, and 270 ° side by side so that the difference in the set phase can be easily understood.

As shown in FIG. 5, the reflected light is received by the light receiving unit 15 in a state in which the phase is delayed from the irradiation timing of the irradiation light by the time ΔT corresponding to the distance to the object. The phase difference generated according to the distance to the object is also referred to as a distance phase to distinguish it from the set phase.

In the first tap 52A, the detection signal obtained by receiving the irradiation light and the same phase setting (set phase 0 °) I _A detection signal I _{A (0),} the irradiation light 90 degrees shifted by the set phase (phase setting 90 detection signal _I a (90 a detection signal _{I a} obtained by receiving in °)), the detection signal of the detection signals _{I a} obtained by receiving the irradiation light 180 degrees shifted by the set phase (phase setting 180 °) I _a (180), the detection signal _I a (270) a detection signal _{I a} obtained by receiving the irradiation light and the 270-degree shifted by the set phase (phase setting 270 °), is referred to as.

Although not shown, the second tap 52B, detection signal I _{B (0)} to detection signals I _B obtained by receiving the same phase setting and illumination light (set phase 0 °), the irradiation light 90 detection signal a detection signal I _B obtained by receiving in degrees staggered phase setting (set phase _{90 °) I B (90)} , is received by the irradiation light 180 degrees shifted by the set phase (phase setting 180 °) to give detection signal _I B a detection signal _{I B} to be (180), the detection signal _I B a detection signal _{I B} obtained by receiving the irradiation light and the 270-degree shifted by the set phase (phase setting 270 °) (270), referred to as I will decide.

FIG. 5 is a diagram illustrating a method of calculating the depth value d by the 2Phase method and the 4Phase method.

In the Indirect ToF method, the depth value d can be obtained by the following equation (1).

In equation (1), c is the speed of light, ΔT is the delay time, and f is the modulation frequency of light. Further, φ in the equation (1) represents the phase shift amount [rad] of the reflected light, that is, the metric phase, and is represented by the following equation (2).

In 4Phase method, I of equation (2), Q is set phase 0 °, 90 °, 180 °, and 270 detect signals _I A (0) obtained in ° to _I A (270) and the detection signal _{I B} (0) through using _I B (270), is calculated by the following formula (3). I and Q are signals obtained by converting the phase of the sine wave from polar coordinates to a Cartesian coordinate system (IQ plane), assuming that the change in brightness of the irradiation light is a sine wave.
I = c _0- c ₁₈₀
_{_{_{= (I A (0) -I}}} B (0)) - (I A (180) -I B (180))
Q = c _90- c ₂₇₀
_{_{_{= (I A (90) -I}}} B (90)) - (I A (270) -I B (270)) ·········· (3)

In 4Phase method, for example, of the formula _{_{(3) "I A (0}} ) -I A (180)" Ya _{_{"I A (90) -I A}} (270)" as the detection of anti-phase at the same pixel By taking the difference between the signals, it is possible to remove the characteristic variation between the taps existing in each pixel, that is, the fixed pattern noise.

On the other hand, in the 2Phase method, setting the phase 0 °, 90 °, 180 °, and 270 detect signals _I A (0) obtained in ° to _I A (270) and the detection signal _I B (0) to _I B (270 ), The depth value d to the object can be obtained by using only two set phases that are orthogonal to each other. For example, when used with the set phase 0 ° of the detection signals _I A (0) and _I B (0), setting the phase 90 ° of the detection signals _I A (90) and _I B (90), formula (2) I and Q are given by the following equation (4).
_{_{_{I = c 0 -c 180 = (}}} I A (0) -I B (0))
_{_{_{_{Q = c 90 -c 270 = (}}}} I A (90) -I B (90)) ·········· (4)

For example, when used with the set phase 180 ° of the detection signals _I A (180) and _I B (180), sets the phase 270 ° of the detection signals _I A (270) and _I B (the 270), formula (2) I and Q are given by the following equation (5).
_{_{_{I = c 0 -c 180 = -}}} (I A (180) -I B (180))
_{_{_{_{Q = c 90 -c 270 = -}}}} (I A (270) -I B (270)) ······· (5)

In the 2Phase method, the characteristic variation between taps existing in each pixel cannot be removed, but the depth value d to the object can be obtained only from the detection signals of the two set phases, which is twice that of the 4Phase method. Distance measurement can be performed at the frame rate.

In other words, if the characteristic variation between taps existing in each pixel can be properly corrected, the distance to the object can be measured accurately and at high speed by adopting the 2Phase method.

As a method for correcting the characteristic variation between taps, for example, assuming an offset c ₀ and a gain c ₁ as correction parameters for correcting the characteristic variation between taps, the offset c ₀ and the gain c ₁ are set by the least squares method. There is a way to calculate.

Specifically, the light receiving period of the first tap 32A and the second tap 32B of each pixel 21, since it is 180 ° out of phase, and offset c ₀ and the gain c _1, detection signal I _{A (0)} to between the I _a (270) and the detection signal _I B (0) to _I B (270), the following relationship holds in the ideal conditions.
_{_{I A (0) = c 0}} + c 1 · I B (180)
_{_{I A (90) = c 0}} + c 1 · I B (270)
_{_{I A (180) = c 0}} + c 1 · I B (0)
_{_{I A (270) = c 0}} + c 1 · I B (90) ····· (6)

The matrices A, x, and y,

Then, since the equation (6) can be expressed as y = Ax, the matrix x, that is, the offset c ₀ and the gain c ₁ can be calculated by the least squares method and the following equation (7). can.

However, when the following equation (8) appears in the denominator of the equation (7), the value of the equation (8) approaches zero as much as possible when the amplitude of the emission waveform of the irradiation light that controls the irradiation timing is small. It becomes a value. For example, when the amplitude of the emission waveform becomes 1/100, the value of the equation (8) becomes about 1/8000. Therefore, the calculation _{of the offset c 0} and the gain c ₁ becomes unstable when the reflectance is low, the object is far away, or the amount of light emitted is reduced to reduce power consumption.

Therefore, the signal processing unit 16 of the distance measuring sensor 13 in FIG. 1 more appropriately corrects the characteristic variation between taps and calculates the distance to the object using the 2Phase method.

<3. First configuration example of signal processing unit>
FIG. 7 is a block diagram showing a first configuration example of the signal processing unit 16 of the distance measuring sensor 13 of FIG.

The signal processing unit 16 of FIG. 7 includes a model determination unit 71, a characteristic calculation unit 72, a signal correction unit 73, and a distance calculation unit 74.

The luminance waveform detected by each pixel 31 of the light receiving unit 15 is a convolution of the emission waveform of the irradiation light output by the light emitting unit 12 and the exposure waveform when each pixel 31 of the light receiving unit 15 exposes (receives light). It becomes a waveform.

Therefore, the model determination unit 71 assumes (predicts) the shape of the emission waveform of the irradiation light output by the light emitting unit 12 and the shape of the exposure waveform when each pixel 31 of the light receiving unit 15 exposes (receives light). Then, a model (luminance model) of the brightness waveform observed by the light receiving unit 15 is determined.

FIG. 8 shows an example of a luminance model determined by the model determination unit 71.

For example, as shown in FIG. 3, the model determination unit 71 assumes a square wave as the shape of the emission waveform of the irradiation light and assumes a square wave as the shape of the exposure waveform as the first luminance model (model 1). Then, it is assumed that the luminance waveform observed by the light receiving unit 15 is a triangular wave.

Further, for example, the model determination unit 71 assumes a sine wave as the shape of the emission waveform of the irradiation light as the second luminance model (model 2), assumes a square wave as the shape of the exposure waveform, and observes with the light receiving unit 15. It is assumed that the luminance waveform to be generated is a sine wave.

Alternatively, in the model determination unit 71, when the shape of the emission waveform of the irradiation light and the shape of the exposure waveform cannot be assumed, the luminance waveform observed by the light receiving unit 15 is a harmonic as the third luminance model (model 3). Is assumed to be.

The model determination unit 71 determines a luminance model, which is a model of the luminance waveform observed by the light receiving unit 15, from among a plurality of luminance models as shown in FIG. 3, and supplies the luminance model to the characteristic calculation unit 72. If the emission waveform is known by the initial setting by the user, the brightness model may be determined based on the set value.

_{The characteristic calculation unit 72 has offset c 0} and gain, which are correction parameters for correcting characteristic variations between taps of each pixel 31 based on a luminance waveform (luminance model) corresponding to the shape of the emission waveform and the shape of the exposure waveform. Calculate c _1. The calculated offset c ₀ and gain c ₁ are supplied to the signal correction unit 73.

For example, when the model determination unit 71 determines the first model as the luminance model, the characteristic calculation unit 72 calculates the offset c ₀ and the gain c _{1 which are the correction parameters, assuming that the luminance waveform is a triangular wave.} do.

Further, when the model determination unit 71 determines the second model as the luminance model, the characteristic calculation unit 72 assumes that the luminance waveform is a sine wave, and sets the offset c ₀ and the gain c _{1 which are correction parameters.} calculate.

Further, when the model determination unit 71 determines the third model as the luminance model, the characteristic calculation unit 72 assumes that the luminance waveform is a harmonic and sets the offset c ₀ and the gain c _{1 which are correction parameters.} calculate.

The detailed calculation method of the correction parameter when the luminance waveform is assumed to be a triangular wave, a sine wave, or a harmonic wave will be described later.

Signal correcting unit 73 uses the offset c ₀ and a gain c ₁ calculated by the characteristic calculating section 72, the detection signal I _B of the detection signals I _{A (θ)} or the second tap 52B of the first tap 52A _(theta) Correct one of the signals.

For example, the signal correcting unit 73 in accordance with the above equation (6), by correcting the detection signal _I B of the second tap 52B of the phase setting 0 ° (0), detection of the first tap 52A of the phase setting 180 ° By generating the signal I " _A _{(180) and correcting the detection signal I B} (0) of the second tap 52B having the set phase 90 °, the _{detection signal I" A} (0) of the first tap 52A having the set phase 270 ° 180) is generated. In the following, the detection signal I (θ) converted using the correction parameter is represented by adding “” ”like the detection signal I” (θ).
_{I "A (180) = c} 0 + c 1 · I B (0)
_{I "A (270) = c} 0 + c 1 · I B (90)

The distance calculation unit 74 is the distance to the object by the 2 Phase method based on the phase images of the two set phases having an orthogonal relationship, specifically, the corrected pixel data supplied from the signal correction unit 73. Calculate the depth value. Then, the distance calculation unit 74 generates a depth map in which the depth value is stored as the pixel value of each pixel 31 and outputs the depth map to the outside of the module.

The light receiving unit 15 of the distance measuring sensor 13 may sequentially change the set phase to 0 °, 90 °, 180 °, and 270 ° to receive light, and for example, only the set phases of 0 ° and 90 °. The light reception may be performed so that the above steps are alternately repeated. The signal processing unit 16 can generate and output a depth map using two adjacent (two set phases) phase images.

<4. Correction parameter calculation process when the luminance waveform is a triangular wave>
Next, the calculation of the correction parameter by the characteristic calculation unit 72 when the first model is determined by the model determination unit 71 and a triangular wave is assumed as the luminance model will be described.

The characteristic calculation unit 72 is supplied with four phase images obtained by sequentially setting the set phase θ to 0 °, 90 °, 180 °, and 270 ° from the light receiving unit 15 (FIG. 1). Since the detection signal I _{A (θ)} and I _B of the reflected light received by the set phase theta in the first tap 52A and the second tap 52B of the pixels 31 of the pixel array 32 _(theta) correspond to the function representing the luminance waveform , the detection signal _{I a} (θ) and _{I B} the (theta), also referred to as the intensity function _{I a} (θ) and _{I B} (theta).

9, when the luminance model determined by the model determination unit 71 is a triangular wave of the first model, the intensity function _I A (0) to _I A (270) and the intensity function _I B (0) to _{I B} The conceptual diagram of (270) is shown.

Characteristic calculation unit

72, 0 ° setting phase, 90 °, 180 °, and, 270 ° of 4 luminance function _{I A} of the first tap 52A of each pixel 31 is detected by the light receiving portion 15 are sequentially set to the phase _{_{(0), I a (90}} ), and I a (180), and _I a (270), the intensity function _I B of the second tap _{52B (0), I B (} 90), I B (180), and based on the I B _(270), calculates the offset c ₀ and the gain c ₁ is a correction parameter for correcting the variation in characteristics between taps.

If the brightness model is a triangular wave of the first model, the intensity function I _A detected by the first tap 52A _(theta), such as shown in FIG. 9, the central center _A, the amplitude # 038 _A, by the offset offset _A Follow the specified triangular wave.

Similarly, the intensity function I _B detected by the second tap 52B _(theta), such as shown in FIG. 9, the central center _B, the amplitude # 038 _B, according to the triangular wave is defined by an offset offset _B.

In this case, the offset c ₀ and the gain c ₁ can be calculated by the following equations (9) and (10).

Here, the amplitude # 038 _A of the triangular wave of the luminance function I _A detected by the first tap 52A _{_(theta),} and the amplitude # 038 _B of the triangular wave of the luminance function I _B detected by the second tap 52B _(theta) is For example, as described in the literature "M. Schmidt." Spatiotemporal Analysis of Range Imagery ". Dissertation, Department of Physics and Astronomy, University of Heidelberg, 2008.", the following equations (11) and (12) ) Can be calculated.

Further, since the brightness function I (θ) of the triangular wave can be expressed by the equation (13), the equations (14) to (17) obtained as θ = 0 °, 90 °, 180 °, or 270 °. Therefore, the center of the triangular wave can be obtained by Eq. (18).

Thus, the triangular wave of the center center _A of the intensity function _{I A} of the first tap 52A (theta), and the central center _B of the triangular wave of the luminance function _{I B} of the second tap 52B (theta), respectively, equation (19) and It can be calculated by the formula (20).

Triangular wave offset offset _A of the intensity function I _A of the first tap 52A _{_(theta),} and the offset offset _B of the triangular wave of the luminance function I _B of the second tap 52B _(theta), so corresponds to the minimum value of the triangular wave, It can be obtained by subtracting the amplitude amp from the center center of the triangular wave as shown below.
_{_{_{offset A = center A -amp A ··········}}} (21)
_{_{_{offset B = center B -amp B ··········}}} (22)

Characteristic calculating unit 72, assuming a triangular wave as a brightness model, the amplitude # 038 _A of the triangular wave of the detected intensity function I _A first tap 52A _{_(theta),} and the intensity function I detected by the second tap 52B _{The amplitude amp B} of the triangular wave of _B (θ) is calculated by the above equations (11) and (12), and the offset offset _A _{and the offset offset B} are calculated by the above equations (21) and (22). calculate. Then, the characteristic calculation unit 72 calculates the offset c ₀ and the gain c ₁ by the above equations (9) and (10).

FIG. 10 shows a conceptual diagram of correction processing performed by the signal correction unit 73 when the luminance model is assumed to be a triangular wave of the first model.

Signal correction unit 73, by correcting using the offset c ₀ and the gain c ₁ a detection signal I _{B (0)} of the second tap 52B of the phase setting 0 °, the detection of the first tap 52A of the phase setting 180 ° The signal I " _A (180) is generated. Further, the signal correction unit 73 corrects the detection signal I _B (90) of the second tap 52B having the set phase of 90 ° by using the _{offset c 0} and the gain c _1. _{Therefore, the detection signal I "A} (270) of the first tap 52A having the set phase of 270 ° is generated.

As a result, the four-phase detection signals I (θ) are aligned, so the depth value can be calculated using equations (1), (2), and (4).

<5. Correction parameter calculation process when the luminance waveform is a sine wave>
Next, the calculation of the correction parameter by the characteristic calculation unit 72 when the second model is determined by the model determination unit 71 and a sine wave is assumed as the luminance model will be described.

When a sine wave is assumed as the luminance model, the _{arithmetic expressions for obtaining the amplitude amp A} and the amplitude amp _B are different from the above-mentioned equations (11) and (12) for the triangular wave. _{Specifically, the amplitude amp A} and the amplitude amp _B when the luminance model is a sine wave are calculated by the following equations (23) and (24).

Characteristic calculating unit 72, assuming a sin wave as a brightness model, the amplitude # 038 _A of sin wave of the detected intensity function I _A first tap 52A _{_(theta),} and the luminance detected by the second tap 52B the amplitude # 038 _B of sin wave function _{I B} (theta), is calculated by the above equation (23) and (24).

Moreover, the characteristic calculating section 72, the intensity function _{I A} (theta) center center _A of sin wave, and, a central center _B of sin waves of the intensity function _{I B} (theta), the above equation (19) and ( Calculated according to 20). Furthermore, characteristic calculation unit 72, the intensity function _{I A} (theta) of the sin wave offset offset _A, and the offset offset _B of the sin wave of the intensity function _{I B} (theta), the above equation (21) and ( Calculated according to 22). Then, the characteristic calculation unit 72 calculates the offset c ₀ and the gain c ₁ by the above equations (9) and (10).

11, assuming that the brightness model is a sin wave of the second model, the intensity function _I A (0) to _I A (270) and the intensity function _I B (0) to _I B of the (270) A conceptual diagram is shown.

FIG. 12 shows a conceptual diagram of correction processing performed by the signal correction unit 73 when the luminance model is assumed to be a sine wave of the second model.

Even if the emission waveform is designed to be rectangular, the actual emission waveform may be distorted from the rectangle depending on the time constant of the emission circuit, the modulation frequency of the emission waveform and the exposure waveform, and the like. In that case, it may be closer to the actual luminance model to assume the luminance model as a sine wave instead of assuming it as a triangular wave, and calculate the offset c ₀ and gain c ₁ by assuming the luminance model as a sine wave. In some cases, the calculation result of the distance is better.

<6. Correction parameter calculation process when the luminance waveform is a harmonic>
Next, the calculation of the correction parameter by the characteristic calculation unit 72 when the third model is determined by the model determination unit 71 and harmonics are assumed as the luminance model will be described.

The characteristic calculation unit 72 calculates the correction parameter using machine learning.

First, the ranging module 11 acquires four phase images with the set phases set to 0 °, 90 °, 180 °, and 270 ° in various scenes (measurement targets). The four phase images of the acquired various scenes are accumulated in the characteristic calculation unit 72.

The characteristic calculation unit 72 has a neural network learner having the same configuration for each of the first tap 52A and the second tap 52B.

FIG. 13 is a diagram illustrating a learning process for learning the luminance model of the harmonics detected by the first tap 52A.

The characteristic calculation unit 72 has a learner 81A that learns a brightness model of harmonics detected by the first tap 52A. Weighting factor of each node learning unit 81A ^{_{_{is, W A = {w 1,}}} w 2, w 3, ···} is.

The luminance function I _{'A (theta),} assuming the luminance model harmonic, expressed by Fourier series expanded following equation (25).

Equation (25), the luminance function I _{'A (theta)} on the assumption that higher harmonics, centered c ^A ₀ harmonics, and cos function from primary to M order, from the primary to M order It is represented by the sin function. Here, the order M (M is a natural number) is determined in advance.

The input v ^A _in of the learner 81A is the first tap 52 A of the predetermined pixel 31 in the four phase images of the accumulated predetermined scenes of set phases 0 °, 90 °, 180 °, and 270 °. brightness value _{_{I a (0), I a}} (90), are I a (180), and _I a (270). That is, the input ^{_{_{_{v A in = {I A (}}}} 0), I A (90), I A (180), and _I A (270)} is.

Output v ^A _out of the learning unit 81A, the coefficient of cos function of M dimensions of formula (25) in which the luminance function I _'A (theta) Fourier series expansion ^{_{{a A 1, ···, a}} A M} and, It is the coefficient of the M-dimensional sin function {b ^A ₁ , ..., b ^A _M }. That is, the output v ^A _out = {a ^A ₁ , ..., a ^A _M , b ^A ₁ , ..., b ^A _M }.

Advance to the input v ^A _in some weight coefficient W ^A is set learner 81A as an initial ^{_{_{_{value, v A in = {I A}}}} (0), I A (90), I A (180), and _I A ( When 270)} is input, the output v ^A _out = {a ^A ₁ , ..., a ^A _M , b ^A ₁ , ..., b ^A _M } is obtained.

The characteristic calculation unit 72 puts the output v ^A _out = {a ^A ₁ , ..., a ^A _M , b ^A ₁ , ..., b ^A _M } obtained from the learner 81A into equation (25). By substituting the average value of the input v ^A _in of various accumulated scenes into the center c ^A ₀ of the harmonic, the luminance function I _A '(θ) assumed to be the harmonic is restored.

If restored harmonic intensity function I _'A (θ) is representative good intensity function _{I A} (theta), it sets the phase 0 °, 90 °, 180 ° , and, the intensity function I when the 270 ° 'is the value of _a (theta), the luminance value _{I' a (0), I} 'a (90), I' a (180), and I _'a (270), the luminance as input learning device 81A is the value of the function _{I a (θ),} the brightness value _{_{I a (0), I a}} (90), I a (180), and _I a (270)} and should be the same.

Therefore, characteristic calculation unit 72 performs learning such that the difference between the value of the intensity function I _{'A (θ)} of the input v ^A _in the output v ^A _out of the learning device 81A is reduced. Specifically, the characteristic calculating section 72, as the first evaluation function L ^A ₁ represented by the following formula (26) decreases, the weighting factor of each node learning unit 81A W ^{^A} = {w ^A ₁ , W ^A ₂ , w ^A ₃ , ...} are updated. W ^A 'of formula (27) represents the weighting coefficients after updating, the eta ^A is a coefficient representing the learning rate.

More weighting factor W update of ^A, the luminance value _I A of the first tap 52A of the same pixel in the four phase images of the stored various scenes _{_{(0), I A (90}} ), I A (180) , and by repeating a predetermined number of times by using the _I a (270), the intensity function I _'a harmonics of the formula (25) (θ) is obtained.

Although not shown, the characteristic calculation unit 72 performs the same learning for the second tap 52B.

In other words, expressed by the luminance function I _'B (theta), assuming the luminance model harmonic Fourier series expanded following equation (28).

Equation (28), the luminance function I _'B (theta) on the assumption that higher harmonics, centered c ^B ₀ harmonic, and cos function from primary to M order, from the primary to M order It is represented by the sin function. Here, the order M (M is a natural number) is determined in advance.

Characteristic calculation unit 72, a weighting factor of each node learning unit 81B (not shown) to learn the luminance model of harmonics detected by the second tap ^{^{52B, W B = {w B}} 1, w B 2, If w ^B ₃ , ...}, _{the set phases of the predetermined scenes accumulated in} ^{the input v B} in of the learner 81B are within the four phase images of 0 °, 90 °, 180 °, and 270 °. the luminance value of the second tap 52B of predetermined pixels _{_{31 I B (0), I}} B (90), I B (180), and _I inputs B (270). That is, the input ^{_{_{_{v B in = {I B (}}}} 0), I B (90), I B (180), and _I B (270)} is.

Output v ^B _out of the learning device 81B, the coefficients of cos function of M dimensions of formula (28) in which the luminance function I _'B (theta) Fourier series expansion ^{_{{a B 1, ···, a}} B M} and, The coefficient of the M-dimensional sin function {b ^B ₁ , ..., b ^B _M }. That is, the output v ^B _out = {a ^B ₁ , ..., a ^B _M , b ^B ₁ , ..., b ^B _M }.

The input v ^B _in the preset learner 81B some weighting factor W ^B is an initial ^{_{_{value, v B in = {I B}}} (0), I B (90), I B (180), and I _B ( When 270)} is input, the output v ^B _out = {a ^B ₁ , ..., a ^B _M , b ^B ₁ , ..., b ^B _M } is obtained.

The characteristic calculation unit 72 puts the output v ^A _out = {a ^B ₁ , ..., a ^B _M , b ^B ₁ , ..., b ^B _M } obtained from the learner 81B into the equation (28). substituted and, in the center c ^B ₀ harmonic, by substituting an average value of the input v ^B _in the stored various scenes, to restore the luminance function I _B were assumed harmonic '(theta).

The characteristic calculating section 72 performs learning such that the difference between the intensity function I _'B (θ) of the input v ^B _in the output v ^B _out of the learning device 81B decreases. Specifically, the characteristic calculating section 72, as the first evaluation function L ^B ₁ represented by the following formula (29) decreases, the weighting factor of each node learning unit 81B W ^{^B} = {w ^B ₁ , W ^B ₂ , w ^B ₃ , ...}. W ^B 'is of formula (30) represents the weighting coefficients after updating, eta ^B is a coefficient representing the learning rate.

More updates weighting factors W ^B, the luminance value _I B of the second tap 52B of the same pixel in the four phase images of the stored various scenes _{_{(0), I B (90}} ), I B (180) , and by repeating a predetermined number of times by using the _I B (270), the intensity function I _'B harmonics of the formula (28) (θ) is obtained.

Incidentally, the intensity function I _'A (θ) and the intensity function I' _B to be restored (theta), since the center of the amplitude is considered to differ with the characteristics of the same light emission waveform as exposure waveform, the intensity function I '( It is considered that the shape of the function obtained by whitening θ) is the same. Therefore, as a constraint condition when updating the weighting coefficient W of each node of the learner 81, the first evaluation functions L ^A ₁ and L ^B ₁ described above are subjected to the second evaluation represented by the following equation (31). A function L ₂ may be added to obtain a weighting coefficient W such that the _{second evaluation function L 2 is also small.} The update of the weighting coefficient W is represented by the equation (32).

FIG. 14 shows a conceptual diagram of an operation for obtaining the _{second evaluation function L 2.}

Thus, 'and _{A (theta),} intensity function I of the detected reflected light at the second tap 52B' luminance function I of the reflected light detected by the first tap 52A _B (θ) is estimated.

Next, characteristic calculation unit 72 uses 'and _A (theta), intensity function I of the second tap 52B' luminance function I of the first tap 52A estimated and _B (theta), the following equation (33) The gain c ₁ is calculated by the formula (34), and the offset c ₀ is calculated by the equation (34).

When harmonics are assumed as the luminance model in the model determination unit 71, the characteristic calculation unit 72 can calculate the correction parameters using machine learning as described above.

Note that the method of calculating harmonics when the luminance model is assumed as harmonics is not limited to machine learning, and other methods may be used. For example, a method as disclosed in the document "Marvin Lindner and Andreas Kolb," Lateral and Depth Calibration of PMD-Distance Sensors, International Symposium on Visual Computing ", 2006." may be adopted. The technique disclosed in this document captures multiple known depths and obtains a harmonic component from the difference between the known depth and the actually measured depth, providing a look-up table for depth correction. The method of generation is disclosed.

By assuming the brightness model as a harmonic, even if the brightness model is unknown, the correction parameters can be calculated more accurately than with a simple model.

<7. Flowchart of measurement process by distance measurement module>
Next, the measurement process by the distance measuring module 11 of FIG. 1 will be described with reference to the flowchart of FIG. This process is started, for example, when the control unit of the host device in which the distance measuring module 11 is incorporated instructs the start of measurement.

First, in step S1, the control unit 14 of the distance measuring sensor 13 determines whether to estimate the correction parameter for correcting the characteristic variation between the taps of each pixel 31 of the pixel array 32 of the light receiving unit 15.

For example, in calibration before shipment, it is determined that the correction parameters are estimated. Further, for example, in the first measurement in which the distance measuring module 11 is activated, the correction parameter can be estimated without fail, or the correction parameter can be estimated every predetermined number of measurements.

If it is determined in step S1 that the correction parameters are to be estimated, the process proceeds to step S2, and the distance measuring module 11 estimates the correction parameters for estimating the _{offset c 0} and the gain c _{1 which are the correction parameters of each pixel 31.} Execute the process.

On the other hand, if it is determined in step S1 that the correction parameter is not estimated, the correction parameter estimation process in step S2 is skipped, and the process proceeds to step S3. When the correction parameter is not estimated, the correction parameter has already been set.

In step S3, the distance measuring module 11 executes a distance measurement process for measuring the distance to the object by the 2 Phase method using the correction parameter for correcting the characteristic variation between the taps of each pixel 31, and ends the process.

FIG. 16 is a detailed flowchart of the correction parameter estimation process executed in step S2 of the measurement process of FIG.

In this process, first, in step S11, the control unit 14 sets the phase difference (set phase) between the light emitting waveform and the exposure waveform to 0 °, and supplies the light emitting control signal to the light emitting unit 12 and the light receiving unit 15.

In step S12, the ranging module 11 emits the irradiation light and receives the reflected light reflected and returned by the object. That is, the light emitting unit 12 emits light while modulating at a timing corresponding to the light emission control signal supplied from the control unit 14, and irradiates the object with the irradiation light. The light receiving unit 15 receives the reflected light from the object. The light receiving unit 15 supplies pixel data composed of detection signals according to the amount of received reflected light to the signal processing unit 16 in units of pixels 31 of the pixel array 32.

In step S13, the control unit 14 determines whether or not four phase images having the set phases of 0 °, 90 °, 180 °, and 270 ° have been acquired.

If it is determined in step S13 that the acquisition of the four phase images has not yet been performed, the process proceeds to step S14, and the control unit 14 updates the set phase to a value incremented by 90 ° from the current value. .. After step S14, the process returns to step S12, and steps S12 and S13 described above are repeated.

On the other hand, when it is determined in step S13 that four phase images have been acquired, the process proceeds to step S15, and the characteristic calculation unit 72 of the signal processing unit 16 changes the shape of the emission waveform and the shape of the exposure waveform. _{Based on the corresponding luminance waveform, the offset c 0} and the gain c ₁ which are the correction parameters for correcting the characteristic variation between the taps of each pixel 31 are calculated. The luminance model is determined before the process of step S15, and is supplied from the model determination unit 71 to the characteristic calculation unit 72.

In step S15, the offset c ₀ and the gain c ₁ are described above for each of the cases where a triangular wave is assumed as the luminance model, a sine wave is assumed as the luminance model, and a harmonic is assumed as the luminance model. It is calculated by the calculation method. The calculated offset c ₀ and gain c ₁ are supplied to the signal correction unit 73, the correction parameter estimation process of FIG. 16 is completed, and the process returns to the measurement process of FIG.

In the correction parameter estimation process of FIG. 16, the correction parameters (offset c ₀ and gain c ₁ ) were calculated using only four phase images capable of generating one depth map, but the influence of noise is reduced. Therefore, even if eight or more phase images corresponding to a plurality of depth maps are used and the summation average of the plurality of correction parameters obtained as a result is used to calculate the final correction parameter. good.

FIG. 17 is a detailed flowchart of the distance measurement process by the 2 Phase method executed in step S3 of the measurement process of FIG.

In this process, first, in step S31, the control unit 14 sets the phase difference (set phase) between the light emitting waveform and the exposure waveform to 0 °, and supplies the light emitting control signal to the light emitting unit 12 and the light receiving unit 15.

In step S32, the ranging module 11 emits the irradiation light and receives the reflected light reflected and returned by the object. That is, the light emitting unit 12 emits light while modulating at a timing corresponding to the light emission control signal supplied from the control unit 14, and irradiates the object with the irradiation light. The light receiving unit 15 receives the reflected light from the object. The light receiving unit 15 supplies pixel data composed of detection signals according to the amount of received reflected light to the signal processing unit 16 in units of pixels 31 of the pixel array 32.

In step S33, the control unit 14 determines whether or not two phase images having the set phases of 0 ° and 90 ° have been acquired.

If it is determined in step S33 that the acquisition of the two phase images has not yet been performed, the process proceeds to step S34, and the control unit 14 sets the phase difference (set phase) between the emission waveform and the exposure waveform to 90 °. Set. After step S34, the process returns to step S32, and steps S32 and S33 described above are repeated.

On the other hand, if it is determined in step S33 that the two phase images have been acquired, the process proceeds to step S35, and the signal correction unit 73 of the signal processing unit 16 has the offset c _{0 calculated by the characteristic calculation unit 72.} and using the gain c _1, in accordance with the equation (6), to correct the detection signals _{I B} of the second tap 52B (theta).

Specifically, the signal correcting unit 73 uses the detection signal _I B of the second tap 52B of the pixels 31 acquired in set phases 0 ° and 90 ° (0) and _I B (90), the following Calculate the corrected I " _A (180) and I" _A (270).
_{I "A (180) = c} 0 + c 1 · I B (0)
_{I "A (270) = c} 0 + c 1 · I B (90)

In step S36, the distance calculation unit 74 uses the two phase images of the corrected set phases of 0 ° and 90 ° supplied from the signal correction unit 73 to reach the object for each pixel 31 of the pixel array 32. The depth value, which is the distance between, is calculated by the 2 Phase method. Then, the distance calculation unit 74 generates a depth map in which the depth value is stored as the pixel value of each pixel 31 and outputs the depth map to the outside of the module.

With the above, the correction parameter estimation process of FIG. 16 is completed, and the entire measurement process of FIG. 15 is also completed.

_{The process of calculating the offset c 0} and the gain c ₁ in step S15 of FIG. 16 _{, which is the process of calculating the offset c 0} and the gain c ₁ using machine learning when harmonics are assumed as the luminance model. Will be described in more detail with reference to the flowchart of FIG.

In the following steps S51 to S56, the processing of the learner 81A for learning the brightness model of the harmonics detected by the first tap 52A will be described, but the brightness of the harmonics detected by the second tap 52B will be described. The same applies to the learner 81B for learning the model.

First, in step S51, characteristic calculation unit 72 sets a predetermined initial value the weight coefficient W ^A learner 81A corresponding to the first tap 52A.

In step S52, the characteristic calculation unit 72 within four phase images of the set phases 0 °, 90 °, 180 °, and 270 ° of a predetermined one scene among the accumulated phase images of the plurality of scenes. the luminance value _I a of the first tap 52A of predetermined pixels _{31 (0), I a (} 90), I a (180), and extracts the _I a (270), the input v ^a _in the learning device 81A = _{_{{I a (0), I}} a (90), I a (180), and _I a (270)} and.

In step S53, the characteristic calculation unit 72 outputs the output according to ^{the input v A} _in ^{of the learner 81 A v A} _out = {a ^A ₁ , ..., a ^A _M , b ^A ₁ , ..., b ^A _M } To get.

In step S54, the characteristic calculation unit 72 formulates the output v ^A _out = {a ^A ₁ , ..., a ^A _M , b ^A ₁ , ..., b ^A _M } obtained from the learner 81A. By substituting into (25) and substituting the average value of the input v ^A _in ^{of various accumulated scenes into the center c A} ₀ of the harmonic, the luminance function I _A '(θ) assumed to be a harmonic. To restore.

In step S55, the characteristic calculating section 72, the restored luminance function _{I A} 'set phase 0 ° to (θ), 90 °, 180 °, and the luminance value obtained by substituting the 270 ° I' _A (0 _{), I 'a (90)} , I' and _a (180), and I _'a (270), the input ^{_{_{_{v a in = {I a (}}}} 0), I a (90), I a (180), and by comparing the I _a (270)}, as the evaluation function L ^a ₁ of the first of the formula (26) decreases, the weighting factor of each node learning unit 81A W ^{^a} = {w ^a ₁ , W ^A ₂ , w ^A ₃ , ...} are updated. Thus, the weighting coefficients after updating W ^A 'is calculated.

In step S56, the characteristic calculating section 72 determines whether performed predetermined number of times to update the weight coefficient W ^A.

Note that determination processes updating end of the weight coefficient W ^A of step S56, instead of determining whether performed predetermined number of times, 'luminance value I (theta)' _A restored luminance function I _A _(0), _{I 'a (90), I} ' a (180), and I 'and _a (270), the input ^{_{_{_{v a in = {I a (}}}} 0), I a (90), I a (180), and _{I a} (270)} and Kaya error becomes within the predetermined range, the difference between the weight coefficient W ^a of before and after the update and may be a criterion it is within the predetermined range.

In step S56, when it is determined that not yet updated weight coefficient W ^A prescribed number, processing returns to step S52, characteristic calculation unit 72, among the accumulated plural scenes phase images, still Four phase images of set phases 0 °, 90 °, 180 °, and 270 ° of a predetermined unselected scene are acquired, and the above-described processes of steps S52 to S56 are repeated.

On the other hand, in step S56, when it is determined that made the update of the weight coefficient W ^A prescribed number, the process proceeds to step S57. As described above, the processes of steps S51 to S56 are also executed for the learner 81B that learns the luminance model of the harmonics detected by the second tap 52B, in the same manner as the process of the learner 81A.

In step S57, the characteristic calculating section 72 uses 'and _{A (theta),} intensity function I of the second tap 52B' luminance function I of the first tap 52A which is calculated and _B (theta), the above Expression ( The offset c ₀ _{is calculated by 34), and the gain c 1} is calculated by the above equation (33).

This completes the process of calculating the _{offset c 0} and the gain c ₁ using machine learning.

According to the measurement process by the ranging module 11 using the first configuration example of the signal processing unit 16, the correction parameter for correcting the characteristic variation between the taps of each pixel 31 is set based on the determined luminance model (luminance waveform). By calculating, it is possible to more appropriately correct the characteristic variation between taps.

<8. Second configuration example of signal processing unit>
FIG. 19 is a block diagram showing a second configuration example of the signal processing unit 16 of the distance measuring sensor 13 of FIG.

In FIG. 19, the parts corresponding to the first configuration example shown in FIG. 7 are designated by the same reference numerals, and the description of the parts will be omitted as appropriate.

The signal processing unit 16 of FIG. 19 includes a model determination unit 71, a characteristic calculation unit 72, a signal correction unit 73, a distance calculation unit 74, and an optimum model selection unit 91.

That is, in the second configuration example of the signal processing unit 16, the optimum model selection unit 91 is further added to the first configuration example of the signal processing unit 16 shown in FIG. 7.

In the first configuration example shown in FIG. 7, the model determination unit 71 determines a predetermined one from a plurality of selectable luminance models, and the characteristic calculation unit 72 is supplied from the model determination unit 71. _{The correction parameters (offset c 0} and gain c ₁ ) were calculated based on one luminance model (luminance waveform).

On the other hand, in the second configuration example of FIG. 19, the model determination unit 71 supplies all the stored luminance models to the characteristic calculation unit 72, and the characteristic calculation unit 72 supplies all the luminance models (luminance). For the waveform), the correction parameters (offset c ₀ and gain c ₁ ) are calculated. Here, N brightness models are stored in the model determination unit 71, and the offset c ₀ and the gain c ₁ calculated for each of the N brightness models are set to (c ₀ ¹ , c ₁ ¹ ), (c). ₀ ² , c ₁ ² ), ..., (c ₀ ^N , c ₁ ^N ). _{The characteristic calculation unit 72 supplies the offset c 0} and the gain c ₁ calculated for each of the N luminance models to the optimum model selection unit 91.

The optimum model selection unit 91 sets the offset and gain of each of the N luminance models (c ₀ ¹ , c ₁ ¹ ), (c ₀ ² , c ₁ ² ), ..., (C ₀ ^N , c _1). ^{Using the} same four phase images with set phases of 0 °, 90 °, 180 °, and 270 °, which are the same as those for calculating _{N), the phase shift amount φ ref} of each pixel 31 is calculated by the 4 Phase method. _{That is, the phase shift amount φ ref} of each pixel 31 is calculated by the above-mentioned equations (3) and (2).

Further, the optimum model selection unit 91 sets the offset and gain (c ₀ ¹ , c ₁ ¹ ), (c ₀ ² , c ₁ ² ), ..., (C ₀ ^N , c ₁ ^N ), respectively. using equation (6), to produce "and a (180), the detection signal _I of the first tap 52A configuration phase 270 °" detection signal _I of the first tap 52A configuration phase 180 ° of the a (180).
_{I "A (180) = c} 0 + c 1 · I B (0)
_{I "A (270) = c} 0 + c 1 · I B (90)

Further, the optimum model selection unit 91 uses the 2Phase method, that is, the phase shift amounts φ (c ₀ ¹ , c ₁ ¹ ) and φ (c ₀ ^{2) of each pixel 31 according to the above equations (4) and (2).} , C ₁ ² ), ···, φ (c ₀ ^N , c ₁ ^N ) is calculated.

Then, the optimum model selection unit 91 has a phase shift amount φ _ref , a phase shift amount φ (c ₀ ¹ , c ₁ ¹ ), φ (c ₀ ² , c ₁ ² ), ..., φ (c ₀ ^N). , C ₁ ^N ) to select the optimum offset and gain pair (c ₀ ^opt , c ₁ ^opt). That is, the optimum model selection unit 91 calculates the following equation (35).

MSE [] in Eq. (35) is a function that calculates the mean square error in []. argmin is a function that finds the minimum MSE [] (c ₀ ⁿ , c ₁ ^n). Therefore, in equation (35), (c ₀ ^opt , c ₁ ^opt ) is set to minimize the mean square error of _{{φ ref} −φ (c ₀ ⁿ , c ₁ ⁿ _{)} (c 0} ⁿ , c ₁ ^n). Indicates that the decision is made.

The function for evaluating the error is not limited to the mean square error, and other functions may be used.

The optimum model selection unit 91 _{supplies the selected offset c 0} ^opt and gain c ₁ ^opt to the signal correction unit 73.

The processing of the signal correction unit 73 and the distance calculation unit 74 is the same as that of the first configuration example.

<9. Flowchart of measurement processing using the second configuration example>
FIG. 20 is a flowchart of the measurement process of the distance measuring module 11 of FIG. 1 when the second configuration example of the signal processing unit 16 is adopted. This process is started, for example, when the control unit of the host device in which the distance measuring module 11 is incorporated instructs the start of measurement.

First, in step S71, the control unit 14 of the distance measuring sensor 13 determines whether to estimate the correction parameter for correcting the characteristic variation between the taps of each pixel 31 of the pixel array 32 of the light receiving unit 15. The process of step S71 is the same as step S1 of the measurement process in the first configuration example described in the flowchart of FIG.

If it is determined in step S71 that the correction parameters are to be estimated, the process proceeds to step S2, and the ranging module 11 executes the correction parameter estimation process for estimating the correction parameters for all the luminance models.

That is, in the process of step S2 in the first configuration example, the correction parameter of one predetermined luminance model determined by the model determination unit 71 is estimated, whereas in the process of step S72 of the second configuration example, the process is The difference is that the model determination unit 71 estimates the correction parameters of all the luminance models stored. _{The characteristic calculation unit 72 supplies the offset c 0} and the gain c ₁ calculated for each of the N luminance models to the optimum model selection unit 91.

Then, after step S72, the process proceeds to step S73, and the optimum model selection unit 91 selects the optimum luminance model from the N luminance models. That is, by calculating equation (35), the optimum offset c ₀ ^opt and gain c ₁ ^opt are selected from _{the correction parameters (offset c 0} ⁿ and gain c ₁ ^{n) of each of the N luminance models.} NS.

On the other hand, if it is determined in step S71 that the correction parameter is not estimated, the processes of steps S72 and S73 are skipped, and the process proceeds to step S74.

In step S74, the distance measuring module 11 executes a distance measurement process for measuring the distance to the object by the 2 Phase method using the correction parameter for correcting the characteristic variation between the taps of each pixel 31, and ends the process.

According to the measurement process by the ranging module 11 using the second configuration example of the signal processing unit 16, the optimum luminance model is selected from various luminance models (luminance waveforms), and between the taps of each pixel 31. By calculating the correction parameter for correcting the characteristic variation, the characteristic variation between taps can be corrected more appropriately.

<10. Third configuration example of signal processing unit>
FIG. 21 is a block diagram showing a third configuration example of the signal processing unit 16 of the distance measuring sensor 13 of FIG.

In FIG. 21, the parts corresponding to the second configuration example shown in FIG. 19 are designated by the same reference numerals, and the description of the parts will be omitted as appropriate.

The signal processing unit 16 of FIG. 21 includes a model determination unit 71, a characteristic calculation unit 72, a signal correction unit 73, a distance calculation unit 74, an optimum model selection unit 91, and an evaluation unit 101.

That is, in the third configuration example of the signal processing unit 16, the evaluation unit 101 is further added to the second configuration example of the signal processing unit 16 shown in FIG.

The depth map calculated by the distance calculation unit 74 is supplied to the evaluation unit 101.

The evaluation unit 101 evaluates whether or not the correction parameter currently used is appropriate by using two depth maps that are continuous in the time direction. If the correction parameter is not a value obtained by appropriately correcting the characteristic variation between taps of the pixel 31, the characteristic variation appears in the depth map as fixed pattern noise. The evaluation unit 101 determines whether or not fixed pattern noise appears in the depth map.

When the evaluation unit 101 determines that fixed pattern noise appears in the depth map, the evaluation unit 101 instructs the model determination unit 71 to recalculate the correction parameters. When the evaluation unit 101 instructs the evaluation unit 101 to recalculate the correction parameters, the model determination unit 71 supplies all the stored luminance models to the characteristic calculation unit 72.

The fixed pattern noise evaluation process performed by the evaluation unit 101 will be described with reference to FIG. 22.

The depth map D _t-1 at time t-1 and the depth map D _{t at} time t are input to the evaluation unit 101.

Evaluation unit 101, initially, each time two consecutive in the direction depth map D _t-1 and D _t, to apply the spatial filter f for smoothing the image, the filtered depth map D _'t Generate _-1 and'D _t. Hereinafter, the depth map D _'t-1 and D' _t after filtering, referred to as a post-processing the depth map D _'t-1 and D' _t.
D' _t-1 = f (D _t-1 )
D' _t = f (D _t )

For the spatial filter f, for example, an edge preservation type filter such as a Gaussian filter or a bilateral filter can be adopted. By applying the spatial filter f, the influence of noise can be reduced from the depth map.

Next, the evaluation unit 101, the area position of the small region g _t-1 and g _t obtained by setting the area size in advance, respectively, are sequentially slid in the post-processing the depth map D _'t-1 and D' _t, the small the dispersion of the depth value d that is detected at each pixel in the region g, both the smallest small region g _t of the difference between the depth value d between two of the processed depth map, used in the evaluation of the fixed pattern noise as the representative small area gs' _t. This process can be expressed as follows.

In equation (36), V () represents the operator for the variance.

Next, the evaluation unit 101, the representative small area gs' _t subregions gs in the same area position, extracted from each of two depth map _{D t-1} and _{D t,} the small region _{gs t-1} and gs _t And.

The evaluation unit 101 calculates the following equation (37) using the small regions gs _t-1 and gs _t extracted from the two depth maps D _t-1 and D _t, _{respectively, and calculates the small region gs t-1.} It is determined whether or not the sum of the difference between gs _t _{and the variance of the small region gs t} is larger than the predetermined threshold value Th. When satisfying the condition of formula (37), the evaluation unit 101, a fixed pattern noise in the depth map D _t it has appeared, that is, determines that the correction parameter is not the value obtained by appropriately correcting the characteristic variation between taps.

<11. Flowchart of measurement processing using the third configuration example>
FIG. 23 is a flowchart of the measurement process of the distance measuring module 11 of FIG. 1 when the third configuration example of the signal processing unit 16 is adopted. This process is started, for example, when the control unit of the host device in which the distance measuring module 11 is incorporated instructs the start of measurement.

Since the processes of steps S101 to S104 of FIG. 23 are the same as the processes of steps S71 to S74 described with reference to FIG. 20, the description thereof will be omitted. The depth map calculated in step S104 is also supplied to the evaluation unit 101, and the evaluation unit 101 stores two depth maps D _t-1 and D _{t that} are continuous in the time direction.

In step S105, the evaluation unit 101 executes a fixed pattern noise evaluation process for evaluating whether or not the correction parameter currently used is appropriate, using two depth maps that are continuous in the time direction. Details of the fixed pattern noise evaluation process in step S105 will be described later with reference to the flowchart of FIG. 24.

In step S106, the evaluation unit 101 uses the result of the fixed pattern noise evaluation process to determine whether fixed pattern noise is generated in the _{depth maps D t-1} and D _t. Specifically, the evaluation unit 101 determines whether or not the conditional expression of the equation (37) is satisfied.

If the conditional expression of the equation (37) is satisfied in step S106 and it is determined that the fixed pattern noise is generated, the process proceeds to step S107, and the evaluation unit 101 informs the model determination unit 71 of the correction parameter. Instruct the calculation. The model determination unit 71 supplies all the stored luminance models to the characteristic calculation unit 72. The characteristic calculation unit 72 executes a correction parameter estimation process for estimating correction parameters for all luminance models. The process of step S107 after the recalculation of the correction parameter is instructed is the same as that of step S102 and step S72 of FIG.

Then, in step S108, the optimum model selection unit 91 selects the optimum brightness model from the N brightness models. The process of step S108 is the same as that of step S103 and step S73 of FIG.

On the other hand, if it is determined in step S106 that fixed pattern noise has not occurred, the processes of steps S107 and S108 are skipped, and the measurement process ends.

FIG. 24 is a detailed flowchart of the fixed pattern noise evaluation process executed in step S105 of FIG. 23.

First, in step S121, the evaluation unit 101 applies a spatial filter f for smoothing an image to each of _{the two continuous depth maps D t-1} and D _{t in the time direction, and the processed depth map D.} ' _t-1 and D' _t are generated.

In step S122, the evaluation unit 101, the processing after the depth map D within _'t-1 and D' _t, respectively, is slid by setting small regions _{g t-1} and _{g t,} a representative small region gs' _t Set. Small representative subregion gs' _t has the formula represented by (36), and the variance of depth value d in the small region g, the smallest both the difference between the depth value d between the two processed depth map It is an area _{g t.}

In step S123, the evaluation unit 101, the representative small area gs' _t subregions gs in the same area position, extracted from each of two depth map _{D t-1} and _{D t,} the small region _{gs t-1} and gs _{Let t} .

In step S124, the evaluation unit 101 uses the small regions gs _t-1 and gs _t _{extracted from the two depth maps D t-1} and D _t, respectively, to use the left side of the equation (37), that is, the small region gs. The sum of the difference between _t-1 and gs _t and the variance of the _{small region gs t is calculated.}

After step S124, the process proceeds to step S106 of FIG. 23, and it is determined whether or not the conditional expression of the equation (37) is satisfied.

This completes the measurement process of the distance measuring module 11 when the third configuration example of the signal processing unit 16 is adopted.

According to the distance measuring module 11 including any of the first configuration example to the third configuration example of the signal processing unit 16 described above, it is possible to more appropriately correct the characteristic variation between taps. The signal processing unit 16 may be configured by any of the above-mentioned first configuration example to third configuration example, or may be configured so that the first configuration example to the third configuration example can be selectively executed. _{By having such a signal processing unit 16, stable estimation of offset c 0} and gain c ₁ can be expected even when the signal-to-noise ratio of the signal is not sufficiently high at the timing of correcting the characteristic variation between taps.

<12. Distance measurement sensor chip configuration example>
FIG. 25 is a perspective view showing a chip configuration example of the distance measuring sensor 13.

For example, as shown in A of FIG. 25, the distance measuring sensor 13 can be composed of one chip in which a sensor die 151 as a plurality of dies (boards) and a logic die 152 are laminated.

The sensor die 151 is configured with a sensor unit 161 (as a circuit), and the logic die 152 is configured with a logic unit 162.

For example, a light receiving unit 15 is formed in the sensor unit 161. The logic unit 162 is formed with, for example, a control unit 14, a signal processing unit 16, input / output terminals, and the like.

Further, the distance measuring sensor 13 may be composed of three layers in which another logic die is laminated in addition to the sensor die 151 and the logic die 152. Of course, it may be composed of a stack of four or more dies (boards).

Alternatively, the ranging sensor 13 is composed of, for example, as shown in FIG. 25B, a first chip 171 and a second chip 172, and a relay board (interposer board) 173 on which they are mounted. You may.

For example, a light receiving portion 15 is formed on the first chip 171. A control unit 14, a signal processing unit 16, and the like are formed on the second chip 172.

The circuit arrangement of the sensor die 151 and the logic die 152 in A of FIG. 25 and the circuit arrangement of the first chip 171 and the second chip 172 in B of FIG. 25 are merely examples. Not limited to. For example, a signal processing unit 16 that performs depth map generation processing and the like may be provided outside (separate chip) of the distance measuring sensor 13 as a signal processing device.

<13. Configuration example of electronic device>
The distance measuring module 11 described above can be mounted on an electronic device such as a smartphone, a tablet terminal, a mobile phone, a personal computer, a game machine, a television receiver, a wearable terminal, a digital still camera, or a digital video camera.

FIG. 26 is a block diagram showing a configuration example of a smartphone as an electronic device equipped with a ranging module.

As shown in FIG. 26, in the smartphone 201, the distance measuring module 202, the image pickup device 203, the display 204, the speaker 205, the microphone 206, the communication module 207, the sensor unit 208, the touch panel 209, and the control unit 210 are connected via the bus 211. Is connected and configured. Further, the control unit 210 has functions as an application processing unit 221 and an operating system processing unit 222 by executing a program by the CPU.

The distance measuring module 11 of FIG. 1 is applied to the distance measuring module 202. For example, the distance measuring module 202 is arranged in front of the smartphone 201, and by performing distance measurement for the user of the smartphone 201, the depth value of the surface shape of the user's face, hand, finger, etc. is measured as a distance measurement result. Can be output as. It is also possible to recognize the user's gesture by using the distance measurement result by the distance measurement module 202.

The image pickup device 203 is arranged in front of the smartphone 201, and by taking an image of the user of the smartphone 201 as a subject, the image taken by the user is acquired. Although not shown, the image pickup device 203 may be arranged on the back surface of the smartphone 201.

The display 204 displays an operation screen for performing processing by the application processing unit 221 and the operation system processing unit 222, an image captured by the image pickup device 203, and the like. The speaker 205 and the microphone 206, for example, output the voice of the other party and collect the voice of the user when making a call by the smartphone 201.

The communication module 207 communicates via the communication network. The sensor unit 208 senses speed, acceleration, proximity, etc., and the touch panel 209 acquires a touch operation by the user on the operation screen displayed on the display 204.

The application processing unit 221 performs processing for providing various services by the smartphone 201. For example, the application processing unit 221 can create a face by computer graphics that virtually reproduces the user's facial expression based on the depth supplied from the distance measuring module 202, and can perform a process of displaying the face on the display 204. Further, the application processing unit 221 can perform a process of creating, for example, three-dimensional shape data of an arbitrary three-dimensional object based on the depth supplied from the distance measuring module 202.

The operation system processing unit 222 performs processing for realizing the basic functions and operations of the smartphone 201. For example, the operation system processing unit 222 can perform a process of authenticating the user's face and unlocking the smartphone 201 based on the depth value supplied from the distance measuring module 202. Further, the operation system processing unit 222 performs, for example, a process of recognizing a user's gesture based on the depth value supplied from the distance measuring module 202, and performs a process of inputting various operations according to the gesture. Can be done.

In the smartphone 201 configured in this way, by applying the distance measuring module 11 described above, for example, a depth map can be generated with high accuracy and high speed. As a result, the smartphone 201 can detect the distance measurement information more accurately.

<14. Application example to mobile>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on a moving body of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 27 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technology according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 27, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating a braking force of a vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives inputs of these radio waves or signals and controls a vehicle door lock device, a power window device, a lamp, and the like.

The vehicle outside information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image pickup unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or a character on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects the in-vehicle information. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (Advanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040. It is possible to perform coordinated control for the purpose of automatic driving, etc., which runs autonomously without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio image output unit 12052 transmits the output signal of at least one of the audio and the image to the output device capable of visually or audibly notifying the passenger of the vehicle or the outside of the vehicle. In the example of FIG. 27, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 28 is a diagram showing an example of the installation position of the imaging unit 12031.

In FIG. 28, the vehicle 12100 has

image pickup units

12101, 12102, 12103, 12104, 12105 as the image pickup unit 12031.

The

imaging units

12101, 12102, 12103, 12104, 12105 are provided at positions such as the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100, for example. The image pickup unit 12101 provided on the front nose and the image pickup section 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The images in front acquired by the

imaging units

12101 and 12105 are mainly used for detecting a preceding vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 28 shows an example of the photographing range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the

imaging units

12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform coordinated control for the purpose of automatic driving or the like in which the vehicle travels autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that can be seen by the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

The above is an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040 among the configurations described above. Specifically, by using the distance measurement by the distance measuring module 11 as the outside information detection unit 12030 and the inside information detection unit 12040, processing for recognizing the driver's gesture is performed, and various types (for example, for example) according to the gesture are performed. It can perform operations on audio systems, navigation systems, air conditioning systems) and detect the driver's condition more accurately. Further, the distance measurement by the distance measurement module 11 can be used to recognize the unevenness of the road surface and reflect it in the control of the suspension.

Note that this technology can be applied to the continuous-Wave method, which is an indirect ToF method that amplitude-modulates the light projected onto an object. The structure of the photodiode 51 of the light receiving unit 15 includes a distance measuring sensor having a CAPD (Current Assisted Photonic Demodulator) structure, a gate type distance measuring sensor that alternately applies an electric charge of the photodiode to two gates, and the like. It can be applied to a distance measuring sensor having a structure that distributes charges to two charge storage units.

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

The present techniques described above in this specification can be independently implemented independently as long as there is no contradiction. Of course, any plurality of the present technologies can be used in combination. For example, some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. It is also possible to carry out a part or all of any of the above-mentioned techniques in combination with other techniques not described above.

Further, for example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the contrary, the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Further, of course, a configuration other than the above may be added to the configuration of each device (or each processing unit). Further, if the configuration and operation of the entire system are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..

Further, in the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

It should be noted that the effects described in the present specification are merely examples and are not limited, and effects other than those described in the present specification may be obtained.

The present technology can have the following configurations.
(1)
A signal processing unit that performs processing to calculate the distance to the object based on the pixel data obtained by the pixels that receive the reflected light that is reflected by the object and the irradiation light emitted from the predetermined light emitting source is received. Prepare,
The signal processing unit determines the characteristics of the first charge detection unit and the second charge detection unit of the pixel based on the luminance model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel. A signal processing device having a characteristic calculation unit for calculating correction parameters to be corrected.
(2)
The signal processing device according to (1), wherein the characteristic calculation unit calculates the correction parameter based on a predetermined luminance model selected from a plurality of luminance models.
(3)
The signal processing device according to (2) above, wherein the predetermined luminance model is a triangular wave.
(4)
The signal processing device according to (2) above, wherein the predetermined luminance model is a sine wave.
(5)
The signal processing device according to (2) above, wherein the predetermined luminance model is a harmonic.
(6)
The characteristic calculation unit estimates the brightness function representing the brightness waveform assumed to be a harmonic for each of the first charge detection unit and the second charge detection unit by machine learning, and calculates the correction parameter (5). The signal processing apparatus according to.
(7)
The machine learning learner learns the luminance function corresponding to the first charge detection unit or the second charge detection unit, and learns the brightness function.
The signal processing device according to (6), wherein the characteristic calculation unit learns the learner so that the difference between the input of the learner and the value of the luminance function obtained by learning becomes small.
(8)
The characteristic calculation unit learns based on the condition that the first luminance function corresponding to the first charge detection unit and the second luminance function corresponding to the second charge detection unit are the same. The signal processing device according to (7) above.
(9)
The characteristic calculation unit calculates the correction parameters for a plurality of luminance models, and calculates the correction parameters.
One of the above (1) to (8) further comprising a selection unit for selecting the correction parameter having a small error from the phase shift amount calculated from the input image among the correction parameters of the plurality of luminance models. The signal processing device described.
(10)
The signal processing apparatus according to any one of (1) to (9), further comprising an evaluation unit for evaluating the correction parameter calculated by the characteristic calculation unit.
(11)
The signal processing unit further includes a distance calculation unit that calculates a distance to the object based on pixel data corrected by the correction parameter calculated by the characteristic calculation unit (1) to (10). The signal processing device according to any one.
(12)
A signal processing device that calculates the distance to the object based on the pixel data obtained by the pixels that receive the reflected light that is reflected by the object and the irradiation light emitted from the predetermined light source. ,
Based on the brightness model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel, a correction parameter for correcting the characteristics of the first charge detection unit and the second charge detection unit of the pixel is calculated. Signal processing method.
(13)
With a given light source
A distance measuring sensor having a pixel that receives the reflected light that is reflected by an object and returned from the irradiation light emitted from the light emitting source is provided.
The ranging sensor determines the characteristics of the first charge detection unit and the second charge detection unit of the pixel based on the luminance model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel. A distance measuring module having a characteristic calculation unit that calculates correction parameters to be corrected.

11 Distance measurement module, 12 light emitting unit, 13 distance measurement sensor, 14 control unit, 15 light receiving unit, 16 signal processing unit, 31 pixels, 32 pixel array, 52A 1st tap, 52B 2nd tap, 71 model determination unit, 72 Characteristic calculation unit, 73 signal correction unit, 74 distance calculation unit, 81A, 81B learner, 91 optimum model selection unit, 101 evaluation unit, 201 smartphone, 202 distance measurement module

Claims

A signal processing unit that performs processing to calculate the distance to the object based on the pixel data obtained by the pixels that receive the reflected light that is reflected by the object and the irradiation light emitted from the predetermined light emitting source is received. Prepare,
The signal processing unit determines the characteristics of the first charge detection unit and the second charge detection unit of the pixel based on the luminance model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel. A signal processing device having a characteristic calculation unit for calculating correction parameters to be corrected.
The signal processing device according to claim 1, wherein the characteristic calculation unit calculates the correction parameter based on a predetermined luminance model selected from a plurality of luminance models.
The signal processing device according to claim 2, wherein the predetermined luminance model is a triangular wave.
The signal processing device according to claim 2, wherein the predetermined luminance model is a sine wave.
The signal processing device according to claim 2, wherein the predetermined luminance model is a harmonic.
The characteristic calculation unit estimates the brightness function representing the brightness waveform assumed to be a harmonic for each of the first charge detection unit and the second charge detection unit by machine learning, and calculates the correction parameter according to claim 5. The signal processing device described.
The machine learning learner learns the luminance function corresponding to the first charge detection unit or the second charge detection unit, and learns the brightness function.
The signal processing device according to claim 6, wherein the characteristic calculation unit learns the learner so that the difference between the input of the learner and the value of the luminance function obtained by learning becomes small.
The characteristic calculation unit learns based on the condition that the first luminance function corresponding to the first charge detection unit and the second luminance function corresponding to the second charge detection unit are the same. The signal processing device according to claim 7, wherein the device is learned.
The characteristic calculation unit calculates the correction parameters for a plurality of luminance models, and calculates the correction parameters.
The signal processing apparatus according to claim 1, further comprising a selection unit for selecting the correction parameter having a small error from the phase shift amount calculated from the input image among the correction parameters of the plurality of luminance models.
The signal processing apparatus according to claim 1, further comprising an evaluation unit for evaluating the correction parameter calculated by the characteristic calculation unit.
The signal processing device according to claim 1, wherein the signal processing unit further includes a distance calculation unit that calculates a distance to the object based on pixel data corrected by the correction parameter calculated by the characteristic calculation unit. ..
A signal processing device that calculates the distance to the object based on the pixel data obtained by the pixels that receive the reflected light that is reflected by the object and the irradiation light emitted from the predetermined light source. ,
Based on the brightness model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel, a correction parameter for correcting the characteristics of the first charge detection unit and the second charge detection unit of the pixel is calculated. Signal processing method.
With a given light source
A distance measuring sensor having a pixel that receives the reflected light that is reflected by an object and returned from the irradiation light emitted from the light emitting source is provided.
The ranging sensor determines the characteristics of the first charge detection unit and the second charge detection unit of the pixel based on the luminance model according to the shape of the emission waveform of the irradiation light and the shape of the exposure waveform of the pixel. A distance measuring module having a characteristic calculation unit that calculates correction parameters to be corrected.