WO2021085128A1

WO2021085128A1 - Distance measurement device, measurement method, and distance measurement system

Info

Publication number: WO2021085128A1
Application number: PCT/JP2020/038712
Authority: WO
Inventors: 光永　知生; セドリックカロン; イェルーンヘルマンス
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2019-10-28
Filing date: 2020-10-14
Publication date: 2021-05-06

Abstract

This technology pertains to a distance measurement device, a measurement method, and a distance measurement system that make it possible to implement a broad measurement range using spot light. This distance measurement device comprises: a distance measurement sensor that receives reflection light when pattern light has been reflected back by an object, the pattern light having two types of brightness, i.e., bright sections and dark sections, and being emitted from a light source device; and a signal processing unit that detects, as a phase difference, a period of time from when the pattern light was emitted to when this light is received as reflection light, calculates a first distance measurement value that is the distance to the object on the basis of the phase difference, detects the position of the bright sections in the pattern light, calculates a second distance measurement value that is the distance to the object according to the principle of triangulation using the detected positions of the bright sections, and calculates a combined distance measurement value in which the first distance measurement value and the second distance measurement value are combined. The present technology can be applied to, for example, a distance measurement system or the like that measures the distance to a subject.

Description

Distance measuring device, measuring method, and distance measuring system

The present technology relates to a distance measuring device, a measuring method, and a distance measuring system, and more particularly to a distance measuring device, a measuring method, and a distance measuring system capable of realizing a wide measurement range by using spot light.

In recent years, advances in semiconductor technology have led to the miniaturization of distance measuring modules that measure the distance to an object. As a result, for example, it has been realized that a distance measuring module is mounted on a mobile terminal such as a so-called smartphone, which is a small information processing device having a communication function.

Examples of the distance measuring method in the distance measuring module include the Indirect ToF (Time of Flight) method and the Structured Light method. The IndirectToF method is a method that irradiates light toward an object, detects the light reflected on the surface of the object, and calculates the distance to the object based on the measured value obtained by measuring the flight time of the light. (See, for example, Patent Document 1). The Structured Light method is a method of irradiating a pattern light toward an object and calculating the distance to the object based on an image obtained by capturing the distortion of the pattern on the surface of the object.

In the Indirect ToF method, for example, planar light and spot light can be used as the irradiation light to irradiate the object. Plane light can irradiate a wide range of light, but since the emission intensity is dispersed, it is not possible to measure an object at a long distance (the measurement range is short). On the other hand, spot light can be measured farther because the density of light power can be increased, and the influence of multipath can be suppressed, so that improvement in accuracy can be expected.

JP-A-2017-150893

However, if the irradiation light is spot light, the amount of light received will be large for objects with a short distance, so the pixels will be saturated and measurement may not be possible.

This technology was made in view of such a situation, and makes it possible to realize a wide measurement range by using spot light.

The distance measuring device on the first side of the present technology is
A ranging sensor that receives the reflected light that is reflected by an object and has two types of brightness, a bright part and a dark part that are emitted from the light source device.
The time from the irradiation of the pattern light to the reception of the reflected light is detected as a phase difference, and the first distance measurement value, which is the distance to the object, is calculated based on the phase difference, and the above-mentioned The position of the bright part of the pattern light is detected, the second distance measurement value which is the distance to the object is calculated by the principle of triangulation using the detected position of the bright part, and the first distance measurement value. It is provided with a signal processing unit that calculates a fusion distance measurement value that is a fusion of the second distance measurement value and the second distance measurement value.

The measurement method of the second aspect of the present technology is
The distance measuring device
The time from when the pattern light having two kinds of brightness of the bright part and the dark part irradiated from the light source device is irradiated to when it is reflected by the object and received as the reflected light is detected as the phase difference, and is based on the phase difference. The first ranging value, which is the distance to the object, is calculated, the position of the bright part of the pattern light is detected, and the detected position of the bright part is used to reach the object by the principle of triangular survey. A second distance measurement value, which is a distance, is calculated, and a fusion distance measurement value obtained by fusing the first distance measurement value and the second distance measurement value is calculated.

The ranging system of the third aspect of this technology is
A light source device that irradiates a pattern light having two types of brightness, a bright part and a dark part,
It is equipped with a ranging device that receives the reflected light that is reflected by the object and returned.
The distance measuring device is
A distance measuring sensor that receives the reflected light and
The time from the irradiation of the pattern light to the reception of the reflected light is detected as a phase difference, and the first distance measurement value, which is the distance to the object, is calculated based on the phase difference, and the above-mentioned The position of the bright part of the pattern light is detected, the second distance measurement value which is the distance to the object is calculated by the principle of triangulation using the detected position of the bright part, and the first distance measurement value. It is provided with a signal processing unit that calculates a fusion distance measurement value that is a fusion of the second distance measurement value and the second distance measurement value.

In the first to third aspects of the present technology, the pattern light having two types of brightness, the bright part and the dark part, emitted from the light source device is received by the object and the reflected light returned is received, and the pattern light is received. The time from when the light is irradiated to when it is received as the reflected light is detected as a phase difference, and the first ranging value, which is the distance to the object, is calculated based on the phase difference, and the pattern light is used. The position of the bright part is detected, and the second distance measurement value, which is the distance to the object, is calculated by the principle of triangulation using the detected position of the bright part, and is combined with the first distance measurement value. A fusion distance measurement value that is fused with the second distance measurement value is calculated.

The distance measuring device and the distance measuring system may be independent devices or may be modules incorporated in other devices.

It is a figure which shows the schematic configuration example of the distance measurement system to which this technology is applied. It is a block diagram which shows the structural example of a light source device and a distance measuring device. It is a figure explaining the method of distance measurement by the ToF method. It is a figure explaining the method of distance measurement by the SL method. It is a figure explaining another example of a pattern light. It is a block diagram which shows the structural example of the distance measuring sensor. It is a block diagram which shows the structural example of a pixel. It is a perspective view which shows the chip structure example of the distance measuring device. It is a flowchart explaining the distance measurement process. It is a flowchart of the 1st fusion distance measurement value calculation process. It is the flowchart of the 2nd fusion distance measurement value calculation processing. It is a figure explaining an error evaluation index. It is a flowchart of the 3rd fusion distance measurement value calculation process. It is a figure explaining the result obtained by the fusion distance measurement value calculation process. It is a figure explaining the high-resolution processing performed by a signal processing unit. It is a flowchart explaining the distance measurement process including the high-resolution process. It is a figure which shows the other configuration example of the distance measurement system to which this technique is applied. It is a figure which shows the other configuration example of the distance measurement system to which this technique is applied. It is a block diagram which shows the structural example of the electronic device to which this technology is applied. It is a block diagram which shows an example of the schematic structure of a vehicle control system. It is explanatory drawing which shows an example of the installation position of the vehicle exterior information detection unit and the imaging unit.

Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted. The explanation will be given in the following order.
1. 1. Schematic configuration example of the ranging system 2. Explanation of distance measurement method by ToF method and SL method 3. Configuration example of distance measuring sensor 4. Example of chip configuration of distance measuring device 5. Distance measurement process 6. High resolution processing 7. Distance measurement process 8. Modification example of the ranging system 9. Configuration example of electronic device 10. Application example to mobile

<1. Schematic configuration example of distance measurement system>
FIG. 1 shows a schematic configuration example of a distance measuring system to which the present technology is applied.

The distance measuring system 1 shown in FIG. 1 includes a light source device 11, a light emitting side optical system 12, a distance measuring device 21, and a light receiving side optical system 22.

The light source device 11 generates and irradiates the pattern light 15 having two types of brightness, a bright part and a dark part. The pattern light 15 has, for example, a plurality of spots SP having a dot (circle) shape arranged at regular or irregular predetermined intervals as shown in FIG. 1 as a bright portion, and the other region as a dark portion. It is said to be a pattern light. As will be described later, the pattern light 15 emitted by the light source device 11 is not limited to a pattern having a dot shape in the bright portion, and may be a grid pattern or the like. The pattern light 15 emitted from the light source device 11 irradiates a predetermined object OBJ as an object to be measured via the light emitting side optical system 12. Then, the pattern light 15 is reflected by the predetermined object OBJ and is incident on the distance measuring device 21 via the light receiving side optical system 22.

The ranging device 21 receives the pattern light 15 that is reflected by the object OBJ and is incident. The distance measuring device 21 generates a detection signal according to the amount of light of the received pattern light 15. Then, the distance measuring device 21 calculates and outputs a distance measuring value which is a measured value of the distance to a predetermined object OBJ based on the detection signal.

FIG. 2 is a block diagram showing a configuration example of the light source device 11 and the distance measuring device 21.

The light source device 11 has a light emitting source 31 and a light source driving unit 32. The distance measuring device 21 includes a synchronization control unit 41, a distance measuring sensor 42, and a signal processing unit 43.

The light emitting source 31 is composed of a light source array in which a plurality of light emitting elements such as a VCSEL (Vertical Cavity Surface Emitting Laser) are arranged in a plane direction. The light emitting source 31 emits light while being modulated at a timing corresponding to the light emission timing signal supplied from the synchronous control unit 41 of the distance measuring device 21 according to the control of the light source driving unit 32, and emits the pattern light 15 as the irradiation light. Irradiate a predetermined object OBJ. As the irradiation light, for example, infrared light having a wavelength in the range of about 850 nm to 940 nm is used.

The light source driving unit 32 is composed of, for example, a laser driver or the like, and causes each light emitting element of the light emitting source 31 to emit light in response to a light emitting timing signal supplied from the synchronous control unit 41.

The synchronization control unit 41 of the distance measuring device 21 generates a light emission timing signal that controls the timing at which each light emitting element of the light emitting source 31 emits light, and supplies it to the light source driving unit 32. Further, the synchronization control unit 41 also supplies a light emission timing signal to the distance measurement sensor 42 in order to drive the distance measurement sensor 42 in accordance with the light emission timing of the light emission source 31. As the light emission timing signal, for example, a rectangular wave signal (pulse signal) that turns on and off at a predetermined frequency (for example, 20 MHz, 50 MHz, etc.) can be used. The emission timing signal is not limited to a rectangular wave as long as it is a periodic signal, and may be, for example, a sine wave.

In the distance measuring sensor 42, the pattern light 15 emitted from the light source device 11 is reflected by the predetermined object OBJ by the pixel array unit 63 (FIG. 6) in which a plurality of pixels 71 (FIG. 6) are two-dimensionally arranged in a matrix. Receives the reflected light. Then, the distance measuring sensor 42 supplies the detection signal according to the received amount of the received reflected light to the signal processing unit 43 in pixel units of the pixel array unit 63.

The signal processing unit 43 calculates the distance measurement value, which is the distance from the distance measurement sensor 42 to the predetermined object OBJ, based on the detection signal supplied from the distance measurement sensor 42. More specifically, the signal processing unit 43 calculates the distance measurement value (first distance measurement value) in the ToF method, which is the first detection method, and also distance measurement in the SL method, which is the second detection method. Calculate the value (second distance measurement value). Then, the signal processing unit 43 calculates the fusion distance measurement value by fusing the first distance measurement value and the second distance measurement value, and outputs the fusion distance measurement value to the outside. The ToF method is a method in which the time from the irradiation of the spot SP, which is the bright part of the pattern light 15, to the reception as the reflected light is detected as the phase difference, and the distance is calculated based on the phase difference. The method is a method in which the position of the spot SP, which is the bright part of the pattern light 15, is detected, and the distance is calculated by the principle of triangulation using the detected position of the spot light.

The ranging system 1 configured as described above irradiates the object OBJ with the pattern light 15 having two types of brightness, light and dark, as shown in the pattern light 15 of FIG. 1, and is the first detection method, ToF. The first distance measurement value by the method and the second distance measurement value by the SL method, which is the second detection method, are calculated, and the fusion distance measurement is performed from the first distance measurement value and the second distance measurement value. Calculate the value and output it to the outside.

In the distance measuring system 1, the synchronization control unit 13 that generates a light emission timing signal may be provided not on the distance measuring device 21 side but on the light source device 11 side.

<2. Explanation of distance measurement method by ToF method and SL method>
Next, with reference to FIGS. 3 and 4, the distance measuring methods of the ToF method and the SL method will be briefly described.

First, the method of distance measurement by the ToF method will be described with reference to FIG.

The distance measurement value D [mm] corresponding to the distance from the distance measurement device 21 to the object OBJ can be calculated by the following equation (1).

Δt in the equation (1) is the time until the pattern light 15 emitted from the light emitting source 31 is reflected by the object OBJ and enters the distance measuring sensor 42, and c represents the speed of light.

As the pattern light 15 emitted from the light emitting source 31, pulsed light that repeatedly turns on and off at a predetermined frequency f (modulation frequency) as shown in FIG. 3 is adopted. One cycle T of pulsed light is 1 / f. In the distance measuring sensor 42, the reflected light (light receiving pattern) is detected out of phase according to the time Δt from the light emitting source 31 to the distance measuring sensor 42. Assuming that the amount of phase shift (phase difference) between the light emitting pattern and the light receiving pattern is φ, the time Δt can be calculated by the following equation (2).

Therefore, the distance measurement value D from the distance measurement sensor 42 to the object OBJ can be calculated from the equations (1) and (2) by the following equation (3).

Next, the above-mentioned calculation method of the phase difference φ will be described.

Each pixel of the pixel array formed on the distance measuring sensor 42 repeats ON / OFF at high speed, and accumulates electric charge only during the ON period.

The distance measuring sensor 42 sequentially switches the ON / OFF execution timing of each pixel of the pixel array, accumulates the electric charge at each execution timing, and outputs a detection signal according to the accumulated electric charge.

There are four types of ON / OFF execution timings, for example, phase 0 degrees, phase 90 degrees, phase 180 degrees, and phase 270 degrees.

The execution timing of the phase 0 degree is a timing at which the ON timing (light receiving timing) of each pixel of the pixel array is set to the phase of the pulsed light emitted by the light emitting source 31 of the light source device 11, that is, the same phase as the light emitting pattern.

The execution timing of the phase 90 degrees is a timing in which the ON timing (light receiving timing) of each pixel of the pixel array is set to a phase 90 degrees behind the pulsed light (light emitting pattern) emitted by the light emitting source 31 of the light source device 11.

The execution timing of the phase 180 degrees is a timing in which the ON timing (light receiving timing) of each pixel of the pixel array is set to a phase 180 degrees behind the pulsed light (light emitting pattern) emitted by the light emitting source 31 of the light source device 11.

The execution timing of the phase 270 degrees is a timing in which the ON timing (light receiving timing) of each pixel of the pixel array is delayed by 270 degrees from the pulsed light (light emitting pattern) emitted by the light emitting source 31 of the light source device 11.

The ranging sensor 42 sequentially switches the light receiving timing in the order of phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree, and acquires the received light amount (accumulated charge) of the reflected light at each light receiving timing. In FIG. 3, in the light receiving timing (ON timing) of each phase, the timing at which the reflected light is incident is shaded.

As shown in FIG. 3, when the light receiving timing is set to phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree, the accumulated charges are Q ₀ , Q ₉₀ , Q ₁₈₀ , respectively. And, assuming that Q ₂₇₀ , the phase difference φ can be calculated by the following equation (4) using _{Q 0} , Q ₉₀ , Q ₁₈₀ , and Q _270.

By inputting the phase difference φ calculated by the equation (4) into the above equation (3), the distance measurement value D from the distance measurement system 1 to the object OBJ can be calculated.

The distance measuring sensor 42 switches the light receiving timing in each pixel of the pixel array in order of phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree as described above, and the accumulated charge (charge) in each phase. The _{detection signals corresponding to Q 0} , charge Q ₉₀ , charge Q ₁₈₀ , and charge Q ₂₇₀ ) are sequentially supplied to the signal processing unit 43. As will be described later, by providing two charge storage units in each pixel of the pixel array and alternately accumulating charges in the two charge storage units, the phases are phased, for example, 0 degrees and 180 degrees. It is possible to acquire the detection signals of the two light receiving timings in which are inverted in one frame. In order to acquire a four-phase detection signal having a phase of 0 degrees, a phase of 90 degrees, a phase of 180 degrees, and a phase of 270 degrees, it is sufficient to have a detection signal of two frames.

Next, the method of distance measurement by the SL method will be described with reference to FIG.

The SL method uses a light source device that projects pattern light and a light receiving sensor (image sensor) that receives the pattern light to search for a pair of a certain position in the projection pattern and the corresponding light receiving sensor position. This is a method of measuring the distance by applying triangulation.

As shown in FIG. 1, the light source device 11 irradiates the object OBJ with a plurality of spots SP arranged at predetermined intervals, but pays attention to one spot SP (hereinafter, referred to as a spot of interest SP) and pays attention to it. It is assumed that the spot SP is detected at a predetermined position P2 in the light receiving region of the distance measuring sensor 42.

At this time, the position P1 in the projection pattern of the light source device 11 that emitted the spot of interest SP is known in the light source device 11. Further, the positional relationship between the light source device 11 and the distance measurement sensor 42 including the baseline distance BL between the light source principal point (projection center) of the light source device 11 and the sensor principal point (light receiving center) of the distance measurement sensor 42 is also known. Is. Therefore, using the position P1, the position P2, and the baseline distance BL, the distance measurement value D corresponding to the distance from the distance measurement device 21 to the object OBJ can be calculated by the principle of triangulation.

Therefore, when the ranging sensor 42 receives a plurality of spot SPs of the pattern light 15, if it is known which spot SP of the plurality of spots SP emitted by the light source device 11 corresponds to each of the received spot SPs. For each of the plurality of spots SP, the distance measurement value D from the distance measurement device 21 to the object OBJ can be calculated.

Therefore, the pattern light 15 emitted by the light source device 11 of the distance measuring system 1 emits the spot SP of the light emitting source 31 when the distance measuring sensor 42 receives a predetermined spot SP at a predetermined position in the light receiving region. It is a pattern in which a plurality of spot SPs are arranged so that (position) can be specified.

Specifically, the position of the spot SP received as reflected light by the distance measuring sensor 42 moves in a predetermined trajectory within the light receiving region according to the distance to the object OBJ, but the trajectory of each spot SP is different. If it does not overlap with the locus of the spot SP, the spot SP (position) of the light emitting source 31 that emitted the spot SP can be identified based on the position of the spot SP received by the distance measuring sensor 42. In other words, the pattern light 15 is a dot pattern in which a plurality of spot SPs are arranged at sufficiently sparse intervals so that the positions of the spot SPs detected by the distance measuring sensor 42 do not overlap with the other spot SPs.

As for the pattern light 15, it is sufficient that the bright portion detected in the light receiving region and the bright portion of the light emitting source 31 that emits the bright portion can be identified for each of the plurality of bright portions, so that the bright portion (dot) is shown in FIG. It does not have to be regularly arranged.

For example, the pattern light 15 may be a pattern in which dots (spot SPs) as bright areas are irregularly arranged as shown in A of FIG. In this case, for example, the correspondence between the light receiving position and the light emitting position can be detected by utilizing the feature of the irregular arrangement of dots. It is also possible to calculate the distance measurement value by the Structured Light method by utilizing the characteristics of the irregular arrangement of dots.

Alternatively, the pattern light 15 does not have to have a dot in the bright part. For example, as shown in B of FIG. 5, a grid pattern in which a grid pattern in which an arbitrary place of a 3x3 square grid is set as a bright part may be arranged may be used. In the case of such a grid pattern, it is possible to specify which grid pattern has received light by arranging the bright part of the 3x3 square grid, so that the position of the grid pattern in the light receiving region of the ranging sensor 42 and its position. It is possible to detect the correspondence with the position of the light emitting source 31 that emits the grid pattern.

When two charge storage units are provided in each pixel of the distance measuring sensor 42, the ToF method requires a detection signal of two frames as described above, but the SL method uses a detection signal of one frame to determine the distance. Since it can be calculated, the distance measurement value D can be calculated using the detection signal of either one of the two frames used in the ToF method.

<3. Configuration example of ranging sensor>
FIG. 6 is a block diagram showing a configuration example of the distance measuring sensor 42.

The distance measuring sensor 42 includes a timing control unit 61, a row scanning circuit 62, a pixel array unit 63, a plurality of AD (Analog to Digital) conversion units 64, a column scanning circuit 65, and a signal processing unit 66. In the pixel array unit 63, a plurality of pixels 71 are two-dimensionally arranged in a matrix in the row direction and the column direction. Here, the row direction means the arrangement direction of the pixels 71 in the horizontal direction, and the column direction means the arrangement direction of the pixels 71 in the vertical direction. The row direction is the horizontal direction in the figure, and the column direction is the vertical direction in the figure.

The timing control unit 61 is composed of, for example, a timing generator that generates various timing signals, and generates various timing signals in synchronization with the light emission timing signal supplied from the synchronization control unit 41 (FIG. 2). It is supplied to the row scanning circuit 62, the AD conversion unit 64, and the column scanning circuit 65. That is, the timing control unit 61 controls the drive timing of the row scanning circuit 62, the AD conversion unit 64, and the column scanning circuit 65.

The row scanning circuit 62 is composed of, for example, a shift register, an address decoder, or the like, and drives each pixel 71 of the pixel array unit 63 at the same time for all pixels or in units of rows. The pixel 71 receives the reflected light under the control of the row scanning circuit 62, and outputs a detection signal (pixel signal) at a level corresponding to the amount of light received. Details of the pixel 71 will be described later in FIG.

With respect to the matrix-like pixel array of the pixel array unit 63, the pixel drive line 72 is wired along the horizontal direction for each pixel row, and the vertical signal line 73 is wired along the vertical direction for each pixel row. The pixel drive line 72 transmits a drive signal for driving when reading a detection signal from the pixel 71. In FIG. 6, the pixel drive line 72 is shown as one wiring, but it is actually composed of a plurality of wirings. Similarly, although the vertical signal line 73 is also shown as one wire, it is actually composed of a plurality of wires.

The AD conversion unit 64 is provided for each column, and synchronizes with the clock signal CK supplied from the timing control unit 61, and transmits the detection signal supplied from each pixel 71 of the corresponding column via the vertical signal line 73. AD conversion. The AD conversion unit 64 outputs the AD-converted detection signal (detection data) to the signal processing unit 66 under the control of the column scanning circuit 65. The column scanning circuit 65 sequentially selects the AD conversion unit 64 and outputs the detection data after the AD conversion to the signal processing unit 66.

The signal processing unit 66 has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing based on the detection data output from the AD conversion unit 64.

<Pixel configuration example>
FIG. 7 is a block diagram showing a configuration example of the pixel 71.

The pixel 71 includes a photoelectric conversion element 81, a transfer switch 82,

charge storage units

83 and 84, and selection switches 85 and 86.

The photoelectric conversion element 81 is composed of, for example, a photodiode, and photoelectrically converts the reflected light to generate an electric charge.

The transfer switch 82 transfers the charge generated by the photoelectric conversion element 81 to either the

charge storage unit

83 or 84 based on the transfer signal SEL_FD. The transfer switch 82 is composed of, for example, a pair of MOS (Metal-Oxide-Semiconductor) transistors.

The

charge storage units

83 and 84 are composed of, for example, a floating diffusion layer, accumulate charges, and generate a voltage corresponding to the accumulated charges. The charges accumulated in the

charge storage units

83 and 84 can be reset based on the reset signal RST.

The selection switch 85 selects the output of the charge storage unit 83 according to the selection signal RD_FD1. The selection switch 86 selects the output of the charge storage unit 84 according to the selection signal RD_FD2. That is, when the

selection switch

85 or 86 is turned on by the selection signal RD_FD1 or RD_FD2, a voltage signal corresponding to the stored charge of the turned-on

charge storage unit

83 or 84 is transmitted as a detection signal via the vertical signal line 73. Is output to the AD conversion unit 64. Each of the selection switches 85 and 86 is composed of, for example, a MOS transistor.

The wiring for transmitting the transfer signal SEL_FD, the reset signal RST, and the selection signals RD_FD1 and RD_FD2 corresponds to the pixel drive line 72 in FIG.

Assuming that the

charge storage units

83 and 84 are referred to as a first tap and a second tap, respectively, in the ToF method, the pixel 71 transfers the charges generated by the photoelectric conversion element 81 to the first tap and the second tap, respectively. By alternately accumulating charges on the two taps, it is possible to acquire detection signals of two light receiving timings whose phases are inverted, such as phase 0 degree and phase 180 degree, in one frame. In the next frame, two light reception timing detection signals having a phase of 90 degrees and a phase of 270 degrees can be acquired.

In the SL method, when the position of the spot SP of the pattern light 15 is detected, the detection signals of the first tap and the second tap are added up to obtain a pixel signal of one pixel. Similar to the image sensor, the position of each spot SP in the light receiving region can be specified based on the pixel signal (luminance information) of each pixel. The total processing of the detection signals of the first tap and the second tap can be performed by the signal processing unit 43 in the subsequent stage.

Therefore, the distance measuring sensor 42 can operate with the same drive for both the ToF method and the SL method.

<4. Example of chip configuration of ranging device>
FIG. 8 is a perspective view showing a chip configuration example of the distance measuring device 21.

As shown in A of FIG. 8, the distance measuring device 21 can be composed of one chip in which the first die (board) 91 and the second die (board) 92 are laminated.

The first die 91 is configured with, for example, a synchronization control unit 41 and a distance measuring sensor 42, and the second die 92 is configured with, for example, a signal processing unit 43.

The distance measuring device 21 may be composed of three layers in which another logic die is laminated in addition to the first die 91 and the second die 92, or may be composed of four or more layers of dies (boards). It may be configured.

Further, in the distance measuring device 21, for example, as shown in B of FIG. 8, the first chip 95 as the distance measuring sensor 42 and the second chip 96 as the signal processing unit 43 are connected to the relay board 97. It can be formed and constructed on top. The synchronization control unit 41 is included in either the first chip 95 or the second chip 96.

<5. Distance measurement processing>
Next, with reference to the flowchart of FIG. 9, the distance measurement process using the two detection methods of the ToF method and the SL method performed by the distance measurement system 1 will be described. This process is started, for example, when a control unit of a higher-level device in which the distance measuring system 1 is incorporated supplies an instruction to start distance measurement.

First, in step S1, the synchronization control unit 41 generates a light emission timing signal and supplies it to the light source driving unit 32 and the distance measuring sensor 42.

In step S2, the light emitting source 31 emits the pattern light 15 while modulating according to the light emitting timing signal according to the control of the light source driving unit 32. As a result, the light emitting source 31 irradiates the predetermined object OBJ with the irradiation light in synchronization with the light emission timing signal. The pattern light 15 is, for example, infrared light having a dot pattern having a plurality of spots SP arranged at predetermined intervals as bright areas and other areas as dark areas, as shown in FIG.

In step S3, the distance measuring sensor 42 receives the reflected light reflected by the predetermined object OBJ from the pattern light 15 emitted from the light emitting source 31. Then, the distance measuring sensor 42 supplies the detection signals of the first tap and the second tap corresponding to the received amount of the received reflected light to the signal processing unit 43 for each pixel 71 of the pixel array unit 63. .. As described above, since the ToF method requires a detection signal of 2 frames, at least 2 frames of the detection signal are acquired and supplied to the signal processing unit 43.

In step S4, the signal processing unit 43 uses the detection signals of the first tap and the second tap of each pixel 71 of the pixel array unit 63 supplied from the distance measuring sensor 42 to obtain a plurality of pattern lights 15. For each spot SP, the distance measurement value D1 by the ToF method is calculated and stored in the internal memory as the first distance measurement value D1.

The calculation method of the first ranging value D1 by the ToF method of step S4 will be described.

Strictly speaking, the detection signal of each tap (first tap or second tap) corresponding to one spot SP, which is the bright part of the pattern light 15, is that the light of the spot SP is directly reflected by the predetermined object OBJ. In addition to the emitted light, the light indirectly reflected by a place other than the predetermined object OBJ (for example, a wall or another object) is also included. Assuming that the detection signal by the direct reflection component of the light of the spot SP is d _A and the detection signal by the indirect reflection component is I _A , the detection signal of each tap of the pixel 71 on which the light of one spot SP is incident is (d _A). It is represented by + I _A). In the dark pixel 71 other than the pixel 71 in which the light of the spot SP is incident, the detection signal I _A corresponding only to the indirect reflection component is detected.

The signal processing unit 43, the detection signal of the pixel 71 corresponding to the bright portion and (d _A + I _A), and a detection signal I _A pixel 71 corresponding to the dark part, the light receiving area of the distance measuring sensor 42 (pixel array portion 63 ), Estimate the position of each spot SP.

Then, the signal processing unit 43 calculates the difference between the _{detection signal (d A} + I _A ) of the pixel 71 corresponding to the bright part and _{the detection signal I A} of the pixel 71 corresponding to the dark part, and removes the indirect reflection component. detection signals of the pixels 71 corresponding to the bright portion d _a = calculates the _{_{(d a + I a) -I}} a.

_{The signal processing unit 43 uses the detection signals d A} of the first tap and the second tap from which the indirect reflection component has been removed for each of the spot SPs estimated in the light receiving region, and places them according to the equation (4). The phase difference φ is calculated, and the first ranging value D1 is calculated.

In step S5, the signal processing unit 43 uses the detection signals of the first tap and the second tap of each pixel 71 of the pixel array unit 63 supplied from the distance measuring sensor 42 to generate a plurality of pattern lights 15. For each spot SP, the distance measurement value D2 by the SL method is calculated and stored in the internal memory as the second distance measurement value D2.

The calculation method of the second distance measurement value D2 by the SL method in step S5 will be described.

First, the signal processing unit 43 adds up the detection signal of the first tap and the detection signal of the second tap for each pixel 71 of the pixel array unit 63 supplied from the distance measuring sensor 42 to generate a pixel signal. To do.

The signal processing unit 43, in the light receiving region (pixel array unit 63) specifies a pixel signal d _B of the pixel 71 corresponding to the bright portion, and a pixel signal I _B of the pixel 71 corresponding to the dark portion of each spot SP Identify the location. The pixel 71 corresponding to the bright portion can be detected by using the luminance information (pixel signal) of the pixel unit or a plurality of pixel units such as 3x3.

Next, the signal processing unit 43 acquires the position of the spot SP on the light source device 11 side corresponding to the position of each specified spot SP from the internal memory. The position of the spot SP on the light source device 11 side corresponding to the position of each spot SP is associated with, for example, the position of each spot SP and is stored in advance in the internal memory.

Then, the signal processing unit 43 determines from the position of the spot SP that has received light, the position on the light source device 11 side corresponding to the position, and the baseline distance BL between the sensor principal point (light receiving center) of the distance measuring sensor 42. The distance measurement value D2 is calculated for each of the plurality of spots SP of the pattern light 15 according to the principle of triangulation. The baseline distance BL is also stored in the internal memory in advance.

Next, in step S6, the signal processing unit 43 executes the fusion distance measurement value calculation process for each of the plurality of spots SP of the received pattern light 15. The fusion distance measurement value calculation process is a process of calculating the fusion distance measurement value DF by fusing the first distance measurement value D1 by the ToF method and the second distance measurement value D2 by the SL method. For example, FIG. Alternatively, any of the first to third fusion distance measurement value calculation processes shown in FIG. 12 is executed.

<First fusion distance measurement value calculation process>
FIG. 10 is a flowchart of the first fusion distance measurement value calculation process that can be executed as the fusion distance measurement value calculation process in step S9 of FIG.

In this process, first, in step S21, the signal processing unit 43 is based on the first distance measurement value D1 obtained by the ToF method and the second distance measurement value D2 obtained by the SL method. Calculate the value of the distance evaluation index (distance evaluation index value). The distance evaluation index value is a value for determining whether the distance to the object OBJ as the object to be measured is a short distance or a long distance. For example, the first distance measurement value D1 and the second distance measurement value D1 and the second distance evaluation index value are used. The average value with the distance measurement value D2 can be used. Alternatively, the distance measurement value of one of the predetermined detection methods, that is, the first distance measurement value D1 or the second distance measurement value D2 may be simply adopted as the distance evaluation index value.

In step S22, the signal processing unit 43 determines whether the distance evaluation index value indicates a short distance. For example, when the distance evaluation index value is equal to or less than a predetermined value, the signal processing unit 43 determines that the distance is short.

If it is determined in step S22 that the distance evaluation index value indicates a short distance, the process proceeds to step S23, and the signal processing unit 43 fuses the second distance measurement value D2 obtained by the SL method. Determine the distance measurement value DF.

On the other hand, if it is determined in step S22 that the distance evaluation index value does not indicate a short distance, that is, indicates a long distance, the process proceeds to step S24, and the signal processing unit 43 is obtained by the ToF method. The first distance measurement value D1 is determined as the fusion distance measurement value DF.

As described above, in the first fusion distance measurement value calculation process, the distance evaluation using the first distance measurement value D1 obtained by the ToF method and the second distance measurement value D2 obtained by the SL method is used. The fusion distance measurement value DF is calculated based on the value of the index.

<Second fusion distance measurement value calculation process>
FIG. 11 is a flowchart of a second fusion distance measurement value calculation process that can be executed as the fusion distance measurement value calculation process in step S9 of FIG.

In this process, first, in step S31, the signal processing unit 43 determines a detection method having a small error based on the error evaluation indexes of each of the ToF method and the SL method.

For example, the error evaluation indexes of the ToF method and the SL method as shown in FIG. 12 are calculated in advance and stored in the internal memory. The error evaluation indexes E1 and E2 are error evaluation functions using the distance (distance measurement value) as a parameter, the error evaluation index E1 represents the error evaluation index by the ToF method, and the error evaluation index E2 is the error by the SL method. Represents an evaluation index. In the error evaluation indexes E1 and E2 of the ToF method and the ToF method shown in FIG. 12, the error of the second distance measurement value D2 by the SL method is smaller and the distance measurement value is larger than the predetermined value at a short distance. Then, the error is smaller in the first distance measurement value D1 by the ToF method.

The signal processing unit 43 calculates an error based on the error evaluation indexes E1 and E2 of the ToF method and the SL method, respectively, and determines the detection method having the smaller error.

Then, in step S32, the signal processing unit 43 determines the distance measurement value (first distance measurement value D1 or second distance measurement value D2) of the detection method having a small error as the fusion distance measurement value DF.

As described above, in the second fusion distance measurement value calculation process, the fusion distance measurement value DF is calculated based on the error evaluation indexes of the ToF method and the SL method respectively.

<Third fusion distance measurement value calculation process>
FIG. 13 is a flowchart of a third fusion distance measurement value calculation process that can be executed as the fusion distance measurement value calculation process in step S9 of FIG.

In this process, first, in step S41, the signal processing unit 43 determines whether the ToF method detection signal (detection data after AD conversion) indicates saturation. The signal processing unit 43 determines that the detection signal of the ToF method indicates saturation when the detection signal (detection data after AD conversion) of the spot SP portion is equal to or higher than a predetermined threshold value corresponding to the saturation level. ..

If it is determined in step S41 that the detection signal of the ToF method indicates saturation, the process proceeds to step S42, and the signal processing unit 43 fuses and measures the second ranging value D2 obtained by the SL method. Determine the distance value DF.

On the other hand, if it is determined in step S41 that the detection signal of the ToF method does not show saturation, the process proceeds to step S43, and the signal processing unit 43 determines the first ranging value D1 obtained by the ToF method. Determine the fusion distance measurement value DF.

As described above, in the third fusion distance measurement value calculation process, the fusion distance measurement value DF is calculated based on whether or not the detection signal of the spot SP portion of the ToF method indicates saturation.

In step S9 of FIG. 9, the fusion distance measurement value DF is calculated by any of the first to third fusion distance measurement value calculation processes, and is output from the distance measurement device 21 to the subsequent stage, and the distance measurement process of FIG. 9 is performed. finish.

Since the fusion distance measurement value calculation process is executed for each of the plurality of spot SPs of the pattern light 15, as shown in FIG. 14, the fusion distance measurement value DF is the first distance measurement value obtained by the ToF method. Whether it corresponds to D1 or the second ranging value D2 obtained by the SL method differs for each spot SP. In FIG. 14, the spot SP marked “ToF” in the vicinity of each spot SP indicates that the first ranging value D1 obtained by the ToF method was selected as the fusion ranging value DF, and “ The spot SP marked "SL" indicates that the second ranging value D2 obtained by the SL method was selected as the fusion ranging value DF.

According to the distance measurement process by the distance measurement system 1 described above, each of the first tap and the second tap obtained by driving the distance measurement sensor 42 having two charge storage units (2 taps) in one pixel. Using the detection signal, it is possible to calculate two distance measurement values, a first distance measurement value D1 by the ToF method and a second distance measurement value D2 by the SL method. Then, the distance measuring system 1 can output a fusion distance measuring value DF in which these two distance measuring values are fused.

By fusing and outputting the distance measurement values obtained by the two detection methods, spot light having a high emission intensity is used as the irradiation light emitted by the light source device 11, and the pixels are saturated in an object having a short distance. Even in the case, the distance can be measured. That is, a wide measurement range can be realized by using spot light.

<6. High resolution processing>
In the distance measurement process described above, the distance measurement value can be calculated only in the position of the spot SP, which is the bright part of the pattern light 15, in the light receiving region of the distance measurement sensor 42. Although there are merits of improving the above and extending the measurement range, the resolution is lower than that when the irradiation light is flat light.

Therefore, the distance measuring system 1 uses the fusion distance measurement value DF of the plurality of spot SPs to execute a high-resolution processing for calculating the fusion distance measurement value DF of the position of the dark part between the plurality of spot SPs, and measures the resolution. The resolution of the distance value can be improved.

The high resolution processing executed by the signal processing unit 43 will be described with reference to FIG.

The signal processing unit 43 calculates the fusion distance measurement value DF of the dark part between the plurality of bright parts by using the bilateral upsampling method as the high resolution processing.

As shown in FIG. 15, among the plurality of spots SP constituting the pattern light 15 received by the distance measuring sensor 42, the positions P of the dark part between the _{three spots SP A} , SP _B , and SP _C. examples will be described for calculating the fusion distance value DF _P.

The signal processing unit 43 detects (positions) of a plurality of spots SP (positions) which are bright parts around the position P of the dark part to be calculated. In the example of FIG. 15, the positions of the three spots SP _A , SP _B , and SP _C are detected.

Then, the signal processor 43, three spots SP _A, SP _B, and the position of the dark part of the respective periphery of the SP _C A ', B', and, 'detection signal I _A of' C, I _B ' , And I _C'get . Then, the signal processing unit 43 sets any of the positions A', B', or C'of the dark part as the position Q (either Q = A', B', or C'), and sets the dark part to be calculated. to the calculated detection signal I _P position P, the similarity w _PQ and the detection signal I _Q 'position Q a (eta). Here, η represents the difference (η = (I _P −I _Q ′)) _{between the detection signal I P} of the position P of the dark part to be calculated _{and the detection signal I Q} ′ of the position Q of the surrounding dark part. The similarity w _PQ (η) can be a function such that the smaller the difference η of the detected signals, the greater the similarity. As the function of the similarity w (η), for example, a Gaussian function or the like can be used.

Next, the signal processing unit 43 sets the positions A', B', and C'of the dark areas around the _{three spots SP A} , SP _B , and SP _{C, respectively, and the position P of the dark area to be calculated.} With the similarity w _PQ (η) as the weights of the surrounding dark areas A', B', and C', the fusion distance measurement of the dark area position P to be calculated is performed by the weighted average of the following equation (5). to calculate the value DF _P.

As described above, the signal processing unit 43 has the dark areas (positions A', B) around each of the _{plurality of bright areas (spots SP A} , SP _B , and SP _C ) around the position P of the dark area to be calculated. ', and, C' detection signal I _Q 'detection signals I _a' of), I _B ', and I _C' a), as a reference, the method of bilateral upsampling, fusion distance value DF _P position P Is calculated. The resolution of the distance measurement value can be improved by using the detection signal of the same frame as a reference.

<7. Distance measurement processing>
FIG. 16 is a flowchart of the distance measurement process including the above-mentioned high resolution process.

Since the processes of steps S61 to S66 of FIG. 16 are the same as the processes of steps S1 to S6 shown in FIG. 9, the description thereof will be omitted.

After the processing of steps S61 to S66, the high resolution processing of step S67 is executed. That is, in step S67, the signal processing unit 43 refers to the position P of the dark portion to be calculated at the position between the plurality of spots SP constituting the pattern light 15, and is located in the dark portion around the bright portion around the position P. Using the detection signal I as a reference, the fusion distance measurement value DFP at position _P is calculated by the bilateral upsampling method. A plurality of positions P of the dark portion to be calculated can be set within the region of the pattern light 15.

Each and fusion distance measurement value DF plurality of spots SP constituting the pattern light 15, a high resolution and fusion distance value DF _P of the plurality of positions P added by treatment, fusion distance value after resolution enhancement Is output from the distance measuring device 21 to the subsequent stage, and the distance measurement process of FIG. 16 is completed.

<8. Modification example of distance measurement system>
17 and 18 show other configuration examples of the ranging system to which the present technology is applied.

The distance measuring system 1 of FIG. 17 differs from the distance measuring system 1 of FIG. 1 in that the light source device 11 of FIG. 1 is replaced with the light source device 111, and other points are configured in the same manner as in FIG. There is.

The light source device 111 irradiates not only the pattern light 15 having two types of brightness of the bright part and the dark part, but also the flat light 121 in which the emission brightness of the entire substantially rectangular area is uniform within a predetermined brightness range by switching. Can be done.

The light source device 111 irradiates the predetermined object OBJ with the pattern light 15 at the first timing. Further, the light source device 111 irradiates the predetermined object OBJ with the plane light 121 at the second timing. The first timing and the second timing may be arbitrary different timings, or may be timings having a predetermined predetermined interval. That is, the irradiation of the pattern light 15 and the irradiation of the plane light 121 may be individually performed independently or continuously with a fixed time relationship.

When the pattern light 15 is irradiated from the light source device 111, the distance measuring device 21 receives the reflected light reflected by the object OBJ and calculates the fusion distance measuring value DF for each of the plurality of spots SP. Further, when the plane light 121 is irradiated from the light source device 111, the distance measuring device 21 receives the reflected light reflected by the object OBJ, and measures each pixel 71 of the pixel array unit 63 by the ToF method. The distance value D3 is calculated.

The distance measuring device 21 can output the calculated fusion distance measuring value DF or the distance measuring value D3 as a measurement result.

Further, when the distance measuring device 21 executes the distance measuring process including the high resolution processing, the distance measuring device 21 uses the distance measuring value D3 measured by the plane light 121 as a reference and uses the bilateral upsampling method to perform the fusion distance measuring of the position P. It calculates the value DF _P, it is possible to output the fusion distance value after resolution enhancement.

It should be noted that the light source device 111 does not have a configuration in which the pattern light 15 and the plane light 121 are switched and irradiated, but a light source device for irradiating the plane light 121 is additionally provided separately from the light source device 11 for irradiating the pattern light 15. May be good.

The distance measuring system 1 of FIG. 18 further includes an RGB sensor 141 capable of capturing visible light including RGB wavelengths and a light receiving side optical system 142 that collects visible light incident on the RGB sensor 141. Unlike the distance system 1, other points are configured in the same manner as in FIG.

The RGB sensor 141 has the same imaging range as the light receiving range of the ranging device 21, captures the same subject as the ranging device 21, and generates an captured image.

When the distance measuring device 21 executes the distance measurement processing including the high resolution processing, the position is determined by the bilateral upsampling method with reference to the image captured by the RGB sensor 141 at the same time as the distance measuring device 21. The fusion distance measurement value of _{P DFP} can be calculated and the fusion distance measurement value after the high resolution processing can be output.

Further, instead of the RGB sensor 141, an IR sensor capable of capturing infrared light may be used. _{In this case, the fusion distance measurement value DFP of the position P} is calculated by the bilateral upsampling method using the captured image obtained at the same time as the distance measurement device 21 by the IR sensor as a reference, and after the high resolution processing. It is possible to output the fusion distance measurement value of.

The combination of each configuration of the distance measuring system 1 of FIGS. 17 and 18, for example, the light source device 111, the light emitting side optical system 12, the distance measuring device 21, the light receiving side optical system 22, the RGB sensor 141, and the light receiving side optical system. A configuration having 142 is also possible. In this case, the distance measurement value D3 measured by the plane light 121 can be used as a reference, or the fusion distance measurement value after the high resolution processing can be output by using the image captured by the RGB sensor 141 as a reference. it can.

<9. Configuration example of electronic device>
The distance measuring system 1 described above can be mounted on electronic devices such as smartphones, tablet terminals, mobile phones, personal computers, game machines, television receivers, wearable terminals, digital still cameras, and digital video cameras.

FIG. 19 is a block diagram showing a configuration example of a smartphone as an electronic device equipped with the distance measuring system 1.

As shown in FIG. 19, in the smartphone 201, the distance measuring module 202, the image pickup device 203, the display 204, the speaker 205, the microphone 206, the communication module 207, the sensor unit 208, the touch panel 209, and the control unit 210 are connected via the bus 211. Is connected and configured. Further, the control unit 210 has functions as an application processing unit 221 and an operation system processing unit 222 by executing a program by the CPU.

The distance measuring system 1 of FIG. 1 is applied to the distance measuring module 202. For example, the distance measuring module 202 is arranged in front of the smartphone 201, and by performing distance measurement for the user of the smartphone 201, the depth value of the surface shape of the user's face, hand, finger, etc. is measured as a distance measurement result. Can be output as.

The image pickup device 203 is arranged in front of the smartphone 201, and by taking an image of the user of the smartphone 201 as a subject, the image taken by the user is acquired. Although not shown, the image pickup device 203 may be arranged on the back surface of the smartphone 201.

The display 204 displays an operation screen for performing processing by the application processing unit 221 and the operation system processing unit 222, an image captured by the image pickup device 203, and the like. The speaker 205 and the microphone 206, for example, output the voice of the other party and collect the voice of the user when making a call by the smartphone 201.

The communication module 207 communicates via the communication network. The sensor unit 208 senses speed, acceleration, proximity, etc., and the touch panel 209 acquires a touch operation by the user on the operation screen displayed on the display 204.

The application processing unit 221 performs processing for providing various services by the smartphone 201. For example, the application processing unit 221 can create a face by computer graphics that virtually reproduces the user's facial expression based on the depth supplied from the distance measuring module 202, and can perform a process of displaying the face on the display 204. Further, the application processing unit 221 can perform a process of creating, for example, three-dimensional shape data of an arbitrary three-dimensional object based on the depth supplied from the distance measuring module 202.

The operation system processing unit 222 performs processing for realizing the basic functions and operations of the smartphone 201. For example, the operation system processing unit 222 can perform a process of authenticating the user's face and unlocking the smartphone 201 based on the depth value supplied from the distance measuring module 202. Further, the operation system processing unit 222 performs, for example, a process of recognizing a user's gesture based on the depth value supplied from the distance measuring module 202, and performs a process of inputting various operations according to the gesture. Can be done.

In the smartphone 201 configured in this way, by applying the distance measuring system 1 described above, for example, a depth map can be generated with high accuracy and high speed. As a result, the smartphone 201 can detect the distance measurement information more accurately.

<10. Application example to mobile>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on a moving body of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 20 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technology according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 20, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating a braking force of a vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives inputs of these radio waves or signals and controls a vehicle door lock device, a power window device, a lamp, and the like.

The vehicle outside information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image pickup unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or characters on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects the in-vehicle information. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (ADvanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, or vehicle lane deviation warning. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, so that the driver can control the driver. It is possible to perform coordinated control for the purpose of automatic driving, etc., which runs autonomously without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio image output unit 12052 transmits the output signal of at least one of the audio and the image to the output device capable of visually or audibly notifying the passenger or the outside of the vehicle of the information. In the example of FIG. 20, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 21 is a diagram showing an example of the installation position of the imaging unit 12031.

In FIG. 21, the vehicle 12100 has

image pickup units

12101, 12102, 12103, 12104, 12105 as the image pickup unit 12031.

The

imaging units

12101, 12102, 12103, 12104, 12105 are provided at positions such as the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100, for example. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The images in front acquired by the

imaging units

12101 and 12105 are mainly used for detecting a preceding vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 21 shows an example of the photographing range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the

imaging units

12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining, it is possible to extract as the preceding vehicle a three-dimensional object that is the closest three-dimensional object on the traveling path of the vehicle 12100 and that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, 0 km / h or more). it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform coordinated control for the purpose of automatic driving or the like in which the vehicle travels autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, electric poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that can be seen by the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

The above is an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040 among the configurations described above. Specifically, by using the distance measurement by the distance measurement system 1 as the outside information detection unit 12030 and the inside information detection unit 12040, processing for recognizing the driver's gesture is performed, and various types according to the gesture (for example, It can perform operations on audio systems, navigation systems, air conditioning systems) and detect the driver's condition more accurately. Further, the distance measurement by the distance measurement system 1 can be used to recognize the unevenness of the road surface and reflect it in the control of the suspension.

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

The present techniques described above in this specification can be independently implemented independently as long as there is no contradiction. Of course, any plurality of the present technologies can be used in combination. For example, some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. It is also possible to carry out a part or all of any of the above-mentioned techniques in combination with other techniques not described above.

Further, for example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the contrary, the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Further, of course, a configuration other than the above may be added to the configuration of each device (or each processing unit). Further, if the configuration and operation of the entire system are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..

Further, in the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

It should be noted that the effects described in the present specification are merely examples and are not limited, and effects other than those described in the present specification may be obtained.

The present technology can have the following configurations.
(1)
A ranging sensor that receives the reflected light that is reflected by an object and has two types of brightness, a bright part and a dark part that are emitted from the light source device.
The time from the irradiation of the pattern light to the reception of the reflected light is detected as a phase difference, and the first distance measurement value, which is the distance to the object, is calculated based on the phase difference, and the above-mentioned The position of the bright part of the pattern light is detected, the second distance measurement value which is the distance to the object is calculated by the principle of triangulation using the detected position of the bright part, and the first distance measurement value. A distance measuring device including a signal processing unit for calculating a fusion distance measuring value obtained by fusing the second distance measuring value and the second distance measuring value.
(2)
The signal processing unit detects the phase difference based on the difference between the detection signal of the pixel corresponding to the bright part of the pattern light and the detection signal of the pixel corresponding to the dark part of the pattern light (1). The distance measuring device described in.
(3)
The signal processing unit detects the position of the bright portion of the pattern light, acquires the position of the detected light source device in the bright portion, and calculates the second ranging value by the principle of triangulation. The distance measuring device according to (1) or (2).
(4)
The signal processing unit calculates the fusion distance measurement value based on the value of the distance evaluation index using the first distance measurement value and the second distance measurement value (1) to (3). The distance measuring device according to any one of.
(5)
The signal processing unit calculates the fusion distance measurement value based on the error indexes of each of the first method for calculating the first distance measurement value and the second method for calculating the second distance measurement value. The distance measuring device according to any one of (1) to (3) above.
(6)
The signal processing unit calculates the fusion distance measurement value based on whether or not the detection signal of the pixel at the time of detecting the phase difference shows saturation. Described in any one of (1) to (3). Distance measuring device.
(7)
The distance measuring device according to any one of (1) to (6) above, wherein the pattern light is a dot pattern in which dots are arranged at regular or irregular predetermined intervals.
(8)
The distance measuring device according to any one of (1) to (7) above, wherein the signal processing unit further executes a high resolution process for calculating a fusion distance measuring value at the position of the dark part.
(9)
The distance measuring device according to (8) above, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part between a plurality of bright parts by using a bilateral upsampling method.
(10)
The distance measuring device according to (9) above, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part with reference to a detection signal of a dark part around each of the plurality of bright parts.
(11)
The distance measuring device according to (9) or (10) above, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part by weighting averaging of the detection signals of the plurality of bright parts.
(12)
The signal processing unit calculates the first distance measurement value and the second distance measurement value using the detection signals of the same frame obtained from the distance measurement sensor (1) to (11). ). The distance measuring device according to any one of.
(13)
The signal processing unit calculates the fusion distance measurement value of the position of the dark portion by using the detection signal received by the distance measurement sensor as a reference for the plane light emitted by the light source device before or after the pattern light. The distance measuring device according to any one of (1) to (11).
(14)
The distance measuring device according to any one of (1) to (11) above, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part with reference to a detection signal received by the RGB sensor.
(15)
The distance measuring device
The time from when the pattern light having two kinds of brightness of the bright part and the dark part irradiated from the light source device is irradiated to when it is reflected by the object and received as the reflected light is detected as the phase difference, and is based on the phase difference. The first ranging value, which is the distance to the object, is calculated, the position of the bright part of the pattern light is detected, and the detected position of the bright part is used to reach the object by the principle of triangular measurement. A measuring method in which a second distance measurement value, which is a distance, is calculated, and a fusion distance measurement value obtained by fusing the first distance measurement value and the second distance measurement value is calculated.
(16)
A light source device that irradiates a pattern light having two types of brightness, a bright part and a dark part,
It is equipped with a ranging device that receives the reflected light that is reflected by the object and returned.
The distance measuring device is
A distance measuring sensor that receives the reflected light and
The time from the irradiation of the pattern light to the reception of the reflected light is detected as a phase difference, and the first distance measurement value, which is the distance to the object, is calculated based on the phase difference, and the above-mentioned The position of the bright part of the pattern light is detected, the second distance measurement value which is the distance to the object is calculated by the principle of triangulation using the detected position of the bright part, and the first distance measurement value is obtained. A distance measuring system including a signal processing unit that calculates a fusion distance measuring value in which the second distance measuring value is fused.
(17)
The distance measuring system according to (16), wherein the light source device irradiates the pattern light and a flat light whose emission brightness of the entire area is uniform within a predetermined brightness range by switching.

1 distance measurement system, 11 light source device, 15 pattern light, 21 distance measurement device, 31 light emission source, 32 light source drive unit, 41 synchronization control unit, 42 distance measurement sensor, 43 signal processing unit, 63 pixel array unit, 66 signal processing Department, 71 pixels, 81 photoelectric conversion element, 111 light source device, 121 plane light, 141 RGB sensor, 201 smartphone, 202 ranging module

Claims

A ranging sensor that receives the reflected light that is reflected by an object and has two types of brightness, a bright part and a dark part that are emitted from the light source device.
The time from the irradiation of the pattern light to the reception of the reflected light is detected as a phase difference, and the first distance measurement value, which is the distance to the object, is calculated based on the phase difference, and the above-mentioned The position of the bright part of the pattern light is detected, the second distance measurement value which is the distance to the object is calculated by the principle of triangulation using the detected position of the bright part, and the first distance measurement value. A distance measuring device including a signal processing unit for calculating a fusion distance measuring value obtained by fusing the second distance measuring value and the second distance measuring value.
The signal processing unit detects the phase difference based on the difference between the detection signal of the pixel corresponding to the bright part of the pattern light and the detection signal of the pixel corresponding to the dark part of the pattern light. The distance measuring device described.
The signal processing unit detects the position of the bright portion of the pattern light, acquires the position of the detected light source device in the bright portion, and calculates the second ranging value by the principle of triangulation. Item 1. The distance measuring device according to item 1.
The distance measurement according to claim 1, wherein the signal processing unit calculates the fusion distance measurement value based on the value of the distance evaluation index using the first distance measurement value and the second distance measurement value. apparatus.
The signal processing unit calculates the fusion distance measurement value based on the error indexes of each of the first method for calculating the first distance measurement value and the second method for calculating the second distance measurement value. The distance measuring device according to claim 1.
The distance measuring device according to claim 1, wherein the signal processing unit calculates the fusion distance measuring value based on whether or not the detection signal of the pixel at the time of detecting the phase difference shows saturation.
The distance measuring device according to claim 1, wherein the pattern light is a dot pattern in which dots are arranged at regular or irregular predetermined intervals.
The distance measuring device according to claim 1, wherein the signal processing unit further executes a high-resolution processing for calculating a fusion distance measuring value at the position of the dark portion.
The distance measuring device according to claim 8, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part between a plurality of bright parts by using a bilateral upsampling method.
The distance measuring device according to claim 9, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part with reference to a detection signal of a dark part around each of the plurality of bright parts.
The distance measuring device according to claim 9, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part by weighting averaging of the detection signals of the plurality of bright parts.
The measurement according to claim 1, wherein the signal processing unit calculates the first distance measurement value and the second distance measurement value by using the detection signal of the same frame obtained from the distance measurement sensor. Distance device.
The signal processing unit calculates the fusion distance measurement value of the position of the dark portion by using the detection signal received by the distance measurement sensor as a reference for the plane light emitted by the light source device before or after the pattern light. Item 9. The ranging device according to item 9.
The distance measuring device according to claim 9, wherein the signal processing unit calculates a fusion distance measuring value of the position of the dark part with reference to a detection signal received by the RGB sensor.
The distance measuring device
The time from when the pattern light having two kinds of brightness of the bright part and the dark part irradiated from the light source device is irradiated to when it is reflected by the object and received as the reflected light is detected as the phase difference, and is based on the phase difference. The first ranging value, which is the distance to the object, is calculated, the position of the bright part of the pattern light is detected, and the detected position of the bright part is used to reach the object by the principle of triangular measurement. A measuring method in which a second distance measurement value, which is a distance, is calculated, and a fusion distance measurement value obtained by fusing the first distance measurement value and the second distance measurement value is calculated.
A light source device that irradiates a pattern light having two types of brightness, a bright part and a dark part,
It is equipped with a ranging device that receives the reflected light that is reflected by the object and returned.
The distance measuring device is
A distance measuring sensor that receives the reflected light and
The time from the irradiation of the pattern light to the reception of the reflected light is detected as a phase difference, and the first distance measurement value, which is the distance to the object, is calculated based on the phase difference, and the above-mentioned The position of the bright part of the pattern light is detected, the second distance measurement value which is the distance to the object is calculated by the principle of triangulation using the detected position of the bright part, and the first distance measurement value is obtained. A distance measuring system including a signal processing unit that calculates a fusion distance measuring value in which the second distance measuring value is fused.
The distance measuring system according to claim 16, wherein the light source device irradiates the pattern light and flat light whose emission brightness of the entire area is uniform within a predetermined brightness range by switching.