JP2021110679A - Ranging sensor, ranging system, and electronic device - Google Patents

Abstract

To enable high-speed control, such as switching of light emission conditions, of an illumination device.SOLUTION: A ranging sensor provided herein comprises: a pixel array unit consisting of two-dimensionally arranged pixels configured to receive reflection of irradiation light emitted from an illumination device and reflected by an object, and output a detection signal corresponding to intensity of the received light; and a control unit configured to control emission of light by the illumination device by setting multiple values to parameters of the illumination device according to a measurement start instruction from another device. The technique disclosed herein can be applied, for example, to ranging modules for measuring distance to an object.SELECTED DRAWING: Figure 15

Description

This technology relates to distance measurement sensors, distance measurement systems, and electronic devices, and in particular, distance measurement sensors, distance measurement systems, and distance measurement systems that enable high-speed control of lighting devices such as switching of light emission conditions. Regarding electronic devices.

In recent years, advances in semiconductor technology have led to miniaturization of distance measuring modules that measure the distance to an object. As a result, for example, it has been realized that a distance measuring module is mounted on a mobile terminal such as a so-called smartphone, which is a small information processing device having a communication function.

As a distance measuring method in the distance measuring module, for example, an indirect ToF (Time of Flight) method can be mentioned. In the indirect ToF method, the irradiation light is emitted toward the object, the reflected light is reflected on the surface of the object and returned, and the light is emitted until the reflected light is received. This method calculates the distance to an object based on the flight time.

In order to measure the flight time, the timing of emission of the irradiation light is controlled by outputting an emission pulse from the distance measuring sensor side that receives the reflected light to the illumination device that emits the irradiation light (for example, Patent Document 1). reference).

International Publication No. 2019/0444487

While the distance measuring sensor outputs a light emission pulse, a certain amount of time is required because the host at the upper level calibrates the light emission intensity and the like.

The present technology has been made in view of such a situation, and enables high-speed control of the lighting device such as switching of light emission conditions.

In the distance measuring sensor on the first side of the present technology, the pixels that receive the reflected light that is reflected by the object and return the irradiation light emitted from the lighting device and output the detection signal according to the amount of light received are two-dimensional. An arranged pixel array unit and a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device.

The distance measuring system on the second side of the present technology includes a lighting device that irradiates an object with irradiation light, and a distance measuring sensor that receives the reflected light that is reflected by the object and returned. The distance measuring sensor is attached to the lighting device based on a pixel array unit in which pixels that receive the reflected light and output a detection signal according to the amount of received light are arranged in two dimensions and a measurement start instruction from another device. , A control unit that sequentially sets a plurality of values of parameters and controls light emission of the lighting device is provided.

The electronic device on the third aspect of the present technology includes a lighting device that irradiates an object with irradiation light, and a distance measuring sensor that receives the reflected light that is reflected by the object and returned. The distance sensor receives the reflected light and outputs a detection signal according to the amount of received light to the lighting device based on a pixel array unit in which pixels are arranged in two dimensions and a measurement start instruction from another device. A distance measuring system including a control unit that sequentially sets a plurality of values of parameters and controls light emission of the lighting device is provided.

In the first to third aspects of the present technology, a plurality of values of parameters are sequentially set in the lighting device based on a measurement start instruction from another device, and light emission of the lighting device is controlled.

The distance measuring sensor, the distance measuring system, and the electronic device may be independent devices or may be modules incorporated in other devices.

It is a block diagram which shows the configuration example of the distance measurement system to which this technology is applied. It is a figure explaining the distance measurement principle of an indirect ToF method. It is a figure explaining the distance measurement principle of an indirect ToF method. It is a block diagram which shows the detailed configuration example of a lighting device and a distance measuring sensor. It is a figure explaining the detail of the pixel array part. It is a figure explaining an example of switching of irradiation light. It is a figure explaining an example of switching of irradiation light. It is a figure which shows the measurement example in the measurement mode. It is a flowchart explaining the measurement process in a measurement mode. It is a block diagram which shows the measurement environment of the distance measurement system in the calibration mode. It is a figure which shows the control example of a parameter in a calibration mode. It is a figure which shows the other control example of a parameter in a calibration mode. It is a figure explaining the example of the parameter adjusted in a calibration mode. It is a figure explaining the example of the parameter adjusted in a calibration mode. It is a block diagram which shows the detailed configuration example of a lighting apparatus and a distance measuring sensor in a calibration mode. It is a flowchart explaining the measurement process in a calibration mode. It is a block diagram which shows the detailed configuration example of a lighting apparatus and a distance measuring sensor in a calibration mode. It is a figure explaining the calibration process during measurement. It is a flowchart explaining the calibration process during distance measurement. It is a perspective view which shows the chip structure example of the distance measuring sensor. It is a block diagram of the distance measuring system which performs another light emission control method as a comparative example. It is a figure explaining the effect as compared with other light emission control methods. It is a block diagram which shows the structural example of the electronic device to which this technology is applied. It is a block diagram which shows an example of the schematic structure of a vehicle control system. It is explanatory drawing which shows an example of the installation position of the vehicle exterior information detection unit and the image pickup unit.

Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted. The explanation will be given in the following order.
1. 1. Schematic configuration example of the ranging system 2. Indirect To F method distance measurement principle 3. Configuration example of distance measuring sensor and lighting device 4. Execution example of measurement mode 5. Execution example of calibration mode 6. Example of device configuration in calibration mode 7. Calibration process during measurement 8. Example of chip configuration of distance measuring sensor 9. Comparison with other light emission control methods 10. Application example to electronic devices 11. Application example to mobile

<1. Schematic configuration example of distance measurement system>
FIG. 1 is a block diagram showing a configuration example of a distance measuring system to which the present technology is applied.

The distance measuring system 1 is composed of a lighting device 11 and a distance measuring sensor 12, and is designated as a subject according to an instruction from a host control unit 13 which is a control unit of a host device in which the distance measuring system 1 is incorporated. The distance to the object is measured, and the distance measurement data is output to the host control unit 13.

More specifically, the lighting device 11 has, for example, an infrared laser diode as a light source, and with respect to a predetermined object as a subject based on a light emitting pulse and a light emitting condition supplied from the distance measuring sensor 12. Irradiate the irradiation light. The emission pulse is a pulse signal having a predetermined modulation frequency (for example, 20 MHz) indicating the timing of emission (on / off), and the emission condition includes, for example, light source setting information such as emission intensity, irradiation area, and irradiation method. The lighting device 11 emits light while being modulated according to the light emission pulse under the light emitting conditions supplied from the distance measuring sensor 12.

The distance measuring sensor 12 acquires a distance measuring start trigger indicating the start of distance measurement and a light emitting condition from the host control unit 13, supplies the acquired light emitting condition to the lighting device 11, and generates a light emitting pulse. It supplies to the lighting device 11 and controls the light emission of the lighting device 11.

Further, the distance measuring sensor 12 receives the reflected light that is reflected by the object and returned from the irradiation light emitted from the lighting device 11 based on the generated emission pulse, and generates the distance measuring data based on the received light reception result. Then, it is output to the host control unit 13.

The host control unit 13 controls the entire host device in which the distance measurement system 1 is incorporated, and sets a light emitting condition when the lighting device 11 irradiates the irradiation light and a distance measurement start trigger indicating the start of distance measurement. It is supplied to the distance measuring sensor 12. Distance measurement data is supplied from the distance measurement sensor 12 in response to the distance measurement start trigger. The host control unit 13 is, for example, an arithmetic unit such as a CPU (central processing unit), an MPU (microprocessor unit), an FPGA (field-programmable gate array)) mounted on the host apparatus, or an application program that operates on the arithmetic unit. Consists of. Further, for example, when the host device is composed of a smartphone, the host control unit 13 is composed of an AP (application processor) or an application program running on the AP (application processor).

The distance measuring system 1 configured as described above uses a predetermined distance measuring method such as an indirect ToF (Time of Flight) method, a direct ToF method, and a Structured Light method, and measures the distance based on the received light reception result of the reflected light. I do. The indirect ToF method is a method of calculating the distance to an object by detecting the flight time from when the irradiation light is emitted to when the reflected light is received as a phase difference. The direct ToF method is a method of directly measuring the flight time from when the irradiation light is emitted to when the reflected light is received and calculating the distance to the object. The Structured Light method is a method in which pattern light is irradiated as irradiation light and the distance to an object is calculated based on the distortion of the received pattern.

The distance measuring method executed by the distance measuring system 1 is not particularly limited, but the specific operation of the distance measuring system 1 will be described below by taking as an example the case where the distance measuring system 1 performs the distance measuring by the indirect ToF method.

<2. Indirect To F method distance measurement principle>
First, the distance measurement principle of the indirect ToF method will be briefly described with reference to FIGS. 2 and 3.

The depth value d [mm] corresponding to the distance from the distance measuring system 1 to the object can be calculated by the following equation (1).

Δt in the formula (1) is the time until the irradiation light emitted from the lighting device 11 is reflected by the object and is incident on the distance measuring sensor 12, and c is the speed of light.

As the irradiation light emitted from the illumination device 11, pulsed light having a light emission pattern that repeats on / off at a high speed at a predetermined modulation frequency f is adopted as shown in FIG. One cycle T of the light emission pattern is 1 / f. In the distance measuring sensor 12, the reflected light (light receiving pattern) is detected out of phase according to the time Δt from the lighting device 11 to the distance measuring sensor 12. Assuming that the amount of phase shift (phase difference) between the light emitting pattern and the light receiving pattern is φ, the time Δt can be calculated by the following equation (2).

Therefore, the depth value d from the distance measuring system 1 to the object can be calculated from the equations (1) and (2) by the following equation (3).

Next, the above-mentioned method for calculating the phase difference φ will be described.

Each pixel of the pixel array formed in the distance measuring sensor 12 repeats ON / OFF at high speed corresponding to the modulation frequency, and accumulates electric charges only during the ON period.

The distance measuring sensor 12 sequentially switches the ON / OFF execution timing of each pixel of the pixel array, accumulates the electric charge at each execution timing, and outputs a detection signal according to the accumulated electric charge.

There are four types of ON / OFF execution timings, for example, phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree.

The execution timing of the phase 0 degree is a timing at which the ON timing (light receiving timing) of each pixel of the pixel array is set to the phase of the pulsed light emitted by the lighting device 11, that is, the same phase as the light emission pattern.

The execution timing of the phase 90 degrees is a timing in which the ON timing (light receiving timing) of each pixel of the pixel array is set to a phase delayed by 90 degrees from the pulsed light (emission pattern) emitted by the lighting device 11.

The execution timing of the phase 180 degrees is a timing in which the ON timing (light receiving timing) of each pixel of the pixel array is set to a phase 180 degrees behind the pulsed light (emission pattern) emitted by the lighting device 11.

The execution timing of the phase 270 degrees is a timing in which the ON timing (light receiving timing) of each pixel of the pixel array is set to a phase delayed by 270 degrees from the pulsed light (emission pattern) emitted by the illumination device 11.

For example, the distance measuring sensor 12 sequentially switches the light receiving timing in the order of phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree, and acquires the light receiving amount (accumulated charge) of the reflected light at each light receiving timing. In FIG. 2, in the light receiving timing (ON timing) of each phase, the timing at which the reflected light is incident is shaded.

As shown in FIG. 2, when the light receiving timing is set to phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree, the accumulated charges are Q ₀ , Q ₉₀ , Q ₁₈₀ , respectively. _and, when _{Q 270,} the phase difference _{φ, Q 0, Q 90,} Q 180 and, _using a _{Q 270,} can be calculated by the following equation (4).

By inputting the phase difference φ calculated by the equation (4) into the above equation (3), the depth value d from the distance measuring system 1 to the object can be calculated.

Further, the reliability conf is a value representing the intensity of the light received by each pixel, and can be calculated by, for example, the following equation (5).

The distance measuring sensor 12 calculates the depth value d, which is the distance from the distance measuring system 1 to the object, based on the detection signal supplied for each pixel of the pixel array. Then, a depth map in which the depth value d is stored as the pixel value of each pixel and a reliability map in which the reliability conf is stored as the pixel value of each pixel are generated and output to the outside.

As will be described later, the distance measuring sensor 12 includes two charge storage units in each pixel of the pixel array. Assuming that these two charge storage units are referred to as a first tap and a second tap, by alternately accumulating charges in the two charge storage units of the first tap and the second tap, for example, the phase is 0 degree. It is possible to acquire the detection signals of two light receiving timings whose phases are inverted, such as 180 degrees in phase, in one frame.

Here, the distance measuring sensor 12 generates and outputs a depth map and a reliability map by either a 2Phase method or a 4Phase method.

The upper part of FIG. 3 shows the generation of the depth map of the 2 Phase method.

In the 2Phase method, as shown in the upper part of FIG. 3, the detection signals having a phase of 0 degrees and a phase of 180 degrees are acquired in the first frame, and in the next second frame, the phases are 90 degrees and 270 degrees. Since the detection signal of four phases can be acquired by acquiring the detection signal, the depth value d can be calculated by the equation (3).

In the 2Phase method, if the unit (1 frame) for generating the detection signal of 0 degree and 180 degree phase or 90 degree phase and 270 degree phase is called a microframe, the data of 4 phases are prepared in 2 microframes. The depth value d can be calculated for each pixel from the data of two microframes. Assuming that a frame in which this depth value d is stored as a pixel value of each pixel is referred to as a depth frame, one depth frame is composed of two microframes.

Further, the ranging sensor 12 acquires a plurality of depth frames by changing the light emission conditions such as the light emission intensity and the modulation frequency, and uses the plurality of depth frames to generate a final depth map. That is, one depth map is generated by using a plurality of depth frames. In the example of FIG. 3, a depth map is generated using three depth frames. One depth frame may be output as it is as a depth map. That is, one depth map can be composed of one depth frame.

The lower part of FIG. 3 shows the generation of the depth map of the 4 Phase method.

In the 4Phase method, as shown in the lower part of FIG. 3, following the first frame and the second frame, in the third frame, the detection signals of 180 degrees and 0 degrees of phase are acquired, and the next third frame. In the 4th frame, the detection signals having a phase of 270 degrees and a phase of 90 degrees are acquired. That is, the detection signals of all four phases of phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree are acquired at each of the first tap and the second tap, and the depth value d is calculated by the equation (3). It is calculated. Therefore, in the 4Phase method, one depth frame is composed of four microframes, and one depth map is generated by using a plurality of depth frames with different light emission conditions.

In the 4-Phase method, the detection signals of all four phases can be acquired by each tap (first tap and second tap), so that the characteristic variation between taps existing in each pixel, that is, the sensitivity difference between taps is eliminated. can do.

On the other hand, in the 2Phase method, the depth value d to the object can be obtained from the data of two microframes, so that the distance can be measured at a frame rate twice that of the 4Phase method. The characteristic variation between taps is adjusted by correction parameters such as gain and offset.

The distance measuring sensor 12 can be driven by either the 2Phase method or the 4Phase method, but for the sake of simplicity, it will be described below as being driven by the 2Phase method.

<3. Configuration example of distance measuring sensor and lighting device>
FIG. 4 is a block diagram showing a detailed configuration example of the lighting device 11 and the distance measuring sensor 12. Note that FIG. 4 also shows a host control unit 13 for easy understanding.

The distance measuring sensor 12 includes a control unit 31, a light emission timing control unit 32, a pixel modulation unit 33, a pixel control unit 34, a pixel array unit 35, a column processing unit 36, a data processing unit 37, an output IF 38, and an input / output terminal 39. -1 to 39-5.

The lighting device 11 includes a light emitting control unit 51, a light emitting source 52, and input / output terminals 53-1 and 53-2.

The host control unit 13 supplies light emission conditions to the control unit 31 of the distance measurement sensor 12 via the input / output terminals 39-1, and supplies a distance measurement start trigger via the input / output terminals 39-2. NS. The control unit 31 controls the operation of the entire distance measuring sensor 12 and the lighting device 11 based on the light emitting condition and the distance measuring start trigger.

More specifically, the control unit 31 uses information such as the light emission intensity, the irradiation area, and the irradiation method, which are a part of the light emission conditions, as the light source setting information based on the light emission conditions supplied from the host control unit 13. , Is supplied to the lighting device 11 via the input / output terminals 39-3.

Further, the control unit 31 supplies the light emission timing control unit 32 with information on the light emission period and the modulation frequency, which is a part of the light emission conditions, based on the light emission conditions supplied from the host control unit 13. The light emission period represents the exposure time per phase.

Further, the control unit 31 transmits the drive control information including the light receiving area of the pixel array unit 35 to the pixel control unit 34, the column processing unit 36, and the column processing unit 36 in accordance with the irradiation area and the irradiation method supplied to the lighting device 11. It is supplied to the data processing unit 37.

The light emission timing control unit 32 generates a light emission pulse based on the information of the light emission period and the modulation frequency supplied from the control unit 31, and supplies the light emission pulse to the lighting device 11 via the input / output terminals 39-4. The light emission pulse becomes a pulse signal having a modulation frequency supplied from the control unit 31, and the integration time of the high period in one microframe of the light emission pulse becomes the light emission period supplied from the control unit 31. The light emitting pulse is supplied to the lighting device 11 via the input / output terminals 39-4 at the timing corresponding to the distance measurement start trigger from the host control unit 13.

Further, the light emission timing control unit 32 generates a light receiving pulse for receiving the reflected light in synchronization with the light emitting pulse, and supplies the light receiving pulse to the pixel modulation unit 33. As described above, the light receiving pulse is a pulse signal whose phase is delayed by any one of phase 0 degrees, phase 90 degrees, phase 180 degrees, and phase 270 degrees with respect to the light emitting pulse.

The pixel modulation unit 33 switches the charge storage operation between the first tap and the second tap of each pixel of the pixel array unit 35 based on the light receiving pulse supplied from the light emission timing control unit 32.

The pixel control unit 34 controls the reset operation, the read operation, and the like of the accumulated charge of each pixel of the pixel array unit 35 based on the drive control information supplied from the control unit 31. For example, the pixel control unit 34 can partially drive only a part of the light receiving area including all the pixels, corresponding to the light receiving area supplied from the control unit 31 as a part of the drive control information. Further, for example, the pixel control unit 34 can also perform control such as thinning out the detection signals of each pixel in the light receiving area at predetermined intervals and adding (pixel addition) the detection signals of a plurality of pixels.

The pixel array unit 35 includes a plurality of pixels 71 (FIG. 5) arranged two-dimensionally in a matrix. Each pixel 71 of the pixel array unit 35 receives reflected light under the control of the pixel modulation unit 33 and the pixel control unit 34, and supplies a detection signal according to the amount of received light to the column processing unit 36.

Here, the details of the pixel array unit 35 will be described with reference to FIG.

As shown in FIG. 5, a plurality of pixels 71 are two-dimensionally arranged in a matrix in the pixel array unit 35.

The pixel 71 includes a photodiode 81, an FD (Floating Diffusion) unit 82A which is a charge storage unit corresponding to the first tap, and an FD (Floating Diffusion) unit 82B which is a charge storage unit corresponding to the second tap. ..

Further, the pixel 71 includes a transfer transistor 83A, a selection transistor 84A, and a reset transistor 85A, which are a plurality of pixel transistors that control charge accumulation in the FD unit 82A as the first tap, and an FD unit as the second tap. It includes a transfer transistor 83B, a selection transistor 84B, and a reset transistor 85B, which are a plurality of pixel transistors that control charge accumulation in the 82B.

The operation of the pixel 71 will be described.

First, a reset operation is performed to reset the excess charge before the start of exposure. Specifically, the pixel control unit 34 controls the distribution signals GDA and GDB and the reset signals RSA and RSB to High, and the transfer transistor 83A and the reset transistor 85A on the first tap side and the transfer transistor on the second tap side. Turn on the 83B and the reset transistor 85B. As a result, the electric charges accumulated in the FD unit 82A and the FD unit 82B are reset, and the accumulated electric charges of the photodiode 81 are reset. After the reset operation is completed, the transfer transistor 83A and the reset transistor 85A and the transfer transistor 83B and the reset transistor 85B on the second tap side are turned off.

Next, the exposure operation is started. Specifically, the pixel modulation unit 33 alternately controls the distribution signals GDA and GDB to High in synchronization with the received pulse, and transfers the transfer transistor 83A on the first tap side and the transfer transistor 83B on the second tap side. , Turn on alternately. As a result, the electric charge generated by the photodiode 81 is distributed to the FD unit 82A as the first tap or the FD unit 82B as the second tap. The operation of distributing the electric charge generated by the photodiode 81 to the first tap or the second tap is periodically repeated for a time corresponding to the light emission period of one microframe. The charges transferred via the transfer transistor 83A are sequentially stored in the FD unit 82A, and the charges transferred via the transfer transistor 83B are sequentially stored in the FD unit 82B.

Then, after the end of the exposure period, the pixel control unit 34 controls the selection signals ROA and ROB to High, so that the detection signal corresponding to the accumulated charge of the FD unit 82A, which is the first tap, and the FD, which is the second tap. A detection signal corresponding to the accumulated charge of the unit 82B is output to the column processing unit 36. That is, when the selection transistor 84A is turned on according to the selection signal ROA, the detection signal A corresponding to the amount of electric charge stored in the FD unit 82A is output from the pixel 71 via the signal line 86A. Similarly, when the selection transistor 84B is turned on according to the selection signal ROB, the detection signal B corresponding to the amount of electric charge stored in the FD unit 82B is output from the pixel 71 via the signal line 86B.

In this way, the pixel 71 distributes the electric charge generated by the reflected light received by the photodiode 81 to the first tap or the second tap according to the delay time ΔT, and outputs the detection signal A and the detection signal B. The detection signal A and the detection signal B are any of the four phases of phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree described above.

Returning to the description of FIG. 4, the column processing unit 36 includes a plurality of AD (Analog to Digital) conversion units, and the AD conversion unit provided for each pixel column of the pixel array unit 35 has a predetermined corresponding pixel sequence. Noise removal processing and AD conversion processing are performed on the detection signal output from the pixel 71. The detection signal after the AD conversion process is supplied to the data processing unit 37.

The data processing unit 37 calculates the depth value d of each pixel 71 based on the detection signal of each pixel 71 after AD conversion supplied from the column processing unit 36, and stores the depth value d as the pixel value of each pixel. Generates the depth frame. Further, the data processing unit 37 uses one or more depth frames to generate a depth map. Further, the data processing unit 37 calculates the reliability conf based on the detection signal of each pixel 71, and sets the reliability frame corresponding to the depth frame in which the reliability conf is stored as the pixel value of each pixel and the depth map. It also generates a corresponding confidence map. The data processing unit 37 supplies the generated depth map and reliability map to the output IF 38.

The output IF 38 converts the depth map and the reliability map supplied from the data processing unit 37 into the signal format of the input / output terminals 39-5 (for example, MIPI: Mobile Industry Processor Interface), and the input / output terminals 39-5. Output from. The depth map and reliability map output from the input / output terminals 39-5 are supplied to the host control unit 13 as distance measurement data.

In FIG. 4, the input / output terminals 39-1 to 39-5 and the input / output terminals 53-1 and 53-2 are divided into a plurality of input / output terminals 53-1 and 53-2 for convenience of explanation, but one terminal having a plurality of input / output contacts ( It may be composed of terminal groups). It is also possible to set light emission conditions and light source setting information using serial communication such as SPI (Serial Peripheral Interface) or I2C (Inter-Integrated Circuit). When SPI or I2C serial communication is used, the distance measuring sensor 12 operates as the master side, and the light source setting information can be set to the lighting device 11 at an arbitrary timing. By using serial communication such as SPI or I2C, complicated and detailed settings can be made using registers.

The light emission control unit 51 of the lighting device 11 is composed of a laser driver or the like, and emits light based on the light source setting information and the light emission pulse supplied from the distance measuring sensor 12 via the input / output terminals 53-1 and 53-2. Drives the source 52.

The light emitting source 52 includes, for example, one or more laser light sources such as a VCSEL (Vertical Cavity Surface Emitting Laser). The light emitting source 52 emits irradiation light in a predetermined light emitting intensity, irradiation area, irradiation method, modulation frequency, and light emitting period according to the drive control of the light emitting control unit 51.

6 and 7 show an example of switching the irradiation light by the light emission control unit 51, assuming that the light emitting source 52 includes three types of laser light sources LD1 to LD3 having different light emitting characteristics.

FIG. 6A shows an example in which the light emission control unit 51 switches the irradiation method of the irradiation light.

Specifically, the light emission control unit 51 includes a first laser light source LD1 that irradiates a predetermined irradiation area with a uniform light emission intensity within a predetermined brightness range, and a plurality of laser light sources LD1 arranged at predetermined intervals. The second laser light source LD2 that performs spot irradiation with the spot (circle) as the irradiation area is switched.

FIG. 6B shows an example in which the light emission control unit 51 switches the light emission intensity of the irradiation light.

Specifically, the light emission control unit 51 has a first laser light source LD1 that performs surface irradiation with a predetermined light emission intensity (standard light emission intensity) and a surface whose light emission intensity is set to be larger than that of the first laser light source LD1. It is switched to the third laser light source LD3 that performs irradiation.

The emission intensity can be changed by a method of switching the laser light source LD or a method of changing the voltage supplied to one laser light source LD (for example, the laser light source LD1).

FIG. 7A shows an example in which the light emission control unit 51 switches the irradiation area of the irradiation light.

Specifically, the light emission control unit 51 switches between full-scale irradiation that irradiates the entire area that can be irradiated by the first laser light source LD1 and partial irradiation that limits the irradiation area to a part. The irradiation area for partial irradiation is, for example, the central portion of the entire area that can be irradiated. The irradiation area can be changed, for example, by driving a diffuser plate or a projection lens arranged in front of the first laser light source LD1. Alternatively, when the first laser light source LD1 is composed of a plurality of light emitting elements, the irradiation area can be narrowed down by limiting the light emitting elements to emit light.

FIG. 7B shows an example in which the light emission control unit 51 switches the modulation frequency of the irradiation light.

Specifically, the light emission control unit 51 uses the first laser light source LD1 to switch between irradiation that emits light at a modulation frequency of 20 MHz and irradiation that emits light at a modulation frequency of 100 MHz. The change of the modulation frequency is realized by changing the frequency of the emission pulse supplied from the distance measuring sensor 12.

The examples described with reference to FIGS. 6 and 7 are examples in which only one light emitting condition is switched, but of course, a plurality of light emitting conditions can be arbitrarily combined and changed at the same time. For example, it is possible to switch the emission intensity and the modulation frequency at the same time, or to switch the emission intensity and the irradiation area at the same time.

Further, the number of laser light source LDs to emit light is not limited to one, and two or more may be emitted at the same time. For example, by simultaneously emitting the first laser light source LD1 having a standard emission intensity and the third laser light source LD3 having a higher emission intensity, the emission intensity is further increased than that of the third laser light source LD3 alone. It is possible to irradiate the laser.

<4. Measurement mode execution example>
The ranging system 1 has a calibration mode and a measurement mode as operation modes. In the calibration mode, the distance measuring system 1 adjusts, for example, the synchronization of the light emitting timing of the lighting device 11 and the light receiving timing of the distance measuring sensor 12, and adjusts the light emitting intensity. In the measurement mode, the distance to a predetermined object is actually measured according to the measurement start instruction (distance measurement start trigger) from the host control unit 13. The calibration mode is executed, for example, at the time of shipping inspection before shipping the ranging system 1, or during a period (idle time) during which measurement by the measurement mode is not performed.

In the following, first, a measurement embodiment by the distance measuring system 1 that is executed when the operation mode is the measurement mode will be described. The operation of the calibration mode will be described after the measurement mode, but when the measurement mode is executed, it is assumed that the calibration process in the calibration mode is performed at least before shipping.

FIG. 8 is a sequence diagram of a measurement example executed by the ranging system 1 in the measurement mode.

Before the distance measurement start trigger is supplied from the host control unit 13, the distance measurement sensor 12 is supplied with a set of light emission conditions including a plurality of light emission conditions. Then, when the distance measurement start trigger is supplied from the host control unit 13 after acquiring one set of light emission conditions, the distance measurement sensor 12 sequentially illuminates a plurality of light emission conditions designated by the host control unit 13. It is set in the device 11 and performs measurement (light receiving operation) based on the specified light emitting condition.

In FIG. 8, a light emitting condition in which three types of light emitting conditions A, B, and C are set is supplied from the host control unit 13 to the distance measuring sensor 12, and the distance measuring sensor 12 supplies the three types of light emitting conditions A. , B, and C are sequentially set in the lighting device 11, and an example of performing measurement is shown. Here, the light emission condition A is, for example, a condition for performing full-scale irradiation at a modulation frequency of 20 MHz with a standard light emission intensity, and the light emission condition B is, for example, a condition for performing spot irradiation with a standard light emission intensity at a modulation frequency of 20 MHz. The light emission condition C is, for example, a condition in which the entire surface is irradiated with a large light emission intensity and a modulation frequency of 20 MHz.

FIG. 8 shows an example in which the distance measuring sensor 12 generates three depth frames under three types of light emission conditions A, B, and C, and generates and outputs one depth map from the three depth frames. There is. Since the number of microframes required for one depth frame is two in the case of the 2Phase method, one depth map is generated with a total of six microframes.

Specifically, when the distance measurement start trigger is supplied from the host control unit 13, the control unit 31 and the light emission timing control unit 32 of the distance measurement sensor 12 first emit light source setting information and light emission corresponding to the light emission condition A. The pulse is output to the illumination device 11 to emit the irradiation light under the light emission condition A, and the light receiving pulse is issued to drive each pixel 71 of the pixel array unit 35 to start receiving light.

Following the predetermined sensor start operation (StartUp), the distance measuring sensor 12 receives the reflected light under the light emission condition A to generate a first microframe having a phase of 0 degrees and a phase of 180 degrees, and then a phase of 90 degrees. And generate a second microframe with a phase of 270 degrees. Then, the distance measuring sensor 12 generates a first depth frame based on the first microframe and the second microframe.

Subsequently, the control unit 31 and the light emission timing control unit 32 of the ranging sensor 12 output the light source setting information and the light emission pulse corresponding to the light emission condition B to the lighting device 11, and cause the irradiation light to be emitted under the light emission condition B. At the same time, a light receiving pulse is issued to drive each pixel 71 of the pixel array unit 35 to start light receiving.

Following the predetermined sensor start operation (StartUp), the distance measuring sensor 12 receives the reflected light under the light emission condition B to generate a first microframe having a phase of 0 degrees and a phase of 180 degrees, and then a phase of 90 degrees. And generate a second microframe with a phase of 270 degrees. Then, the ranging sensor 12 generates a second depth frame based on the first microframe and the second microframe.

Subsequently, the control unit 31 and the light emission timing control unit 32 of the ranging sensor 12 output the light source setting information and the light emission pulse corresponding to the light emission condition C to the lighting device 11, and cause the irradiation light to be emitted under the light emission condition C. At the same time, a light receiving pulse is issued to drive each pixel 71 of the pixel array unit 35 to start light receiving.

Following the predetermined sensor start operation (StartUp), the distance measuring sensor 12 receives the reflected light under the light emission condition C to generate a first microframe having a phase of 0 degrees and a phase of 180 degrees, and then a phase of 90 degrees. And generate a second microframe with a phase of 270 degrees. Then, the distance measuring sensor 12 generates a third depth frame based on the first microframe and the second microframe.

Subsequently, the distance measuring sensor 12 generates one depth map (first depth map) and a reliability map from the first to third depth frames, and sends them to the host control unit 13 via the input / output terminals 39-5. Output.

This completes the operation corresponding to one distance measurement start trigger. The data processing unit 37 executes the processes of generating a microframe, generating a depth frame, and generating a depth map based on the result of receiving the reflected light. In the drive corresponding to the next distance measurement start trigger, for example, a depth map (second depth map) is generated under light emission conditions having different modulation frequencies.

As described above, the distance measuring sensor 12 acquires a set consisting of a plurality of light emitting conditions from the host control unit 13, and performs a light receiving operation while controlling the light emitting conditions of the lighting device 11 to be sequentially switched in depth frame units. (Exposure operation) can be performed.

Since the distance measuring sensor 12 knows the timing of its own light receiving operation (exposure operation), it can control to change the light emitting condition of the lighting device 11 according to its own operation.

<Flowchart of measurement processing in measurement mode>
Next, the measurement process by the distance measuring system 1 when the operation mode is the measurement mode will be described with reference to the flowchart of FIG. This process is started, for example, when a light emitting condition is supplied from the host control unit 13.

First, in step S1, the control unit 31 of the distance measuring sensor 12 acquires a set of light emission conditions including a plurality of light emission conditions supplied from the host control unit 13 and stores them in the internal memory.

In step S2, the control unit 31 substitutes 1 as an initial value for the variable N representing the number of types of light emission conditions.

In step S3, the control unit 31 determines whether the distance measurement start trigger has been supplied from the host control unit 13, and waits until the distance measurement start trigger is supplied.

If it is determined in step S3 that the distance measurement start trigger has been supplied, the process proceeds to step S4, and the control unit 31 causes the lighting device 11 to emit the irradiation light under the Nth light emission condition, and also by itself. The light receiving operation is started. More specifically, the control unit 31 outputs the light source setting information and the light emission pulse corresponding to the Nth light emission condition to the lighting device 11, and supplies the light emission period and the modulation frequency information to the light emission timing control unit 32. do. The light emission timing control unit 32 generates a light emission pulse based on the information of the light emission period and the modulation frequency, supplies the light emission pulse to the lighting device 11, and supplies the light reception pulse to the pixel modulation unit 33. Further, the control unit 31 supplies the drive control information including the light receiving area of the pixel array unit 35 to the pixel control unit 34, the column processing unit 36, and the data processing unit 37. The lighting device 11 emits irradiation light based on the light source setting information from the distance measuring sensor 12 and the light emitting pulse.

In step S5, each pixel 71 of the pixel array unit 35 receives the reflected light that is reflected by the object and returned from the irradiation light emitted from the illumination device 11 in accordance with the control of the pixel modulation unit 33 and the pixel control unit 34. Then, a detection signal corresponding to the amount of received light is supplied to the data processing unit 37 in pixel units.

In step S6, the data processing unit 37 calculates the depth frame and the reliability frame based on the detection signal of each pixel 71.

In step S7, the control unit 31 determines whether the measurement is performed under all the light emission conditions supplied from the host control unit 13.

If it is determined in step S7 that all the light emission conditions have not been measured yet, the process proceeds to step S8, and the control unit 31 increments the variable N indicating the number of types of light emission conditions by 1, and steps the process. Return to S4. As a result, the above-mentioned steps S4 to S7 are executed again, the light emitting conditions that have not been measured are set, and the light receiving operation is performed.

On the other hand, if it is determined in step S7 that the measurement is performed under all the light emission conditions, the process proceeds to step S9, and the data processing unit 37 starts from a plurality of depth frames corresponding to the plurality of light emission conditions to obtain one depth map. And a reliability map is generated and output to the host control unit 13 as distance measurement data.

This completes the measurement process when the operation mode is the measurement mode.

As described above, the distance measuring system 1 acquires a set of light emitting conditions including a plurality of light emitting conditions from the host control unit 13 before the distance measuring start trigger is supplied, and a plurality of light emitting conditions designated by the host control unit 13. The light emitting conditions of the above can be sequentially set in the lighting device 11, and the measurement (light receiving operation) based on the designated light emitting conditions can be performed. Since it is not necessary for the host control unit 13 to give a change instruction every time the light emitting condition is changed, the lighting device 11 can be controlled at high speed.

In the example of FIG. 8, the timing at which the light emission condition is changed is set in the depth frame unit, but it can also be performed in the micro frame unit.

Further, in the above-mentioned measurement process, an example in which a plurality of light emission conditions are acquired and measurement is performed has been described, but it goes without saying that only one type of light emission condition can be acquired and measurement can be performed.

<5. Calibration mode execution example>
Next, the operation of the ranging system 1 when the operation mode is the calibration mode will be described. The description of the parts common to the measurement mode described above will be omitted as appropriate.

FIG. 10 shows the measurement environment of the distance measuring system 1 when the operation mode is the calibration mode.

When the distance measuring system 1 executes the calibration process in the calibration mode, the distance measuring system 1 is arranged in, for example, an indoor space in which the ambient brightness is adjusted. A light emission observation device 101 that directly detects the irradiation light emitted from the illumination device 11 is arranged in front of the ranging system 1 which is the irradiation direction of the irradiation light. Alternatively, instead of the light emission observing device 101, a reflecting object such as a reflector having a known light reflectance is arranged. The distance from the distance measuring system 1 to the light emission observing device 101 is set to a fixed value. The same applies when a reflective object is placed.

In a state where the light emission observation device 101 is arranged at a position separated from the distance measuring system 1 by a fixed distance, the light emission observation device 10 acquires the intensity (emission intensity) of the irradiation light (emission intensity) and the irradiation range at a fixed distance from the lighting device 11. .. The data acquired by the light emission observing device 10 is supplied to the host control unit 13 or another data processing device.

In a state where the reflecting object is arranged at a position separated from the distance measuring system 1 by a fixed distance, the irradiation light emitted from the illuminating device 11 is reflected by the reflecting object, and the reflected light is detected by the distance measuring sensor 12 to detect the reflected light. The intensity (light emission intensity) and irradiation range are inspected. The data detected by the distance measuring sensor 12 is supplied to the host control unit 13.

The host control unit 13 supplies the setting parameters for adjustment to the distance measuring sensor 12. Then, the host control unit 13 supplies the distance measurement start trigger indicating the start of distance measurement to the distance measurement sensor 12. The distance measuring sensor 12 determines the light emitting condition based on the setting parameter supplied from the host control unit 13 and supplies the light emitting condition to the lighting device 11. When the distance measurement start trigger is supplied from the host control unit 13, the distance measurement sensor 12 generates a light emitting pulse and supplies the light emission pulse to the lighting device 11 to cause the lighting device 11 to emit light.

An example of a calibration process in which a reflecting object is installed at a fixed distance from the distance measuring system 1 and the reflected light reflected by the reflecting object is detected by the distance measuring sensor 12 will be described with reference to FIGS. 11 and 12. do.

FIG. 11 shows a control example of parameters supplied from the distance measuring sensor 12 to the lighting device 11 as a part of the light emitting condition based on the setting parameters supplied from the host control unit 13.

For example, the host control unit 13 designates the parameter A indicating the type (parameter name) of the parameter to be adjusted, the initial value V1 indicating the setting range of the parameter, and the end value V1end to the distance measuring sensor 12 as setting parameters.

As shown in FIG. 11, the ranging sensor 12 has a predetermined step width STEP from the initial value V1 to the end value V1end in units of depth frames according to its own light receiving operation. Then, the value of the parameter A is sequentially transmitted to the lighting device 11 and switched.

More specifically, when the distance measurement start trigger is supplied from the host control unit 13, the control unit 31 and the light emission timing control unit 32 of the distance measurement sensor 12 first transmit the set value V1 of the parameter A and the light emission pulse. It is output to the lighting device 11 to emit the irradiation light according to the set value V1 of the parameter A, and a light receiving pulse is issued to drive each pixel 71 of the pixel array unit 35 to start light receiving.

Following the predetermined sensor start operation (StartUp), the distance measuring sensor 12 receives the reflected light at the set value V1 of the parameter A, generates a first microframe having a phase of 0 degree and a phase of 180 degrees, and then generates a first microframe. A second microframe with 90 degrees and 270 degrees of phase is generated. Then, the distance measuring sensor 12 generates a first depth frame based on the first microframe and the second microframe.

Subsequently, the control unit 31 and the light emission timing control unit 32 of the distance measuring sensor 12 output the set value (V1 + STEP) of the parameter A and the light emitting pulse to the lighting device 11, and the irradiation light according to the set value (V1 + STEP) of the parameter A Along with causing light emission, a light receiving pulse is issued to drive each pixel 71 of the pixel array unit 35 to start light receiving.

Following the predetermined sensor start operation (StartUp), the distance measuring sensor 12 receives the reflected light at the set value (V1 + STEP) of the parameter A, generates a first microframe having a phase of 0 degree and a phase of 180 degrees, and then generates a first microframe. In addition, a second microframe having a phase of 90 degrees and a phase of 270 degrees is generated. Then, the ranging sensor 12 generates a second depth frame based on the first microframe and the second microframe.

Subsequently, the control unit 31 and the light emission timing control unit 32 of the distance measuring sensor 12 output the set value (V1 + STEP * 2) of the parameter A and the light emitting pulse to the lighting device 11, and the set value of the parameter A (V1 + STEP * 2). In addition to causing the irradiation light to be emitted by the above, a light receiving pulse is issued to drive each pixel 71 of the pixel array unit 35 to start light receiving.

The distance measuring sensor 12 receives the reflected light at the set value (V1 + STEP * 2) of the parameter A following the predetermined sensor start operation (StartUp), and generates the first microframe having a phase of 0 degrees and a phase of 180 degrees. Next, a second microframe having a phase of 90 degrees and a phase of 270 degrees is generated. Then, the distance measuring sensor 12 generates a third depth frame based on the first microframe and the second microframe.

The same process is repeated until the set value of parameter A becomes V1end = (V1 + STEP * (M-1)). The depth frame generated according to each set value of the parameter A is supplied to the host control unit 13 as distance measurement data.

There may be a plurality of setting parameters to be adjusted.

FIG. 12 shows an example of controlling parameters from the distance measuring sensor 12 to the lighting device 11 when a plurality of types of setting parameters are supplied from the host control unit 13 as setting parameters for adjustment.

For example, parameter A and parameter B are specified as the type (parameter name) of the parameter to be adjusted from the host control unit 13 to the distance measuring sensor 12, and the initial value V1 and the end value V1end are the parameter B as the setting range of the parameter A. The initial value V2 and the end value V2end are specified as the setting range of.

As shown in FIG. 11, the distance measuring sensor 12 sets one of the two parameters, for example, the value of the parameter A to the initial value V1, and sets the value of the parameter B from the initial value V2 to the end value V2end (=). Up to V2 + STEP * 3), the step width STEP is determined in advance, and the steps are sequentially switched. The value of the parameter A is the initial value V1, and each depth frame from the initial value V2 to the end value V2end (= V2 + STEP * 3) is acquired and supplied to the host control unit 13.

Next, the distance measuring sensor 12 sets the value of the parameter A to the next set value (V1 + STEP), and sets the value of the parameter B from the initial value V2 to the end value V2end in a predetermined step width STEP. Switch. When the value of parameter A is set as the set value (V1 + STEP), the depth frame is acquired for the value of parameter B from the initial value V2 to the end value V2end (= V2 + STEP * 3) and sent to the host control unit 13. Be supplied.

Similarly, the value of parameter A is set in order up to the end value V1end, and for all the values of parameter A, each depth frame when the value of parameter B is changed from the initial value V2 to the end value V2end in the step width STEP It is acquired and supplied to the host control unit 13 as distance measurement data.

As an example of controlling the parameters in the calibration mode described above, the parameter A, the initial value V1 indicating the setting range to be adjusted, and the end value V1end are supplied from the host control unit 13, and the distance measuring sensor 12 starts from the initial value V1 to the end value. It was decided to change the parameter values sequentially in the predetermined step width STEP until V1end was reached.

However, the host control unit 13 may also supply and specify the step width STEP. Further, in the above-mentioned example, the example of incrementing the step width STEP has been described, but the parameter value may be changed by decrementing.

In the parameter control example described with reference to FIG. 12, there are two examples of setting parameters to be adjusted, but the parameters can be controlled in the same manner even if there are three or more types of setting parameters to be adjusted.

Even in the calibration mode, since the distance measuring sensor 12 knows the timing of its own light receiving operation (exposure operation), it can control to change the parameters of the lighting device 11 according to its own operation.

When a predetermined set time is required for the lighting device 11 or the distance measuring sensor 12 due to the change of the parameter, a wait time may be set to wait until the setting is completed at the timing of parameter switching.

The example of the calibration process described with reference to FIGS. 11 and 12 is an example in which the parameters are changed in depth frame units to search for the optimum setting value, but the parameters are changed in microframe units or depth map units to be optimal. You may search for various setting values.

Further, in the example of the calibration process described with reference to FIGS. 11 and 12, a reflecting object is installed at a position fixed distance away from the ranging system 1, and the reflected light reflected by the reflecting object is received by the ranging sensor 12. It was an example of getting the result.

In the calibration process in which the light emission observation device 101 is arranged at a fixed distance from the distance measurement system 1 and the irradiation light emitted from the illumination device 11 is directly detected, the light emission timing control unit 32 issues a light receiving pulse to pixel. The operation of driving each pixel 71 of the array unit 35 and receiving light is omitted.

An example of the parameters adjusted in the calibration mode will be described with reference to FIGS. 13 and 14.

FIG. 13 shows an example of adjusting the emission intensity as a parameter to be adjusted.

As shown in FIG. 13, the distance measuring sensor 12 increments or decrements the light emission intensity when the lighting device 11 emits the irradiation light in a predetermined step width STEP, and searches for the optimum light emission intensity. be able to.

FIG. 14 shows an example of adjusting the delay time td from the input of the light emission pulse to the light emission and the slope of the light emission intensity at the start of light emission as the parameters to be adjusted.

As shown in FIG. 14, the lighting device 11 has a function of setting a delay time td from the input of the light emission pulse to the light emission. As described with reference to FIG. 2, the light emission timing of the lighting device 11 and the light reception timing of the distance measuring sensor 12 need to be the timings of phase 0 degree, phase 90 degree, phase 180 degree, and phase 270 degree. Therefore, the actual light emission timing can be adjusted by setting the delay time td for the light emission pulse.

Further, the lighting device 11 has a function of adjusting the inclination of the light emission intensity, such as whether the light emission intensity is gradually increased or the emission intensity is rapidly increased at the start of light emission. When the emission intensity is increased slowly, it takes time to reach the desired emission intensity, and conversely, when the emission intensity is increased steeply, hunting occurs or the initial emission intensity is large. Since it may be too much, it is possible to sequentially set a parameter for adjusting the inclination of the emission intensity to a plurality of values and search for the optimum value.

<6. Device configuration example in calibration mode>
FIG. 15 is a block diagram showing a detailed configuration example of the lighting device 11 and the distance measuring sensor 12 in the calibration mode. In the description of FIG. 15, the description of the portion common to the measurement mode described with reference to FIG. 4 will be omitted as appropriate. In FIG. 15, the host control unit 13 is also shown in the same manner as in FIG.

A setting parameter for adjustment is supplied from the host control unit 13 to the control unit 31 of the distance measurement sensor 12 via the input / output terminal 39-1, and a distance measurement start trigger is supplied via the input / output terminal 39-2. Is supplied. The control unit 31 controls the operation of the entire distance measuring sensor 12 and the lighting device 11 based on the setting parameter and the distance measuring start trigger.

More specifically, the control unit 31 has a parameter value based on a setting range consisting of a parameter type (parameter name) to be adjusted and an initial value and an end value of the parameter supplied from the host control unit 13. Is sequentially synchronized with the light receiving operation (exposure operation) of the pixel array unit 35 in a predetermined step width STEP from the initial value to the end value, and is supplied to the lighting device 11 via the input / output terminals 39-3.

The data processing unit 37 calculates the depth value d of each pixel 71 based on the detection signal of each pixel 71 after AD conversion supplied from the column processing unit 36, and stores the depth value d as the pixel value of each pixel. Generates the depth frame. Further, the data processing unit 37 calculates the reliability conf based on the detection signal of each pixel 71, and also generates a reliability frame corresponding to the depth frame in which the reliability conf is stored as the pixel value of each pixel. Depth maps and reliability maps are also generated when the parameters are changed on a depth map basis.

The output IF 38 converts the depth frame and the reliability frame supplied from the data processing unit 37 into the signal format of the input / output terminal 39-5 (for example, MIPI: Mobile Industry Processor Interface), and converts the depth frame and the reliability frame from the input / output terminal 39-5. Output. The depth frame and the reliability frame output from the input / output terminals 39-5 are supplied to the host control unit 13 as distance measurement data. When the depth map and the reliability map are generated, they are also supplied to the host control unit 13 as distance measurement data.

When the signal connection terminal between the lighting device 11 and the distance measuring sensor 12 is an SPI or I2C serial communication terminal, the parameter to be adjusted and the parameter setting value can be specified in a complicated and detailed manner by using a register. It can be set.

<Flowchart of measurement process in calibration mode>
Next, the measurement process by the distance measuring system 1 when the operation mode is the calibration mode will be described with reference to the flowchart of FIG. The measurement process of FIG. 16 will be described by taking as an example the case of executing the calibration process for searching for the optimum value of one type of parameter A described with reference to FIG. This process is started, for example, when the host control unit 13 supplies a setting parameter for adjustment.

First, in step S21, the control unit 31 of the distance measuring sensor 12 acquires the setting parameters for adjustment supplied from the host control unit 13 and stores them in the internal memory. In the example of FIG. 11, for example, the parameter A indicating the type of the parameter, the initial value V1 indicating the parameter setting range, and the end value V1end are supplied from the host control unit 13 as setting parameters for adjustment.

In step S22, the control unit 31 substitutes 1 as an initial value into the variable M that counts the number of changes in the step width STEP.

In step S23, the control unit 31 determines whether the distance measurement start trigger has been supplied from the host control unit 13, and waits until the distance measurement start trigger is supplied.

If it is determined in step S23 that the distance measurement start trigger has been supplied, the process proceeds to step S24, and the control unit 31 sets the initial value V1 as the set value of the parameter A to the lighting device via the input / output terminals 39-3. It supplies to 11 and sets the value of parameter A to the initial value V1.

In step S25, the control unit 31 outputs a light emitting pulse to the lighting device 11, causes the lighting device 11 to emit the irradiation light, and starts a light receiving operation by itself. More specifically, the control unit 31 causes the lighting device 11 to emit the irradiation light by generating and outputting the light emission pulse to the light emission timing control unit 32. Further, the control unit 31 starts the light receiving operation of the pixel array unit 35 by causing the light emitting timing control unit 32 to generate and output a light receiving pulse. The lighting device 11 sets the parameter A to the initial value V1 and emits the irradiation light. Although detailed description is omitted, the light source setting information other than the parameter A is also set in the same manner as in the measurement mode.

In step S26, each pixel 71 of the pixel array unit 35 receives the reflected light that is reflected by the object and returned from the irradiation light emitted from the illumination device 11 in accordance with the control of the pixel modulation unit 33 and the pixel control unit 34. Then, a detection signal corresponding to the amount of received light is supplied to the data processing unit 37 in pixel units.

In step S27, the data processing unit 37 calculates the depth frame and the reliability frame based on the detection signal of each pixel 71. The calculated depth frame and reliability frame are converted into the signal format of the input / output terminals 39-5 by the output IF 38, and then output as distance measurement data from the input / output terminals 39-5 to the host control unit 13.

In step S28, the control unit 31 determines whether the measurement is performed by setting the parameter A up to the end value V1end specified by the host control unit 13.

If it is determined in step S28 that the end value V1end of the parameter A has not been set and the measurement has not been performed, the process proceeds to step S29, and the control unit 31 sets the variable M for counting the number of changes in the step width STEP to 1. Increment only, set the value of parameter A to (V1 + STEP * M), and return the process to step S25. As a result, the above-mentioned steps S25 to S28 are executed again, the value of the parameter A that has not been measured is set in the lighting device 11, and the light receiving operation is performed.

On the other hand, if it is determined in step S28 that the parameter A is set up to the designated end value V1end and the measurement is performed, the measurement process in the calibration mode ends.

As described above, in the calibration mode, the distance measuring system 1 is used for adjustment including the parameter type and the setting range specified by the initial value and the ending value before the distance measuring start trigger is supplied. The setting parameters are acquired from the host control unit 13, the values of each parameter included in the setting range specified by the host control unit 13 are sequentially set in the lighting device 11, and the measurement based on the values of the specified parameters ( Light receiving operation) can be performed. When the host control unit 13 also supplies the step width STEP, the distance measuring system 1 performs measurement by sequentially changing the parameter values in the step width STEP specified by the host control unit 13.

In the measurement mode and the calibration mode, a plurality of types of setting values of light emission conditions are acquired at once from the host control unit 13, and a plurality of types of setting values are sequentially applied to the distance measurement start trigger from the host control unit 13. It is common in that it can be set in the lighting device 11 to sequentially acquire distance measurement data based on desired light emission conditions.

As described above, the distance measuring sensor 12 acquires a plurality of types of setting values set in the lighting device 11 and controls to change the plurality of types of setting values of the lighting device 11 according to its own operation. , The host control unit 13 does not need to control the lighting device 11. Then, the distance measuring sensor 12 can control the lighting device 11 at high speed by executing the control of changing the set value of the lighting device 11 at the timing according to its own operation. Since the calibration process can be executed in a short time, the time for pre-shipment inspection and acceptance inspection can be shortened, which can contribute to the reduction of the manufacturing cost of the entire device.

In the measurement process in the calibration mode of FIG. 16, a reflecting object such as a reflector having a known light reflectance is arranged at a position separated from the distance measuring system 1 by a fixed distance, and the result of receiving the reflected light by the distance measuring sensor 12 This is a process for searching (adjusting) the optimum value of a parameter based on.

When the light emission observation device 101 is arranged at a position separated from the distance measurement system 1 by a fixed distance and the optimum value of the parameter is searched (adjusted) based on the detection result of directly detecting the irradiation light by the light emission observation device 101, the step is performed. The process of starting the light receiving operation of the pixel array unit 35 in S25 and the process of steps S26 and S27 are omitted.

<7. Calibration process during measurement>
The distance measuring sensor 12 can execute the calibration process in the calibration mode described above even between the distance measurements in the measurement mode. In this case, as shown in FIG. 17, distance measurement data such as the depth frame and the reliability frame, the depth map and the reliability map calculated by the data processing unit 37 are also supplied to the control unit 31, and the control unit 31 also supplies the distance measurement data. , Determine whether it is necessary to execute the calibration process based on the distance measurement data.

FIG. 18 shows an example in which measurement is performed under three types of light emission conditions A, B, and C, which are the same as the measurement sequence in the measurement mode shown in FIG.

As shown in FIG. 18, the distance measurement sensor 12 generates and outputs predetermined distance measurement data based on the distance measurement start trigger, and then the next distance measurement start, that is, the next distance measurement start trigger is performed. The calibration process can be performed in the period until it is supplied. For example, the calibration process can be executed during a period in which some calculation is performed on the host control unit 13 side, a Wait Vsync period which is a signal indicating measurement standby, a Vblank period, and the like.

The calibration process during distance measurement will be described with reference to the flowchart of FIG. Since this process is executed in the measurement mode, it is started when the light emission condition and the distance measurement start trigger are supplied from the host control unit 13.

First, in step S41, the control unit 31 of the distance measuring sensor 12 sets a plurality of light emitting conditions in the lighting device 11, and performs distance measurement control for generating a depth map and a reliability map. This process is similar to the control of the entire measurement process in the measurement mode described with reference to FIG.

In step S42, the control unit 31 determines whether the distance measurement data has been generated. When the distance measurement data is generated, the generated distance measurement data is also supplied to the control unit 31, so that the control unit 31 can determine that the distance measurement data has been generated.

If it is determined in step S42 that the ranging data has not been generated, the process is returned to step S41, and the process of step S41 is continuously executed. Therefore, the distance measurement control in step S41 is continuously executed until the distance measurement data output corresponding to the distance measurement start trigger supplied from the host control unit 13 is generated.

If it is determined in step S42 that the ranging data has been generated, the process proceeds to step S43, and the control unit 31 determines whether it is necessary to confirm the parameters that control the lighting device 11. The control unit 31 is provided with, for example, a temperature sensor for detecting the internal temperature in the lighting device 11 or the distance measuring sensor 12, and when the temperature detected by the temperature sensor changes by a certain value or more, the parameter It can be determined that confirmation is required. Alternatively, it can be determined that the parameters need to be confirmed when the distance of the subject to be measured changes. Furthermore, it can be determined that it is necessary to confirm the parameters even when there is a change in noise due to ambient light or the like in the acquired ranging data.

If it is determined in step S43 that the parameter confirmation is not necessary, the process proceeds to step S44, and the control unit 31 determines whether the next distance measurement start trigger has been supplied from the host control unit 13, and then proceeds to the next step. Wait until it is determined that the distance measurement start trigger has been supplied. Then, when it is determined in step S44 that the next distance measurement start trigger has been supplied, the process is returned to step S41, and the distance measurement control for generating the depth map and the reliability map is executed.

On the other hand, if it is determined in step S43 that the parameter needs to be confirmed, the process proceeds to step S45, and the control unit 31 executes the calibration process. This process is basically the same as the measurement process in the calibration mode described in the flowchart of FIG. 16, but the host control unit 13 does not supply the setting parameters for adjustment from the host control unit 13. The difference is that the parameter type and setting range are determined based on the current setting value and executed. For example, when it is determined that the influence of noise caused by ambient light or the like is large, a set value for increasing the emission intensity from the current set value is adopted and set in the lighting device 11. On the contrary, when it is determined that the influence of noise caused by ambient light or the like is small, a set value for reducing the emission intensity from the current set value is adopted and set in the lighting device 11.

When the calibration process in step S45 is completed, the host control unit 13 proceeds to step S46, selects the optimum value of the parameter based on the measurement result acquired in the calibration process, and changes the setting.

After step S46, the process proceeds to step S44. Then, it waits until the next distance measurement start trigger is supplied, and when the next distance measurement start trigger is supplied, the process is returned to step S41, and the next distance measurement control that generates the depth map and the reliability map is generated. Is executed.

As described above, the distance measuring system 1 also executes the calibration process in the above-mentioned calibration mode between the distance measurements in the measurement mode (from the end of the distance measurement to the start of the next distance measurement). be able to. As a result, it is possible to respond to environmental changes such as temperature changes and external light changes during measurement, and it is possible to reduce deterioration of distance measurement accuracy.

The host control unit 13 may determine whether or not it is necessary to confirm the parameters of the lighting device 11 based on the acquired distance measurement data. In this case, if the light emitting condition supplied from the host control unit 13 is a parameter set value for calibration, the calibration process can be executed in the same manner as the measurement process in the measurement mode.

<8. Distance measurement sensor chip configuration example>
FIG. 20 is a perspective view showing a chip configuration example of the distance measuring sensor 12.

As shown in FIG. 20A, the distance measuring sensor 12 can be composed of one chip in which the first die (board) 141 and the second die (board) 142 are laminated.

For example, a pixel array unit 35 as a light receiving unit is arranged on the first die 141, and a depth frame or a depth frame or a depth frame is used on the second die 142 by using, for example, a detection signal output from the pixel array unit 35. A data processing unit 37 or the like that performs processing for generating a depth map or the like is arranged.

The distance measuring sensor 12 may be composed of three layers in which another logic die is laminated in addition to the first die 141 and the second die 142, or may be composed of four or more layers of dies (boards). It may be configured.

Further, some functions of the distance measuring sensor 12 may be performed by a signal processing chip different from the distance measuring sensor 12. For example, as shown in B of FIG. 20, the sensor chip 151 as the distance measuring sensor 12 and the logic chip 152 that performs signal processing in the subsequent stage can be formed on the relay board 153. The logic chip 152 may be configured to perform a part of the processing performed by the data processing unit 37 of the distance measuring sensor 12 described above, for example, a processing for generating a depth frame or a depth map.

<9. Comparison with other light emission control methods>
The distance measuring system 1 described above supplies the light emitting conditions of the lighting device 11 from the host control unit 13 to the distance measuring sensor 12, and the distance measuring sensor 12 controls the light emission of the lighting device 11 based on the acquired light emitting conditions. Was supposed to be.

On the other hand, as shown in FIG. 21, a method in which the host control unit 181 supplies the light emitting condition to the lighting device 183 and the light receiving condition corresponding to the light emitting condition is supplied to the ranging sensor 182 is also conceivable. The light receiving conditions corresponding to the light emitting conditions include, for example, a modulation frequency, an exposure period corresponding to the light emitting period, and the like. The distance measurement start trigger is supplied from the host control unit 181 to the distance measurement sensor 182 as in the distance measurement system 1 of FIG. 1, and the light emitting pulse is generated by the distance measurement sensor 182 and supplied to the lighting device 183. ..

In such a control method, since the host control unit 181 changes the light emitting condition, when trying to supply a new light emitting condition to the lighting device 183, it is necessary to temporarily stop the light emitting of the lighting device 183. Therefore, it is necessary to stop the output of the emission pulse, and therefore, it is also necessary to temporarily stop the operation of the distance measuring sensor 182. Since the light emitting pulse is generated and output in synchronization with the distance measurement start operation by receiving the distance measurement start trigger, when the output of the light emission pulse is stopped, the host control unit 181 outputs the distance measurement start trigger again. There is a need to.

22 shows a host control system in which the host control unit 181 supplies light emission conditions to the lighting device 183 as in the distance measurement system of FIG. 21, and a distance measurement sensor 12 as in the distance measurement system 1 of FIG. It is a figure which compared and showed the control sequence in the case of the distance measuring sensor control system which supplies light emission condition to a lighting apparatus 11.

As an example of changing the light emission condition in FIG. 22, the value of the parameter A described in FIG. 11 is changed from the initial value V1 to the end value V1end to acquire the distance measurement data.

In the host control method shown in the upper part of FIG. 22, as described above, when the value of the parameter A as the light emission condition is changed, it is necessary to stop the operation of the distance measuring sensor 182 in order to stop the light emission pulse. It is necessary to repeat a series of processes of setting (output) A, outputting the distance measurement start trigger, and driving / stopping the distance measurement sensor 182.

On the other hand, in the distance measuring sensor control method shown in the lower part of FIG. 22, the distance measuring sensor 12 controls both the setting of the parameter A and the output start and output stop of the emission pulse, and receives light from the pixel array unit 35. Since the operation (exposure operation) is also its own control, it is possible to perform control for repeatedly executing the setting of the parameter A and the output start and output stop of the emission pulse according to the own operation.

Therefore, according to the distance measurement sensor control method of the distance measurement system 1, parameters can be set (changed) at high speed, and calibration processing and normal distance measurement processing can be executed in a short time. By shortening the time required for the calibration process performed in the pre-shipment inspection, it is possible to contribute to the reduction of the manufacturing cost of the entire device.

Further, since the host control unit 13 is not involved in the control of switching the light emission condition, the host control unit 13 itself can be put into a stop state (standby state) after the distance measurement start trigger output, so that the distance measurement system 1 can be used. It can also contribute to the low power consumption of the entire embedded host device.

The control method for changing the light emission condition of the distance measuring system 1 described above is not limited to the indirect ToF type ranging system, but can also be applied to the Structured Light method and the direct ToF distance measuring system.

<10. Application example to electronic devices>
The distance measuring system 1 described above can be mounted on electronic devices such as smartphones, tablet terminals, mobile phones, personal computers, game machines, television receivers, wearable terminals, digital still cameras, and digital video cameras.

FIG. 23 is a block diagram showing a configuration example of a smartphone as an electronic device equipped with the distance measuring system 1.

As shown in FIG. 23, in the smartphone 201, the distance measuring module 202, the image pickup device 203, the display 204, the speaker 205, the microphone 206, the communication module 207, the sensor unit 208, the touch panel 209, and the control unit 210 are connected via the bus 211. Is connected and configured. Further, the control unit 210 has functions as an application processing unit 221 and an operating system processing unit 222 by executing a program by the CPU.

The distance measuring system 1 of FIG. 1 is applied to the distance measuring module 202. For example, the distance measuring module 202 is arranged in front of the smartphone 201, and by performing distance measurement for the user of the smartphone 201, the depth value of the surface shape of the user's face, hand, finger, etc. is measured as a distance measurement result. Can be output as. The host control unit 13 of FIG. 1 corresponds to the control unit 210 of FIG.

The image pickup device 203 is arranged in front of the smartphone 201, and takes an image of the user of the smartphone 201 as a subject to acquire an image of the user. Although not shown, the image pickup device 203 may be arranged on the back surface of the smartphone 201.

The display 204 displays an operation screen for performing processing by the application processing unit 221 and the operation system processing unit 222, an image captured by the image pickup apparatus 203, and the like. The speaker 205 and the microphone 206, for example, output the voice of the other party and collect the voice of the user when making a call by the smartphone 201.

The communication module 207 communicates via the communication network. The sensor unit 208 senses speed, acceleration, proximity, etc., and the touch panel 209 acquires a touch operation by the user on the operation screen displayed on the display 204.

The application processing unit 221 performs processing for providing various services by the smartphone 201. For example, the application processing unit 221 can create a face by computer graphics that virtually reproduces the user's facial expression based on the depth map supplied from the distance measuring module 202, and can perform a process of displaying the face on the display 204. .. Further, the application processing unit 221 can perform a process of creating, for example, three-dimensional shape data of an arbitrary three-dimensional object based on the depth map supplied from the distance measuring module 202.

The operation system processing unit 222 performs processing for realizing the basic functions and operations of the smartphone 201. For example, the operation system processing unit 222 can perform a process of authenticating the user's face and unlocking the smartphone 201 based on the depth map supplied from the distance measuring module 202. Further, the operation system processing unit 222 performs, for example, a process of recognizing a user's gesture based on the depth map supplied from the distance measuring module 202, and performs a process of inputting various operations according to the gesture. Can be done.

In the smartphone 201 configured in this way, by applying the distance measuring system 1 described above, for example, a depth map can be generated with high accuracy and high speed. As a result, the smartphone 201 can detect the distance measurement information more accurately.

<11. Application example to mobile>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on a moving body of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 24 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technique according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 24, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating a braking force of a vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives inputs of these radio waves or signals and controls a vehicle door lock device, a power window device, a lamp, and the like.

The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image pickup unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or a character on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects information in the vehicle. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether or not the driver has fallen asleep.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (Advanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040. It is possible to perform coordinated control for the purpose of automatic driving, etc., which runs autonomously without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio-image output unit 12052 transmits an output signal of at least one of audio and an image to an output device capable of visually or audibly notifying the passenger of the vehicle or the outside of the vehicle. In the example of FIG. 17, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 25 is a diagram showing an example of the installation position of the imaging unit 12031.

In FIG. 25, the vehicle 12100 has imaging units 12101, 12102, 12103, 12104, 12105 as imaging units 12031.

The imaging units 12101, 12102, 12103, 12104, 12105 are provided at positions such as, for example, the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The imaging units 12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The images in front acquired by the imaging units 12101 and 12105 are mainly used for detecting a preceding vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 25 shows an example of the photographing range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the imaging units 12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform coordinated control for the purpose of automatic driving or the like in which the vehicle travels autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that can be seen by the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 is used via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine.
When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

The example of the vehicle control system to which the technique according to the present disclosure can be applied has been described above. The technique according to the present disclosure can be applied to the vehicle exterior information detection unit 12030 and the vehicle interior information detection unit 12040 among the configurations described above. Specifically, by using the distance measurement by the distance measurement system 1 as the outside information detection unit 12030 and the inside information detection unit 12040, processing for recognizing the driver's gesture is performed, and various types according to the gesture (for example, It can perform operations on audio systems, navigation systems, air conditioning systems) and detect the driver's condition more accurately. Further, the distance measurement by the distance measurement system 1 can be used to recognize the unevenness of the road surface and reflect it in the control of the suspension.

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

The present techniques described above and below in the present specification can be independently implemented independently as long as there is no contradiction. Of course, any plurality of the present technologies can be used in combination. It is also possible to carry out a part or all of any of the above-mentioned techniques in combination with other techniques not described above.

Further, for example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the contrary, the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Further, of course, a configuration other than the above may be added to the configuration of each device (or each processing unit). Further, if the configuration and operation of the entire system are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..

Further, in the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

It should be noted that the effects described in the present specification are merely examples and are not limited, and effects other than those described in the present specification may be obtained.

The present technology can have the following configurations.
(1)
A pixel array unit in which pixels that receive the reflected light that is reflected by an object and output a detection signal according to the amount of light received are two-dimensionally arranged.
A distance measuring sensor including a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device.
(2)
The control unit acquires the initial value and the end value of the parameter from the other device, changes the value of the parameter by a predetermined step width from the initial value of the parameter, and sets the value in the lighting device. The ranging sensor according to 1).
(3)
The distance measuring sensor according to (2) above, wherein the predetermined step width is determined in advance.
(4)
The distance measuring sensor according to (2) or (3), wherein the predetermined step width is supplied from the other device.
(5)
The control unit sequentially sets a plurality of values of the parameters in the lighting device for two types of parameters, the first parameter and the second parameter, and controls the light emission of the lighting device (1) to (1). The distance measuring sensor according to any one of 4).
(6)
The distance measuring sensor according to any one of (1) to (5), wherein the control unit also performs a light receiving operation of the pixel array unit for each value of the parameter.
(7)
The distance measuring sensor according to (6), wherein the control unit outputs distance measurement data generated by a light receiving operation of the pixel array unit to the other device for each value of the parameter.
(8)
The distance measuring sensor according to any one of (1) to (7) above, wherein the parameter is a parameter corresponding to the emission intensity of the irradiation light.
(9)
The distance measuring sensor according to any one of (1) to (8) above, wherein the parameter is a parameter corresponding to a delay time from the input of a light emitting pulse to the lighting device to the light emission.
(10)
The distance measuring sensor according to any one of (1) to (9) above, wherein the parameter is a parameter corresponding to an inclination of the emission intensity of the irradiation light.
(11)
The control unit determines whether or not the parameter needs to be confirmed, and if it is determined that the parameter needs to be confirmed, a plurality of values of the parameter are sequentially set in the lighting device (1) to (10). The ranging sensor according to any one of.
(12)
The distance measuring sensor according to (11), wherein the control unit determines that it is necessary to confirm the parameters when the temperature of the lighting device or the distance measuring sensor changes by a certain value or more.
(13)
The distance measuring sensor according to (11) or (12) above, wherein the control unit determines that it is necessary to confirm the parameters when the distance of the subject to be measured changes.
(14)
The distance measuring sensor according to any one of (11) to (13) above, wherein the control unit determines that it is necessary to confirm the parameters when there is a change in noise in the measurement data.
(15)
The control unit according to any one of (11) to (14), wherein a plurality of values of the parameters are sequentially set in the lighting device from the end of the distance measurement to the start of the next distance measurement. Distance measurement sensor.
(16)
A lighting device that irradiates an object with irradiation light,
It is provided with a distance measuring sensor that receives the reflected light that is reflected by the object and returned.
The distance measuring sensor is
A pixel array unit in which pixels that receive the reflected light and output a detection signal according to the amount of light received are two-dimensionally arranged.
A distance measuring system including a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device.
(17)
A lighting device that irradiates an object with irradiation light,
It is provided with a distance measuring sensor that receives the reflected light that is reflected by the object and returned.
The distance measuring sensor is
A pixel array unit in which pixels that receive the reflected light and output a detection signal according to the amount of light received are two-dimensionally arranged.
An electronic device including a distance measuring system including a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device.

1 Distance measurement system, 11 Lighting device, 12 Distance measurement sensor, 13 Host control unit, 31 Control unit, 32 Emission timing control unit, 33 Pixel modulation unit, 35 Pixel array unit, 37 Data processing unit, 51 Emission control unit, 52 Light emission source, 71 pixels, 101 light emission observation device, 201 smartphone, 202 distance measurement module

Claims (17)

A pixel array unit in which pixels that receive the reflected light that is reflected by an object and output a detection signal according to the amount of light received are two-dimensionally arranged.
A distance measuring sensor including a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device. The control unit acquires the initial value and the end value of the parameter from the other device, changes the value of the parameter by a predetermined step width from the initial value of the parameter, and sets the value in the lighting device. The ranging sensor according to 1. The distance measuring sensor according to claim 2, wherein the predetermined step width is determined in advance. The distance measuring sensor according to claim 2, wherein the predetermined step width is supplied from the other device. The first aspect of claim 1, wherein the control unit sequentially sets a plurality of values of the parameters in the lighting device for two types of parameters, the first parameter and the second parameter, and controls the light emission of the lighting device. Distance measurement sensor. The distance measuring sensor according to claim 1, wherein the control unit also performs a light receiving operation of the pixel array unit for each value of the parameter. The distance measuring sensor according to claim 6, wherein the control unit outputs distance measurement data generated by a light receiving operation of the pixel array unit to the other device for each value of the parameter. The distance measuring sensor according to claim 1, wherein the parameter is a parameter corresponding to the emission intensity of the irradiation light. The distance measuring sensor according to claim 1, wherein the parameter corresponds to a delay time from the input of a light emitting pulse to the lighting device to the time when the light is emitted. The distance measuring sensor according to claim 1, wherein the parameter is a parameter corresponding to an inclination of the emission intensity of the irradiation light. The distance measuring unit according to claim 1, wherein the control unit determines whether or not the parameter needs to be confirmed, and if it is determined that the parameter needs to be confirmed, a plurality of values of the parameter are sequentially set in the lighting device. Sensor. The distance measuring sensor according to claim 11, wherein the control unit determines that it is necessary to confirm the parameters when the temperature of the lighting device or the distance measuring sensor changes by a certain value or more. The distance measuring sensor according to claim 11, wherein the control unit determines that it is necessary to confirm the parameters when the distance of the subject to be measured changes. The distance measuring sensor according to claim 11, wherein the control unit determines that it is necessary to confirm the parameters when there is a change in noise in the measurement data. The distance measuring sensor according to claim 11, wherein the control unit sequentially sets a plurality of values of the parameters in the lighting device from the end of the distance measurement to the start of the next distance measurement. A lighting device that irradiates an object with irradiation light,
It is provided with a distance measuring sensor that receives the reflected light that is reflected by the object and returned.
The distance measuring sensor is
A pixel array unit in which pixels that receive the reflected light and output a detection signal according to the amount of light received are two-dimensionally arranged.
A distance measuring system including a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device. A lighting device that irradiates an object with irradiation light,
It is provided with a distance measuring sensor that receives the reflected light that is reflected by the object and returned.
The distance measuring sensor is
A pixel array unit in which pixels that receive the reflected light and output a detection signal according to the amount of light received are two-dimensionally arranged.
An electronic device including a distance measuring system including a control unit that sequentially sets a plurality of values of parameters in the lighting device based on a measurement start instruction from another device and controls light emission of the lighting device.

Images