WO2021059699A1

WO2021059699A1 - Distance measurement device, distance measurement device control method, and electronic device

Info

Publication number: WO2021059699A1
Application number: PCT/JP2020/027983
Authority: WO
Inventors: 一輝大橋
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2019-09-25
Filing date: 2020-07-20
Publication date: 2021-04-01
Also published as: US20220373682A1; JP2021050988A

Abstract

This distance measurement device is provided with an optical detection unit which receives light from a subject, a depth calculation unit which calculates depth information of the subject on the basis of output of the optical detection unit, and an artifact removal unit which divides an image into segments on the basis of the depth information, activates segments in which the number of pixels in the segment exceeds a prescribed threshold value, and deactivates segments in which the number of pixels is less than or equal to the threshold value.

Description

Distance measuring device, control method of range measuring device, and electronic device

The present disclosure relates to a distance measuring device, a control method of the distance measuring device, and an electronic device.

As a distance measuring device (distance measuring device) that acquires distance information (distance image information) to the subject, there is a device (sensor) that uses the ToF (Time of Flight) method. The ToF method irradiates a subject (measurement object) with light from a light source, and detects the flight time of the light until the irradiated light is reflected by the subject and returned to the light detection unit to reach the subject. This is a method for measuring distance.

As one of the ToF methods, pulsed light of a predetermined cycle emitted from a light source is reflected by a subject, and the cycle when the reflected light is received by the light detection unit is detected, and the position of the light emission cycle and the light reception cycle. There is an indirect ToF method that measures the distance to the subject by measuring the light flight time from the phase difference (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-035398

In the indirect ToF type distance measuring device, the maximum distance that can be measured is determined according to the emission frequency (emission cycle) of the laser light emitted from the light source, but if the maximum distance is exceeded, aliasing distortion called aliasing is performed. (Aliased distance) will occur. Then, for an object at a distance where aliasing occurs (alias distance), the actual measurement distance becomes closer than the original correct distance (true distance). As a result, the distance measurement result of the object at the alias distance is output as an erroneous distance measurement result, and in the subsequent system that performs various controls based on the distance measurement result, the erroneous control is performed. ..

The present disclosure is a distance measuring device and a distance measuring device capable of invalidating the distance measuring result of an object at an alias distance and outputting only the distance measuring result of an object whose correct distance (true distance) is measured. It is an object of the present invention to provide a control method of a device and an electronic device having the distance measuring device.

The ranging device of the present disclosure for achieving the above object is
Photodetector that receives light from the subject,
A depth calculation unit that calculates the depth information of the subject based on the output of the light detection unit, and
An artifact removal unit that divides an image into segments based on the depth information calculated by the depth calculation unit, enables segments in which the number of pixels in each segment exceeds a predetermined threshold value, and invalidates segments below a predetermined threshold value.
To be equipped.

The control method of the ranging device of the present disclosure for achieving the above object is described.
A photodetector that receives light from the subject, and
Depth calculation unit, which calculates the depth information of the subject based on the output of the light detection unit,
In a distance measuring device equipped with
The image is divided into segments based on the depth information, the segments in which the number of pixels of each segment exceeds a predetermined threshold value are valid, and the segments below the predetermined threshold value are invalidated.

Further, the electronic device of the present disclosure for achieving the above object has a distance measuring device having the above configuration.

FIG. 1 is a conceptual diagram of a ToF type ranging system. FIG. 2 is a block diagram showing an example of the configuration of a ToF type distance measuring device to which the technique of the present disclosure is applied. FIG. 3 is a block diagram showing an example of the configuration of the photodetector in the distance measuring device. FIG. 4 is a circuit diagram showing an example of a pixel circuit configuration in the photodetector. FIG. 5 is a timing waveform diagram for explaining the calculation of the distance in the ToF type distance measuring device. FIG. 6 is a block diagram showing an example of the configuration of the distance image calculation unit of the distance measuring unit in the distance measuring device. FIG. 7 is a diagram showing the relationship between the true depth and the distance measurement depth when the laser drive frequency is 100 MHz. FIG. 8A is a diagram showing the size of the subject reflected in the image when the true depth is 1.0 m and the distance measurement depth is 1.0 m, and FIG. 8B is a diagram showing the true depth of 2.5 m and the distance measurement depth of 1.0 m. It is a figure which shows the size of the subject which appears in the image in the case of. FIG. 9 is a flowchart showing an example of a process for solving an aliasing problem executed by the distance measuring device according to the embodiment of the present disclosure. FIG. 10 is a block diagram showing a modified example of the embodiment of the present disclosure. FIG. 11A is an external view of the smartphone according to the specific example 1 of the electronic device of the present disclosure as viewed from the front side, and FIG. 11B is an external view as viewed from the back side. FIG. 12A is an external view of the digital still camera according to the second embodiment of the electronic device of the present disclosure as viewed from the front side, and FIG. 12B is an external view as viewed from the back side.

Hereinafter, embodiments for carrying out the technique of the present disclosure (hereinafter, referred to as “embodiments”) will be described in detail with reference to the drawings. The technique of the present disclosure is not limited to the embodiment, and various numerical values and the like in the embodiment are examples. In the following description, the same reference numerals will be used for the same elements or elements having the same function, and duplicate description will be omitted. The explanation will be given in the following order.
1. 1. Description of the ranging device and electronic device of the present disclosure, and general description 2. ToF type ranging system 3. Distance measuring device to which the technology of the present disclosure is applied 3-1. System configuration 3-2. Configuration example of photodetector 3-3. Pixel circuit configuration example 3-4. Configuration example of the distance image calculation unit 3-5. Aliasing issues 4. Embodiments of the present disclosure 4-1. Processing example for solving the aliasing problem 4-2. Threshold setting for the number of pixels in a segment 5. Modification example 6. Application example 7. Electronic device of the present disclosure 7-1. Specific example 1 (example of smartphone)
7-2. Specific example 2 (example of digital still camera)
8. Configuration that can be taken by this disclosure

<Explanation of the ranging device and electronic device of the present disclosure, in general>
In the distance measuring device and the electronic device of the present disclosure, each segment of the artifact removing unit can be labeled, and a component object can be created for each label. Further, the artifact removing unit can be configured to be labeled differently for each segment by the process of labeling the nearby pixels and the pixels having continuous depth information with the same label. Further, the component object can be configured to have the number of pixels of the segment and the average depth.

In the distance measuring device and the electronic device of the present disclosure including the above-mentioned preferable configuration, the artifact removing unit may be configured to change a predetermined threshold value according to the distance of the segment from the distance measuring device. Further, the artifact removing unit can be configured such that the distance of the segment from the distance measuring device sets a large threshold value for a relatively short short distance and a small threshold value for a relatively long long distance. ..

<ToF distance measurement system>
FIG. 1 is a conceptual diagram of a ToF type ranging system. In the distance measuring device 1 according to this example, the ToF method is adopted as a measuring method for measuring the distance to the subject 10 which is a measurement target. The ToF method is a method of measuring the time until the light emitted toward the subject 10 is reflected by the subject 10 and returned. In order to realize the distance measurement by the ToF method, the distance measuring device 1 emits light emitted toward the subject 10 (for example, a laser beam having a peak wavelength in the infrared wavelength region), and a light source 20 and the subject. The light detection unit 30 for detecting the reflected light reflected by 10 and returned is provided.

<Distance measuring device to which the technology of the present disclosure is applied>
[System configuration]
FIG. 2 is a block diagram showing an example of the configuration of a ToF type distance measuring device to which the technique of the present disclosure is applied. The ranging device 1 according to the present application example (that is, the ranging device 1 of the present disclosure) performs exposure control based on a signal value output by the light detection unit 30 in addition to the light source 20 and the light detection unit 30. (Automatic Exposure) A control unit 40 and a distance measuring unit 50 are provided. Then, the ToF type distance measuring device 1 according to this application example detects the distance information for each pixel of the photodetector 30, and acquires a highly accurate distance image (Deptth Map: depth map) in units of imaging frames. be able to.

The distance measuring device 1 according to this application example is an indirect ToF type distance measuring device (so-called indirect ToF type distance image sensor). The indirect ToF type ranging device 1 detects the cycle when the pulsed light of a predetermined cycle emitted from the light source 20 is reflected by the measurement object (subject) and the reflected light is received by the photodetector 30. Then, the distance to the object to be measured is measured by measuring the light flight time from the phase difference between the period of light emission and the period of light reception.

The light source 20 irradiates light toward the object to be measured by repeating the on / off operation at a predetermined cycle under the control of the AE control unit 40. As the irradiation light of the light source 20, for example, near-infrared light in the vicinity of 850 nm is often used. The light detection unit 30 receives the light emitted from the light source 20 after being reflected by the object to be measured and returns, and detects the distance information for each pixel. The light detection unit 30 outputs the RAW image data of the current frame including the distance information detected for each pixel and the light emission / exposure setting information, and supplies the RAW image data to the AE control unit 40 and the distance measuring unit 50.

The AE control unit 40 has a configuration including a next frame light emission / exposure condition calculation unit 41 and a next frame light emission / exposure control unit 42. The next frame emission / exposure condition calculation unit 41 calculates the emission / exposure condition of the next frame based on the RAW image data of the current frame supplied from the light detection unit 30 and the emission / exposure setting information. The light emission / exposure conditions of the next frame are the light emission time and light emission intensity of the light source 20 when acquiring the distance image of the next frame, and the exposure time of the photodetector 30. The next frame light emission / exposure control unit 42 detects the light emission time, light emission intensity, and light detection of the light source 20 of the next frame based on the light emission / exposure conditions of the next frame calculated by the next frame light emission / exposure condition calculation unit 41. The exposure time of the part 30 is controlled.

The distance measuring unit 50 has a configuration including a distance image calculation unit 51 that calculates a distance image. The distance image calculation unit 51 calculates a distance image by performing a calculation using the RAW image data of the current frame including the distance information detected for each pixel of the light detection unit 30, and obtains the depth, which is the depth information of the subject, and the depth, which is the depth information of the subject. , The distance image information including each information of the reliability value which is the light receiving information of the light detection unit 30 is output to the outside of the distance measuring device 1. Here, the distance image is, for example, an image in which a distance value (depth / depth value) based on the distance information detected for each pixel is reflected in each pixel.

[Configuration example of photodetector]
Here, a specific configuration example of the photodetector 30 will be described with reference to FIG. FIG. 3 is a block diagram showing an example of the configuration of the photodetector 30.

The photodetector 30 has a laminated structure including a sensor chip 31 and a circuit chip 32 laminated to the sensor chip 31. In this laminated structure, the sensor chip 31 and the circuit chip 32 are electrically connected through a connecting portion (not shown) such as a via (VIA) or a Cu—Cu junction. Note that FIG. 3 illustrates a state in which the wiring of the sensor chip 31 and the wiring of the circuit chip 32 are electrically connected via the above-mentioned connection portion.

A pixel array unit 33 is formed on the sensor chip 31. The pixel array unit 33 includes a plurality of pixels 34 arranged in a matrix (array shape) in a two-dimensional grid pattern on the sensor chip 31. In the pixel array unit 33, each of the plurality of pixels 34 receives incident light (for example, near-infrared light), performs photoelectric conversion, and outputs an analog pixel signal. _{Two vertical signal lines VSL 1} and VSL ₂ are wired in the pixel array unit 33 for each pixel row. Assuming that the number of pixel rows of the pixel array unit 33 is M (M is an integer), a total of (2 × M) vertical signal lines VSL are wired to the pixel array unit 33.

Each of the plurality of pixels 34 has a first tap A and a second tap B (details thereof will be described later). Of the two vertical signal lines VSL ₁ and VSL ₂ , the vertical signal line VSL ₁ outputs _{an analog pixel signal AIN P1} based on the charge of the first tap A of the pixel 34 of the corresponding pixel sequence. _{Further, an analog pixel signal AIN P2} based on the charge of the second tap B of the pixel 34 of the corresponding pixel sequence is output to the vertical signal line VSL _2. The analog pixel signals AIN _P1 and AIN _P2 will be described later.

A row selection unit 35, a column signal processing unit 36, an output circuit unit 37, and a timing control unit 38 are arranged on the circuit chip 32. The row selection unit 35 drives each pixel 34 of the pixel array unit 33 in units of pixel rows, and outputs _{pixel signals AIN P1} and AIN _P2. Under the drive of the row selection unit 35, the analog pixel signals AIN _P1 and AIN _P2 output from the pixel 34 of the selected row are supplied to the column signal processing unit 36 through the two vertical signal lines VSL ₁ and VSL _2. To.

The column signal processing unit 36 has a configuration having, for example, a plurality of analog-to-digital converters (ADCs) 39 provided for each pixel array corresponding to the pixel array of the pixel array unit 33. The analog-to-digital converter 39 performs analog-to-digital conversion processing on the analog pixel signals AIN _P1 and AIN _P2 _{supplied through the vertical signal lines VSL 1} and VSL _{2, and outputs them to the output circuit unit 37.} The output circuit unit 37 executes CDS (Correlated Double Sampling) processing or the like _{on the digitized pixel signals AIN P1} and AIN _P2 output from the column signal processing unit 36, and outside the circuit chip 32. Output to.

The timing control unit 38 generates various timing signals, clock signals, control signals, etc., and drives the row selection unit 35, the column signal processing unit 36, the output circuit unit 37, etc. based on these signals. Take control.

[Pixel circuit configuration example]
FIG. 4 is a circuit diagram showing an example of the circuit configuration of the pixel 34 in the photodetector 30.

The pixel 34 according to this example has, for example, a photodiode 341 as a photoelectric conversion unit. In addition to the photodiode 341, the pixel 34 includes overflow transistors 342, two

transfer transistors

343, 344, two

reset transistors

345, 346, two floating diffusion layers 347, 348, two

amplification transistors

349, 350, and two. It has a configuration having one

selection transistor

351 and 352. The two floating diffusion layers 347 and 348 correspond to the first and second taps A and B (hereinafter, may be simply referred to as "tap A and B") shown in FIG. 3 described above.

The photodiode 341 photoelectrically converts the received light to generate an electric charge. The photodiode 341 may have, for example, a back-illuminated pixel structure that captures light emitted from the back surface side of the substrate. However, the pixel structure is not limited to the back-illuminated pixel structure, and a surface-irradiated pixel structure that captures the light emitted from the surface side of the substrate can also be used.

The overflow transistor 342 is connected between the cathode electrode of the photodiode 341 and the power _{supply line of the power supply voltage V DD} , and has a function of resetting the photodiode 341. Specifically, the overflow transistor 342 becomes conductive in response to the overflow gate signal OFG supplied from the row selection unit 35, so that the electric charge of the photodiode 341 is sequentially discharged to the power supply line of the power _{supply voltage V DD.} To do.

The two

transfer transistors

343 and 344 are connected between the cathode electrode of the photodiode 341 and each of the two floating diffusion layers 347 and 348 (tap A and B). Then, the

transfer transistors

343 and 344 are brought into a conductive state in response to the transfer signal TRG supplied from the row selection unit 35, so that the charges generated by the photodiode 341 are sequentially transferred to the floating diffusion layers 347 and 348, respectively. Transfer to.

The floating diffusion layers 347 and 348 corresponding to the first and second taps A and B accumulate the electric charge transferred from the photodiode 341 and convert it into a voltage signal having a voltage value corresponding to the amount of the electric charge, and convert it into an analog voltage signal. Generates pixel signals AIN _P1 and AIN _P2.

The two

reset transistors

345 and 346 are connected between each of the two floating diffusion layers 347 and 348 and the power supply line of the power _{supply voltage V DD.} Then, the

reset transistors

345 and 346 are brought into a conductive state in response to the reset signal RST supplied from the row selection unit 35, so that charges are extracted from each of the floating diffusion layers 347 and 348, and the amount of charges is initialized. To do.

The two

amplification transistors

349 and 350 are _{connected between the power supply line of the power supply voltage V DD} and each of the two

selection transistors

351 and 352, and are converted from electric charge to voltage by the floating diffusion layers 347 and 348, respectively. Each voltage signal is amplified.

The two

selection transistors

351 and 352 are connected between the two

amplification transistors

349 and 350 and the vertical signal lines VSL ₁ and VSL _{2, respectively.} Then, the

selection transistors

351 and 352 enter a conductive state in response to the selection signal SEL supplied from the row selection unit 35, so that the voltage signals amplified by the

amplification transistors

349 and 350 are converted into analog pixel signals. Output to _two vertical signal lines VSL ₁ and VSL 2 as AIN _P1 and AIN _P2.

The two vertical signal lines VSL ₁ and VSL ₂ are connected to the input end of one analog-to-digital converter 39 in the column signal processing unit 36 for each pixel row, and are output from the pixel 34 for each pixel row. The analog pixel signals AIN _P1 and AIN _P2 are transmitted to the analog-to-digital converter 39.

The circuit configuration of the pixel 34 is not limited to the circuit configuration illustrated in FIG. 3 as long as it can generate _{analog pixel signals AIN P1} and AIN _{P2 by photoelectric conversion.}

Here, the calculation of the distance by the ToF method will be described with reference to FIG. FIG. 5 is a timing waveform diagram for explaining the calculation of the distance in the ToF type distance measuring device 1. The light source 20 and the light detection unit 30 in the ToF type distance measuring device 1 operate at the timing shown in the timing waveform diagram of FIG.

Light source 20, under control by the AE control unit 40 for a predetermined period, for example, only during the period of the pulse emission time T _p, for irradiating light to the object of measurement. The irradiation light emitted from the light source 20 is reflected by the object to be measured and returned. This reflected light (active light) is received by the photodiode 341. The time for the photodiode 341 to receive the reflected light after the irradiation of the irradiation light to the measurement object is started, that is, the light flight time is the time corresponding to the distance from the distance measuring device 1 to the measurement object. ..

4, photodiode 341, from the time the irradiation is started in the irradiation light, only during the period of the pulse emission time T _p, for receiving reflected light from the object to be measured. At this time, as the light received by the photodiode 341, the light radiated to the object to be measured is reflected by an object, the atmosphere, or the like in addition to the reflected light (active light) that is reflected by the object to be measured and returned. -Scattered ambient light is also included.

At the time of one light reception, the charge photoelectrically converted by the photodiode 341 is transferred to the tap A (floating diffusion layer 347) and accumulated. _{Then, a signal n 0} of a voltage value corresponding to the amount of electric charge accumulated in the floating diffusion layer 347 is acquired from the tap A. When the accumulation timing of the tap A is completed, the charge photoelectrically converted by the photodiode 341 is transferred to the tap B (suspended diffusion layer 348) and accumulated. _{Then, a signal n 1} having a voltage value corresponding to the amount of electric charge accumulated in the floating diffusion layer 348 is acquired from the tap B.

In this way, the tap A and the tap B are driven by 180 degrees out of phase of the accumulation timing (driving with the phases completely reversed), so that the signal n ₀ and the signal n ₁ are acquired, respectively. To. Then, such driving is repeated a plurality of times, and the signal n ₀ and the signal n ₁ are accumulated and integrated to _{acquire the accumulated signal N 0} and the accumulated signal N ₁ , respectively.

For example, for one pixel 34, light reception is performed twice in one phase, and signals of 0 degree, 90 degree, 180 degree, and 270 degree are accumulated in tap A and tap B four times each. Based on the accumulated signal N ₀ and the accumulated signal N ₁ acquired in this way, the distance D from the distance measuring device 1 to the object to be measured is calculated.

In addition to the components of the reflected light (active light) that is reflected by the object to be measured and returned, the stored signal N ₀ and the stored signal N _{1 include ambient light that is reflected and scattered by an object or the atmosphere.} Ingredients are also included. Therefore, in the above-mentioned operation, in order to remove the influence of the ambient light component and leave the reflected light (active light) component, the signal n ₂ based on the ambient light is also accumulated and integrated, and the environment. The accumulated signal N ₂ for the optical component is acquired.

_{Using the stored signal N 0} and the stored signal N ₁ including the ambient light component and the stored signal N ₂ for the ambient light component obtained in this way, the following equations (1) and (2) are used. The distance D from the distance measuring device 1 to the object to be measured can be calculated by the arithmetic processing based on the above.

In the formulas (1) and (2), D represents the distance from the distance measuring device 1 to the object to be measured, c represents the speed of light, and T _p represents the pulse emission time.

The distance image calculation unit 51 shown in FIG. 2 is output from the photodetection unit 30 by using the _{storage signal N 0} and the storage signal N ₁ including the ambient light component and the storage signal N _{2 for the ambient light component.} The distance D from the distance measuring device 1 to the object to be measured is calculated by the arithmetic processing based on the above equations (1) and (2), and is output as the distance image information. As the distance image information, for example, image information colored with a color having a density corresponding to the distance D can be exemplified. Here, the calculated distance D is output as the distance image information, but the calculated distance D may be output as the distance information as it is.

[Structure example of distance image calculation unit]
FIG. 6 shows an example of the configuration of the distance image calculation unit 51 of the distance measuring unit 50 in the distance measuring device 1. The distance image calculation unit 51 according to this example has a configuration including a depth calculation unit 511, a calibration unit 512, a noise reduction unit 513, and an artifact removal unit 514.

In the distance image calculation unit 51 having the above configuration, the depth calculation unit 511 uses the RAW image data given by the light detection unit 30 to reflect the light emitted from the light source 20 on the subject and detect the reflected light. Depth information and reliability value information are calculated from the phase difference reached by the unit 30. Here, the "depth" is the depth information (distance information) of the subject, the "reliability value" is the light receiving information of the photodetector 30, and the light emitted from the light source 20 is the subject. It is the amount (degree) of reflected light that is reflected and returned to the photodetector 30.

The calibration unit 512, for example, matches the phase of the light emitted from the light source 20 with the light incident on the photodetector 30, and performs calibration processing such as waveform correction and temperature correction. The noise reduction unit 513 is composed of, for example, a low-pass filter, and performs a process of reducing noise. The artifact removing unit 514 has various filter functions, and for each information of the depth and the reliability value passed through the noise reducing unit 513, the distance measurement is incorrect or the light receiving reliability value of the photodetector unit 30 is low. Perform the process of excluding things.

Here, the configurations in which the calibration unit 512, the noise reduction unit 513, and the artifact removal unit 514 are arranged in that order are illustrated, but the order of arrangement is arbitrary, that is, changed. You may.

[About aliasing issues]
By the way, in the above-mentioned indirect ToF type distance measuring device, the maximum distance that can be measured is determined according to the emission frequency (emission cycle) of the laser light emitted from the light source 20. As an example, FIG. 7 shows the relationship between the true depth and the distance measurement depth when the laser drive frequency (emission frequency) of the light source 20 is 100 MHz. In the case of this example, the maximum distance that can be measured is 1.5 m.

In the indirect ToF type distance measuring device, the distance is measured based on the phase difference, and it returns to the same phase after one round. Therefore, when the distance to the subject exceeds the maximum distance that can be measured, aliasing (folded distance) occurs. It will occur. Aliasing problems are particularly likely to occur in high-reflectivity subjects with a large specular reflection component, such as metal and glass.

And for objects at the alias distance (distance where aliasing occurs), the actual measurement distance (distance measurement depth) is closer than the original correct distance (true depth). For example, the actual measurement distance (distance measurement depth) of a subject at a distance of 2.5 m is 1.0 m. As described above, the distance measurement result of the object at the alias distance is output from the distance measurement device 1 as an erroneous distance measurement result.

As a result, in the subsequent system that performs various controls based on the distance measurement result (distance image information including the depth and the reliability value) of the distance measurement device 1, an error is made based on the incorrect distance measurement result of the distance measurement device 1. Control will be done. As an example, when the distance measurement result of the distance measuring device 1 is applied to autofocus that automatically focuses the camera, accurate focus control is not performed.

Here, consider the size of the subject reflected in the image. The size of the subject reflected in the image when the true depth is 1.0 m and the distance measuring depth is 1.0 m is shown in FIG. 8A, and the size of the subject reflected in the image when the true depth is 2.5 m and the distance measuring depth is 1.0 m is shown. The size is shown in FIG. 8B. FIG. 8B is an example in the case of a subject at an alias distance.

In both the case of FIG. 8A and the case of FIG. 8B, the distance measuring distance (distance measuring depth) is the same 1.0 m. As is clear from the comparison between FIGS. 8A and 8B, the distance is measured at the same distance of 1.0 m, but the size of the subject is different. That is, the size of the subject reflected in the image is smaller in the case of the true depth of 2.5 m (FIG. 8B) than in the case of the true depth of 1.0 m (FIG. 8A). This means that the size of the subject reflected in the image differs depending on the distance of the subject from the distance measuring device 1.

<Embodiment of the present disclosure>
In the embodiment of the present disclosure, as described above, it is a problem peculiar to the indirect ToF type distance measuring device by utilizing the property that the size of the subject reflected in the image changes depending on the distance of the subject from the distance measuring device 1. Try to solve an aliasing problem. In the present embodiment, the process of solving this aliasing problem is executed as one of the filter functions in the artifact removing unit 514 in the distance image calculation unit 51 of the distance measuring unit 50.

[Processing example for solving aliasing problem]
Hereinafter, an example of a process for solving the aliasing problem, which is an example of the process of the control method of the ranging device of the present disclosure, which is executed by the artifact removing unit 514, will be described with reference to the flowchart of FIG.

In the following, in the case of a configuration in which the function of the artifact removing unit 514 is realized by a processor, a series of processes for solving the aliasing problem will be executed under the control of the processor constituting the artifact removing unit 514. ..

The processor (hereinafter, simply referred to as “processor”) constituting the artifact removal unit 514 divides the image into segments (parts / objects) based on the depth information calculated by the depth calculation unit 511 in FIG. Perform the process (step S11). In this segmentation process, taking the cases of FIGS. 8A and 8B as an example, a process of dividing a large background portion and a round portion in the center into segments is performed.

Next, the processor performs a labeling process for assigning labels 1 to n (n is an integer of 2 or more) to each segment (step S12). In this labeling process, the image is scanned line by line, and the same label is attached to pixels in which neighboring pixels and depth information are continuous, so that different labels are assigned to each segment (part / object). ..

Next, the processor creates a component object for each label 1 to n (step S13). Here, the "component object" is the number of pixels of the segment and the depth average value (distance average value).

Next, the processor increments the label counter i (step S14), and then determines the threshold number of pixels of the segment from the depth mean value for label 1 (i = 1) (step S15). In the process of step S15, the threshold value of the number of pixels of the segment is determined in consideration of the distance for each segment. More specifically, for a short distance where the distance of the segment from the distance measuring device 1 is relatively short. Set the threshold value large and set the threshold value for relatively long distances small.

For short distance / long distance, a predetermined distance (for example, 1 m) can be defined as a reference (threshold value). However, the threshold value of the number of pixels of the segment is not limited to the two patterns of short distance / long distance, and may be further subdivided and increased to three or more patterns. Details of setting the threshold value for the number of pixels in the segment will be described later.

Subsequently, the processor determines whether or not the number of pixels exceeds the threshold value for the segment of label 1 (step S16). In this determination process, when the processor determines that the number of pixels exceeds the threshold value (YES in S16), the segment is valid (step S17), and when it determines that the number of pixels is equal to or less than the threshold value (YES in S16). NO in S16), the segment is invalidated (step S18).

After the process of step S17 or the process of step S18 is completed, the processor determines whether or not the process of valid / invalidating the segment is completed for all the labels 1 to n (step S19), and the process is completed. If (YES in S19), the above-mentioned series of processes for solving the aliasing problem is completed. If the processor has not finished for all labels 1 to n (NO in S19), the processor returns to step S14 and increments the label counter i, and thereafter, the processes of steps S15 to S19 are performed for all labels 1 to n. Is repeatedly executed until it is determined that the valid / invalid processing of the segment is completed.

As described above, in the distance measuring device 1 according to the present embodiment, the depth calculation unit 511 calculates the depth information of the subject based on the RAW image data given by the light detection unit 30, and segmentation is performed based on the depth information. Processing is performed and processing is performed to invalidate segments whose number of pixels is smaller than the threshold value. By this processing, the aliasing problem, which is a problem peculiar to the indirect ToF type distance measuring device, can be solved. That is, the distance measurement result of the object at the alias distance is not output from the distance measurement device 1 as an erroneous distance measurement result, and only the distance measurement result of the object for which the correct distance (true distance) is measured. Can be output.

[Threshold setting for the number of pixels in a segment]
Here, the threshold setting of the number of pixels of the segment, which is the criterion for determining the validity / invalidity of the segment, will be described.

The size of the subject (object) to be distance-measured is defined, the number of pixels of the segment to be imaged for each distance is calculated from the focal length and pixel pitch of the light detection unit 30, and the calculated number of pixels is used as a threshold value. be able to.
-For the subject to be distance-measured, select an object that is particularly prone to aliasing tasks, such as metal or glass, which has a high specular reflection component and high reflectance.
-The aliasing problem does not occur in the entire object where the aliasing problem is likely to occur, but tends to occur in the vicinity of the region where the laser beam faces. Therefore, even if it is a large object (subject), the size of the area where the aliasing problem occurs may be set.

<Modification example>
Although the technique of the present disclosure has been described above based on the preferred embodiment, the technique of the present disclosure is not limited to the embodiment. The configuration and structure of the distance measuring device described in each of the above embodiments are examples, and can be changed as appropriate. For example, in the above embodiment, the threshold value of the number of pixels of the segment is determined based on the depth average value, but the present invention is not limited to this, and the following configuration can be adopted.

For example, as shown in FIG. 10, the RGB camera 61 is used, and the threshold value for each subject is set in advance. Then, the object recognition unit 62 recognizes the subject based on the image pickup output of the RGB camera 61, and the parameter control unit 63 determines a preset threshold value based on the recognition result. However, the subject recognition is not performed based on the output of the RGB camera 61, but is performed based on the final output of the distance measuring device 1, that is, the distance image information including each information of the depth and the reliability value. Alternatively, it may be performed based on both the output of the RGB camera 61 and the final output of the distance measuring device 1.

<Application example>
The ranging device of the present disclosure is a moving object of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, a robot, a construction machine, and an agricultural machine (tractor). It can be used as a distance measuring device mounted on the vehicle. Further, in the above embodiment, the case where the distance measuring device of the present disclosure is used as a means for acquiring a distance image (depth map) has been described as an example, but it is not only used as a means for acquiring a distance image. , Can be applied to autofocus, which automatically focuses the camera.

<Electronic device of the present disclosure>
The distance measuring device of the present disclosure described above can be used as a distance measuring device mounted on various electronic devices. Examples of electronic devices equipped with a distance measuring device include mobile devices such as smartphones, digital cameras, and tablets. However, it is not limited to mobile devices.

Hereinafter, smartphones and digital still cameras will be illustrated as specific examples of electronic devices using the distance measuring device of the present disclosure. The distance image information (depth information) of the distance measuring device mounted on the smartphone and the digital still camera is used as the lens driving information for autofocus. However, the specific examples illustrated here are only examples, and are not limited to these specific examples.

[Specific example 1: Example of smartphone]
FIG. 11A shows an external view of the smartphone according to the specific example 1 of the electronic device of the present disclosure as seen from the front side, and FIG. 11B shows an external view as seen from the back side. The smartphone 100 according to this specific example includes a display unit 120 on the front side of the housing 110. Further, the smartphone 100 is provided with an image pickup unit 130 on the upper side of the back surface side of the housing 110, for example.

The distance measuring device 1 according to the embodiment of the present disclosure described above can be mounted on, for example, a smartphone 100 which is an example of a mobile device having the above configuration. In this case, the light source 20 and the light detection unit 30 of the distance measuring device 1 can be arranged in the vicinity of the imaging unit 130, for example, as shown in FIG. 11B. However, the arrangement example of the light source 20 and the light detection unit 30 shown in FIG. 11B is an example, and is not limited to this arrangement example.

As described above, the smartphone 100 according to the specific example 1 is manufactured by mounting the distance measuring device 1 according to the embodiment of the present disclosure. Then, since the smartphone 100 according to the specific example 1 can obtain an accurate distance measurement result while solving the aliasing problem by mounting the above distance measurement device 1, the focus can be obtained based on the distance measurement result. A suitable captured image can be obtained.

[Specific example 2: Example of digital still camera]
FIG. 12B is an external view of the interchangeable lens type single-lens reflex type digital still camera according to Specific Example 2 of the electronic device of the present disclosure as viewed from the front side, and FIG. 12B is an external view as viewed from the back side.

The interchangeable lens single-lens reflex type digital still camera 200 has, for example, an interchangeable photographing lens unit (interchangeable lens) 212 on the front right side of the camera body (camera body) 211, and is gripped by the photographer on the front left side. It has a grip portion 213 for using the lens. A monitor 214 is provided at substantially the center of the back surface of the camera body 211. A viewfinder (eyepiece window) 215 is provided above the monitor 214. By looking into the viewfinder 215, the photographer can visually recognize the optical image of the subject guided by the photographing lens unit 212 and determine the composition.

The ranging device 1 according to the embodiment of the present disclosure described above can be mounted on, for example, a digital still camera 200 which is an example of a mobile device having the above configuration. In this case, the light source 20 and the light detection unit 30 of the distance measuring device 1 can be arranged in the vicinity of the photographing lens unit 212, for example, as shown in FIG. 12A. However, the arrangement example of the light source 20 and the light detection unit 30 shown in FIG. 12A is an example, and is not limited to this arrangement example.

As described above, the digital still camera 200 according to the second embodiment is manufactured by mounting the distance measuring device 1 according to the embodiment of the present disclosure. Then, since the digital still camera 200 according to the specific example 2 can obtain an accurate distance measurement result while solving the aliasing problem by mounting the above distance measurement device 1, it is possible to obtain an accurate distance measurement result based on the distance measurement result. An in-focus captured image can be obtained.

<Structure that can be taken by this disclosure>
The present disclosure may also have the following configuration.

≪A. Distance measuring device ≫
[A-1] A photodetector that receives light from a subject,
A depth calculation unit that calculates the depth information of the subject based on the output of the light detection unit, and
An artifact removal unit that divides an image into segments based on the depth information calculated by the depth calculation unit, enables segments in which the number of pixels in each segment exceeds a predetermined threshold value, and invalidates segments below a predetermined threshold value.
A distance measuring device equipped with.
[A-2] The artifact removal unit labels each segment and creates a component object for each label.
The distance measuring device according to the above [A-1].
[A-3] The artifact removal unit assigns a different label to each segment by a process of labeling neighboring pixels and pixels having continuous depth information with the same label.
The distance measuring device according to the above [A-2].
[A-4] The component object is the number of pixels and the average depth of the segment.
The distance measuring device according to the above [A-2] or the above [A-3].
[A-5] The artifact removal unit changes a predetermined threshold value according to the distance of the segment from the distance measuring device.
The distance measuring device according to any one of the above [A-1] to the above [A-4].
[A-6] The artifact removing unit sets a large threshold value for a relatively short distance and a small threshold value for a relatively long distance for the distance of the segment from the distance measuring device.
The distance measuring device according to the above [A-5].

≪B. Electronic equipment ≫
[B-1] Photodetector that receives light from the subject,
A depth calculation unit that calculates the depth information of the subject based on the output of the light detection unit, and
An artifact removal unit that divides an image into segments based on the depth information calculated by the depth calculation unit, enables segments in which the number of pixels in each segment exceeds a predetermined threshold value, and invalidates segments below a predetermined threshold value.
An electronic device having a ranging device.
[B-2] The artifact removal unit labels each segment and creates a component object for each label.
The electronic device according to the above [B-1].
[B-3] The artifact removal unit assigns a different label to each segment by a process of labeling neighboring pixels and pixels having continuous depth information with the same label.
The electronic device according to the above [B-2].
[B-4] The component object is the number of pixels and the average depth of the segment.
The electronic device according to the above [B-2] or the above [B-3].
[B-5] The artifact removal unit changes a predetermined threshold value according to the distance of the segment from the distance measuring device.
The electronic device according to any one of the above [B-1] to the above [B-4].
[B-6] The artifact removing unit sets a large threshold value for a relatively short distance and a small threshold value for a relatively long distance for the distance of the segment from the distance measuring device.
The electronic device according to the above [B-5].

1 ... Distance measuring device, 10 ... Subject (measurement object), 20 ... Light source, 30 ... Light detection unit, 40 ... AE control unit, 41 ... Next frame light emission / exposure Condition calculation unit, 42 ... Next frame light emission / exposure control unit, 50 ... Distance measurement unit, 51 ... Distance image calculation unit, 61 ... RGB camera, 62 ... Object recognition unit, 63.・・ Parameter control unit

Claims

Photodetector that receives light from the subject,
A depth calculation unit that calculates the depth information of the subject based on the output of the light detection unit, and
An artifact removal unit that divides an image into segments based on the depth information calculated by the depth calculation unit, enables segments in which the number of pixels in each segment exceeds a predetermined threshold value, and invalidates segments below a predetermined threshold value.
A distance measuring device equipped with.
The artifact remover labels each segment and creates a component object for each label.
The ranging device according to claim 1.
The artifact removal unit assigns a different label to each segment by a process of labeling nearby pixels and pixels having continuous depth information.
The ranging device according to claim 2.
The component object is the number of pixels and the average depth of the segment.
The ranging device according to claim 2.
The artifact remover changes a predetermined threshold value according to the distance of the segment from the distance measuring device.
The ranging device according to claim 1.
The artifact remover sets a large threshold for short distances, which is relatively short, and a small threshold for long distances, which is relatively far from the distance measuring device.
The ranging device according to claim 5.
A photodetector that receives light from the subject, and
Depth calculation unit, which calculates the depth information of the subject based on the output of the light detection unit,
In controlling the distance measuring device equipped with
The image is divided into segments based on the depth information, the segments in which the number of pixels of each segment exceeds a predetermined threshold are enabled, and the segments below the predetermined threshold are invalidated.
How to control the ranging device.
Photodetector that receives light from the subject,
A depth calculation unit that calculates the depth information of the subject based on the output of the light detection unit, and
An artifact removal unit that divides an image into segments based on depth information, enables segments in which the number of pixels in each segment exceeds a predetermined threshold, and invalidates segments below a predetermined threshold.
An electronic device having a ranging device.