JP2021051015A - Distance measuring device, distance measuring method, and program - Google Patents

Abstract

To enable distance measurement to be performed with better accuracy.SOLUTION: The present invention comprises: a first determination unit for determining whether a difference in depth value between a first pixel in a depth map and a second pixel adjacent to the first pixel is larger than a first threshold; and a second determination unit for determining, when it is determined by the first determination unit that the difference in distance between the first and second pixels is larger than the first threshold, whether a difference in reliability between the first and second pixels is larger than a second threshold. When it is determined by the second determination unit that the difference in reliability between the first and second pixels is larger than the second threshold, it is confirmed that the first pixel is a defective pixel. The present technology can be applied to, for example, a distance measuring device.SELECTED DRAWING: Figure 1

Description

The present technology relates to a distance measuring device, a distance measuring method, and a program, for example, a distance measuring device, a distance measuring method, and a program capable of accurately detecting an erroneous distance measuring result.

In recent years, advances in semiconductor technology have led to miniaturization of distance measuring devices that measure the distance to an object. As a result, for example, it has been realized that the distance measuring device is mounted on a mobile terminal such as a so-called smartphone, which is a small information processing device having a communication function. As a distance measuring device (sensor) for measuring the distance to an object, for example, there is a TOF (Time Of Flight) sensor (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-090268

When there is an erroneous distance measurement result, it is desired to improve the accuracy of the distance measurement itself by detecting the erroneous distance measurement result with high accuracy.

This technique was made in view of such a situation, and makes it possible to accurately detect an erroneous distance measurement result.

In the first ranging device of one aspect of the present technology, whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value. When it is determined by the first determination unit and the first determination unit that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the first A second determination unit for determining whether or not the difference between the reliability of the pixel and the reliability of the second pixel is larger than the second threshold value is provided. When it is determined that the difference between the reliability of the pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel.

In the second ranging device of one aspect of the present technology, whether or not the difference between the depth value of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value. When it is determined by the first determination unit and the first determination unit that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the first A second determination unit for determining whether or not the difference between the reflectance of the pixel and the reflectance of the second pixel is larger than the second threshold value is provided, and the first determination unit provides the first determination unit. When it is determined that the difference between the reflectance of the pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel.

In the first distance measuring method of one aspect of the present technology, the distance measuring device measures the difference between the depth value of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel. When it is determined whether or not it is larger than the first threshold value and it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reliability of the first pixel is determined. It is determined whether or not the difference in reliability of the second pixel is larger than the second threshold value, and the difference between the reliability of the first pixel and the reliability of the second pixel is greater than the second threshold value. If it is determined that is also large, the first pixel is determined to be a defective pixel.

In the second distance measuring method of one aspect of the present technology, the distance measuring device measures the difference between the depth value of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel. When it is determined whether or not it is larger than the first threshold value and it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, it is determined to be the reflectance of the first pixel. It is determined whether or not the difference in the reflectance of the second pixel is larger than the second threshold value, and the difference between the reflectance of the first pixel and the reflectance of the second pixel is greater than the second threshold value. If it is determined that is also large, the first pixel is determined to be a defective pixel.

The first program of one aspect of the present technology asks the computer whether the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value. If it is determined whether or not the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reliability of the first pixel and the reliability of the second pixel are determined. When it is determined whether or not the difference in degrees is larger than the second threshold value, and it is determined that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value. , The first pixel is subjected to a process including a step of determining that it is a defective pixel.

In the second program of one aspect of the present technology, the computer determines whether the difference between the depth value of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value. If it is determined whether or not the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reflectance of the first pixel and the reflection of the second pixel are determined. When it is determined whether or not the difference in rate is larger than the second threshold value, and it is determined that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value. , The first pixel is subjected to a process including a step of determining that it is a defective pixel.

In the first distance measuring device, the distance measuring method, and the program of one aspect of the present technology, the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is the first. When it is determined that it is larger than the threshold value of 1, it is further determined whether or not the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value. Then, when it is determined that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel.

In the second distance measuring device, the distance measuring method, and the program of one aspect of the present technology, the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is the first. When it is determined that it is larger than the threshold value of 1, it is further determined whether or not the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value. Then, when it is determined that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel.

The distance measuring device may be an independent device or an internal block constituting one device.

In addition, the program can be provided by transmitting via a transmission medium or by recording on a recording medium.

It is a figure which shows the structure of one Embodiment of the distance measuring apparatus to which this technique is applied. It is a figure which shows the structural example of the light receiving part. It is a figure which shows the structural example of a pixel. It is a figure explaining the distribution of electric charge in a pixel. It is a figure which shows an example of four kinds of light-receiving which the phase was delayed every 90 degrees. It is a figure which shows an example of the detection signal in the detection period by the light reception with a phase lag of 0 degree. It is a figure which shows an example of the detection signal in the detection period by the light reception with a phase lag of 90 degrees. It is a figure which shows an example of the detection signal in the detection period by the light reception with a phase lag of 180 degrees. It is a figure which shows an example of the detection signal in the detection period by the light reception with a phase delay of 270 degrees. It is a figure for demonstrating the detection signal in one frame. It is a figure for demonstrating the relationship between a detection period and a detection signal. It is a figure for demonstrating the detection signal in one frame. It is a figure for demonstrating the detection signal in one frame. It is a figure for demonstrating the flying pixel. It is a figure for demonstrating the flying pixel. It is a figure for demonstrating the flying pixel. It is a figure for demonstrating the flying pixel. It is a flowchart for demonstrating the 1st process which concerns on the detection of a flying pixel. It is a figure for demonstrating the relationship between the processing target and the peripheral pixel. It is a flowchart for demonstrating the 2nd process which concerns on the detection of a flying pixel. It is a figure for demonstrating the setting of a threshold value. It is a block diagram which shows the structural example of an electronic device. It is a block diagram which shows the configuration example of a personal computer. It is a figure which shows an example of the schematic structure of the endoscopic surgery system. It is a block diagram which shows an example of the functional structure of a camera head and a CCU. It is a block diagram which shows an example of the schematic structure of a vehicle control system. It is explanatory drawing which shows an example of the installation position of the vehicle exterior information detection unit and the image pickup unit.

Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described.

This technique can be applied to, for example, a light receiving element constituting a distance measuring system that measures a distance by an indirect TOF method, an imaging device having such a light receiving element, and the like.

For example, a distance measuring system is an in-vehicle system that is mounted on a vehicle and measures the distance to an object outside the vehicle, or measures the distance to an object such as a user's hand, and based on the measurement result, the user It can be applied to a system for recognizing gestures. In this case, the result of gesture recognition can be used, for example, for operating a car navigation system.

<Configuration example of distance measuring device>
FIG. 1 shows a configuration example of an embodiment of a distance measuring device to which the present technology is applied.

The distance measuring device 10 includes a lens 11, a light receiving unit 12, a signal processing unit 13, a light emitting unit 14, a light emitting control unit 15, and a filter unit 16. The distance measuring device 10 of FIG. 1 irradiates an object with light, and the light (irradiation light) receives the light reflected by the object (reflected light) to measure the distance to the object.

The light emitting system of the distance measuring device 10 includes a light emitting unit 14 and a light emitting control unit 15. In the light emitting system, the light emitting control unit 15 irradiates infrared light (IR) with the light emitting unit 14 according to the control from the signal processing unit 13. An IR band filter may be provided between the lens 11 and the light receiving unit 12, and the light emitting unit 14 may emit infrared light corresponding to the transmission wavelength band of the IR bandpass filter.

The light emitting unit 14 may be arranged inside the housing of the distance measuring device 10 or may be arranged outside the housing of the distance measuring device 10. The light emitting control unit 15 causes the light emitting unit 14 to emit light at a predetermined frequency.

The signal processing unit 13 functions as a calculation unit that calculates the distance (depth value) from the distance measuring device 10 to the object based on the detection signal (pixel data) supplied from the light receiving unit 12, for example. The signal processing unit 13 generates a depth map in which a depth value (depth information) is stored as a pixel value of each pixel 50 (FIG. 2) of the light receiving unit 12, and outputs the depth map to the filter unit 16. Further, the signal processing unit 13 also calculates the reliability of the calculated depth value for each pixel 50 of the light receiving unit 12, and stores the reliability (luminance information) as the pixel value of each pixel 50 of the light receiving unit 12. A map is generated and output to the filter unit 16.

<Structure of image sensor>
FIG. 2 is a block diagram showing a configuration example of the light receiving unit 12. The light receiving unit 12 can be a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The light receiving unit 12 includes a pixel array unit 41, a vertical drive unit 42, a column processing unit 43, a horizontal drive unit 44, and a system control unit 45. The pixel array unit 41, the vertical drive unit 42, the column processing unit 43, the horizontal drive unit 44, and the system control unit 45 are formed on a semiconductor substrate (chip) (not shown).

In the pixel array unit 41, unit pixels (for example, pixel 50 in FIG. 3) having a photoelectric conversion element that generates an electric charge of an amount corresponding to the amount of incident light and accumulates it inside are two-dimensionally arranged in a matrix. There is. In the following, the light charge of the amount of charge corresponding to the amount of incident light may be simply described as "charge", and the unit pixel may be simply described as "pixel".

Further, in the pixel array unit 41, pixel drive lines 46 are formed for each row with respect to the matrix-like pixel array along the left-right direction (arrangement direction of pixels in the pixel row) in the figure, and vertical signal lines 47 for each column. Is formed along the vertical direction (arrangement direction of pixels in the pixel array) in the figure. One end of the pixel drive line 46 is connected to the output end corresponding to each line of the vertical drive unit 42.

The vertical drive unit 42 is composed of a shift register, an address decoder, and the like, and is a pixel drive unit that drives each pixel of the pixel array unit 41 at the same time for all pixels or in units of rows. The pixel signal output from each unit pixel of the pixel row selectively scanned by the vertical drive unit 42 is supplied to the column processing unit 43 through each of the vertical signal lines 47. The column processing unit 43 performs predetermined signal processing on the pixel signal output from each unit pixel of the selected row through the vertical signal line 47 for each pixel column of the pixel array unit 41, and the pixel signal after the signal processing. Temporarily hold.

Specifically, the column processing unit 43 performs at least noise removal processing, for example, CDS (Correlated Double Sampling) processing as signal processing. By the correlated double sampling by the column processing unit 43, fixed pattern noise peculiar to the pixel such as reset noise and threshold variation of the amplification transistor is removed. In addition to the noise removal processing, the column processing unit 43 can be provided with, for example, an AD (analog-digital) conversion function, and the signal level can be output as a digital signal.

The horizontal drive unit 44 is composed of a shift register, an address decoder, and the like, and sequentially selects unit circuits corresponding to the pixel strings of the column processing unit 43. By the selective scanning by the horizontal drive unit 44, the pixel signals signal-processed by the column processing unit 43 are sequentially output to the signal processing unit 48.

The system control unit 45 is composed of a timing generator or the like that generates various timing signals, and the vertical drive unit 42, the column processing unit 43, the horizontal drive unit 44, or the like is based on the various timing signals generated by the timing generator. Drive control is performed.

In the pixel array unit 41, the pixel drive lines 46 are wired along the row direction for each pixel row with respect to the matrix-like pixel array, and two vertical signal lines 47 are wired along the column direction in each pixel row. ing. For example, the pixel drive line 46 transmits a drive signal for driving when reading a signal from a pixel. In FIG. 2, the pixel drive line 46 is shown as one wiring, but the wiring is not limited to one. One end of the pixel drive line 46 is connected to the output end corresponding to each line of the vertical drive unit 42.

<Unit pixel structure>
Next, a specific structure of the unit pixels 50 arranged in a matrix in the pixel array unit 41 will be described.

The pixel 50 includes a photodiode 61 (hereinafter, referred to as PD61) which is a photoelectric conversion element, and is configured so that the electric charge generated by the PD61 is distributed to the tap 51-1 and the tap 51-2. Then, of the electric charges generated by the PD 61, the electric charges distributed to the taps 51-1 are read out from the vertical signal line 47-1 and output as the detection signal SIG1. Further, the electric charge distributed to the tap 51-2 is read out from the vertical signal line 47-2 and output as a detection signal SIG2.

The tap 51-1 is composed of a transfer transistor 62-1, an FD (Floating Diffusion) 63-1, a reset transistor 64, an amplification transistor 65-1, and a selection transistor 66-1. Similarly, the tap 51-2 is composed of a transfer transistor 62-2, an FD63-2, a reset transistor 64, an amplification transistor 65-2, and a selection transistor 66-2.

As shown in FIG. 3, the reset transistor 64 may be shared by the FD63-1 and the FD63-2, or may be provided in each of the FD63-1 and the FD63-2.

When the reset transistor 64 is provided in each of the FD63-1 and the FD63-2, the reset timing can be controlled individually for the FD63-1 and the FD63-2, so that fine control can be performed. .. When the reset transistor 64 common to the FD63-1 and the FD63-2 is provided, the reset timing can be made the same for the FD63-1 and the FD63-2, the control becomes simple, and the circuit configuration is also simple. Can be reset.

In the following description, a case where a reset transistor 64 common to the FD63-1 and the FD63-2 is provided will be taken as an example to continue the description.

The distribution of electric charges in the pixel 50 will be described with reference to FIG. Here, the distribution means that the electric charges accumulated in the pixel 50 (PD61) are read out at different timings, so that the electric charges are read out for each tap.

As shown in FIG. 4, irradiation light modulated (1 cycle = Tp) so as to repeat irradiation on / off within the irradiation time is output from the light emitting unit 14, and only the delay time Td according to the distance to the object. After a delay, the reflected light is received by the PD61.

The transfer control signal TRT_A controls the on / off of the transfer transistor 62-1, and the transfer control signal TRT_B controls the on / off of the transfer transistor 62-2. As shown in the figure, the transfer control signal TRT_A has the same phase as the irradiation light, while the transfer control signal TRT_B has the phase in which the transfer control signal TRT_A is inverted.

Therefore, the electric charge generated by the photodiode 61 receiving the reflected light is transferred to the FD unit 63-1 while the transfer transistor 62-1 is on according to the transfer control signal TRT_A. Further, it is transferred to the FD unit 63-2 while the transfer transistor 62-2 is turned on according to the transfer control signal TRT_B. As a result, the electric charges transferred via the transfer transistor 62-1 are sequentially accumulated in the FD section 63-1 during a predetermined period in which the irradiation light of the irradiation time T is periodically irradiated, and the transfer transistor 62-2. The charges transferred via the FD unit 63-2 are sequentially accumulated in the FD unit 63-2.

Then, when the selection transistor 66-1 is turned on according to the selection signal SELm1 after the end of the period for accumulating the electric charge, the electric charge accumulated in the FD unit 63-1 is read out via the vertical signal line 47-1. A detection signal A corresponding to the amount of electric charge is output from the light receiving unit 12. Similarly, when the selection transistor 66-2 is turned on according to the selection signal SELm2, the electric charge accumulated in the FD unit 63-2 is read out via the vertical signal line 47-2, and the detection signal corresponding to the amount of the electric charge is read out. B is output from the light receiving unit 12.

The electric charge stored in the FD unit 63-1 is discharged when the reset transistor 64 is turned on according to the reset signal RST. Similarly, the electric charge accumulated in the FD unit 63-2 is discharged when the reset transistor 64 is turned on according to the reset signal RST.

In this way, the pixel 50 distributes the electric charge generated by the reflected light received by the photodiode 61 to the taps 51-1 and the taps 51-2 according to the delay time Td, and outputs the detection signal A and the detection signal B. can do. The delay time Td corresponds to the time during which the light emitted by the light emitting unit 14 flies to the object, is reflected by the object, and then flies to the light receiving unit 12, that is, the delay time Td corresponds to the distance to the object. Therefore, the distance measuring device 10 can obtain the distance (depth) to the object according to the delay time Td based on the detection signal A and the detection signal B.

<Phase shift measurement on the light receiving side>
A case where a detection signal is acquired by receiving light with a phase shift on the receiving side will be described with reference to FIGS. 5 to 9.

For example, as shown in FIG. 5, four types of light reception with a phase delay are performed every 90 degrees. That is, based on the light receiving with a phase delay of 0 degrees, which is received without any phase shift on the irradiation side, the light receiving with a phase delay of 90 degrees, the light receiving with a phase delay of 180 degrees, and the light receiving with a phase delay of 270 degrees are performed, and the detection signals are respectively. A period (quad) for detecting A and the detection signal B is provided four times.

That is, as shown in FIG. 6, for example, a detection period Q0 for detecting reflected light by receiving a light with a phase delay of 0 degrees, a detection period Q1 for detecting reflected light with a light receiving with a phase delay of 90 degrees, and a light receiving with a phase delay of 180 degrees. A detection period Q2 for detecting the reflected light and a detection period Q3 for detecting the reflected light by receiving a light with a phase delay of 270 degrees are continuously provided.

FIG. 6 shows an example of the received light, the reflected light, the transfer control signal TRT_A and the transfer control signal TRT_B, and the detection signal A and the detection signal B in the detection period Q0. As shown in FIG. 6, at the same timing (phase delay 0 degree) as the start of light emission of the irradiation light, the transfer control signal TRT_A of the tap 51-1 is turned on, and the light reception (transfer) is started at the tap 51-1. .. Further, at the timing when the transfer control signal TRT_A is turned off, the transfer control signal TRT_B is turned on, and light reception (transfer) is started at the tap 51-2.

By receiving light with a phase delay of 0 degrees in this way, the electric charges are distributed to the taps 51-1 and 51-2 by the amount of electric charges corresponding to the delay time TR, and the electric charges are accumulated in the integration period, respectively. After that, in the read-out period, the electric charges of the amount of charges accumulated in the integration period are read out, and the detection signal A0 and the detection signal B0 in the detection period Q0 are output.

FIG. 7 shows an example of the received light, the reflected light, the transfer control signal TRT_A and the transfer control signal TRT_B, and the detection signal A and the detection signal B in the detection period Q1. As shown in FIG. 7, the transfer control signal TRT_A of the tap 51-1 is turned on at the timing when the phase is delayed by 90 degrees from the start of the emission of the irradiation light, and the light reception (transfer) is started at the tap 51-1. Further, at the timing when the transfer control signal TRT_A is turned off, the transfer control signal TRT_B is turned on, and light reception (transfer) is started at the tap 51-2.

By receiving light with a phase delay of 90 degrees in this way, the charges are distributed to the taps 51-1 and 51-2 by the amount of charges corresponding to the delay time TR, and the charges are accumulated in the integration period, respectively. After that, in the read-out period, the electric charges of the amount of charges accumulated in the integration period are read out, and the detection signal A90 and the detection signal B90 in the detection period Q1 are output.

FIG. 8 shows an example of the received light, the reflected light, the transfer control signal TRT_A and the transfer control signal TRT_B, and the detection signal A and the detection signal B in the detection period Q2. As shown in FIG. 8, the transfer control signal TRT_A of the tap 51-1 is turned on at the timing when the phase is delayed by 180 degrees from the start of the emission of the irradiation light, and the light reception (transfer) is started at the tap 51-1. Further, at the timing when the transfer control signal TRT_A is turned off, the transfer control signal TRT_B is turned on, and light reception (transfer) is started at the tap 51-2.

By receiving light with a phase delay of 180 degrees in this way, the charges are distributed to the taps 51-1 and 51-2 by the amount of charges corresponding to the delay time TR, and the charges are accumulated during the integration period, respectively. After that, in the read-out period, the electric charges of the amount of charges accumulated in the integration period are read out, and the detection signal A180 and the detection signal B180 in the detection period Q2 are output.

FIG. 9 shows an example of the received light, the reflected light, the transfer control signal TRT_A and the transfer control signal TRT_B, and the detection signal A and the detection signal B in the detection period Q3. As shown in FIG. 9, the transfer control signal TRT_A of the tap 51-1 is turned on at the timing when the phase is delayed by 270 degrees from the start of the emission of the irradiation light, and the light reception (transfer) is started at the tap 51-1. Further, at the timing when the transfer control signal TRT_A is turned off, the transfer control signal TRT_B is turned on, and light reception (transfer) is started at the tap 51-2.

By receiving light with a phase delay of 270 degrees in this way, the charges are distributed to the taps 51-1 and 51-2 by the amount of charges corresponding to the delay time TR, and the charges are accumulated during the integration period, respectively. After that, in the read-out period, the electric charges of the amount of charges accumulated in the integration period are read out, and the detection signal A270 and the detection signal B270 in the detection period Q3 are output.

As described above, in the detection period Q0, the detection signal A0 and the detection signal B0 are detected by the light reception with a phase delay of 0 degrees, and in the detection period Q1, the detection signal A90 and the detection signal B90 are detected by the light reception with a phase delay of 90 degrees. Similarly, in the detection period Q2, the detection signal A180 and the detection signal B180 are detected by receiving a light with a phase delay of 180 degrees, and in the detection period Q3, the detection signal A270 and the detection signal B270 are detected by receiving a light with a phase delay of 270 degrees.

On the irradiation side, regardless of the phase delay at which the light reception is started, the irradiation of the irradiation light is always started at the same timing without the phase delay.

In this way, when receiving light with four phase delays with the two taps 51 and measuring the distance to a predetermined object, the signal processing unit 13 (FIG. 1) is based on the signals obtained by the four detection periods. Processing is done.

<Calculation of distance>
As shown in FIG. 10, the detection period of one frame is composed of a detection period Q0, a detection period Q1, a detection period Q2, and a detection period Q3. In the detection period Q0, the detection signal A0 and the detection signal B0 are acquired, and in the detection period Q1, the detection signal A90 and the detection signal B90 are acquired. Further, in the detection period Q2, the detection signal A180 and the detection signal B180 are acquired, and in the detection period Q3, the detection signal A270 and the detection signal B270 are acquired.

Using these detection signals, the signal processing unit 13 (FIG. 1) calculates the phase difference θ based on the following equation (1), calculates the distance D based on the following equation (2), and calculates the distance D based on the following equation (1). The reliability c is calculated based on 3).

In the formula (1), I represents a value obtained by subtracting the value C180 obtained by subtracting the detection signal B180 from the detection signal A180 from the value C0 obtained by subtracting the detection signal B0 from the detection signal A0. Q represents a value obtained by subtracting the value C270 obtained by subtracting the detection signal B270 from the detection signal A270 from the value C90 obtained by subtracting the detection signal B90 from the detection signal A90. The phase difference θ is calculated by calculating the arc tangent of (Q / I).

In equation (2), C is the speed of light and Tp is the pulse width. The delay time Td can be obtained based on the phase difference θ, and the distance D to the object can be obtained from the delay time Td.

Equation (3) is an equation for calculating a value representing the reliability of the calculated distance. The reliability c is obtained by calculating the square root of the value obtained by adding the square of I and the square of Q. The calculation of the reliability c is not an indispensable element in the calculation of the distance D, and can be omitted. Further, the reliability c may be calculated by an equation other than the equation (3). For example, the sum of the absolute values of I and Q can be the reliability c. Here, the description will be continued assuming that the reliability is calculated by the formula (3), but the case where the reliability is calculated by another formula is also within the scope of the present technology.

In the following description, a case where the distance to a predetermined object is measured by using two taps 51 with irradiation light having four phase differences (hereinafter, appropriately referred to as two taps and four phases) will be described as an example. However, when measuring the distance to a predetermined object using two phase-difference irradiation lights with two taps 51, or to a predetermined object using four phase-difference irradiation lights with one tap 51. This technology can also be applied to the case of measuring the distance of.

<About 2 taps and 2 phases>
When measuring the distance to a predetermined object by using irradiation light with two phase differences with two taps 51 or by receiving light with two phase differences (hereinafter, appropriately referred to as two taps and two phases). ) Is briefly explained. Here, the description will be continued by taking as an example the case where the distance to a predetermined object is measured by receiving light with two phase differences.

FIG. 11 is a diagram showing exposure timings of four phases of 0 degree, 90 degree, 180 degree, and 270 degree with their heads aligned so that the phase difference can be easily understood.

Actually, as shown in FIG. 10, the imaging for acquiring the detection signal A0 and the detection signal B0 is performed in the detection period Q0, and the imaging for acquiring the detection signal A90 and the detection signal B90 is performed in the detection period Q1. In the detection period Q2, the imaging for acquiring the detection signal A180 and the detection signal B180 is performed, and the imaging for acquiring the detection signal A270 and the detection signal B270 is performed in the detection period Q3.

FIG. 11 shows the images taken sequentially in the time direction, with the heads of the detection periods aligned and arranged vertically. From the beginning of the detection period Q0, the exposure for imaging the detection signal A0 is performed, and then the exposure for imaging the detection signal B0 is performed.

From the time when the phase shifts by 90 degrees from the beginning of the detection period Q1, the exposure for imaging the detection signal A90 is performed, and then the exposure for imaging the detection signal B90 is performed.

From the time when the phase shifts by 180 degrees from the beginning of the detection period Q2, the exposure for imaging the detection signal A180 is performed, and then the exposure for imaging the detection signal B180 is performed.

From the time when the phase shifts by 270 degrees from the beginning of the detection period Q3, the exposure for imaging the detection signal A270 is performed, and then the exposure for imaging the detection signal B270 is performed.

Here, when the exposure time of the detection signal B0 in the detection period Q0 and the exposure time of the detection signal A180 in the detection period Q2 are compared, it can be read that the exposure is performed at the same timing. Therefore, the detection signal A180 in the detection period Q2 can be replaced by the detection signal B0 in the detection period Q0. Similarly, the detection signal B180 in the detection period Q2 can be replaced by the detection signal A0 in the detection period Q0.

Similarly, when the exposure time of the detection signal B90 in the detection period Q1 and the exposure time of the detection signal A270 in the detection period Q3 are compared, it can be read that the exposure is performed at the same timing. Therefore, the detection signal A270 in the detection period Q3 can be replaced by the detection signal B90 in the detection period Q1. Similarly, the detection signal B270 in the detection period Q3 can be replaced by the detection signal A90 in the detection period Q1.

Therefore, as shown in FIG. 12, the detection period Q0 and the detection period Q1 are set as the detection period of one frame, and the detection signal A0 and the detection signal B0 are acquired in the detection period Q0.

The detection signal A0 acquired in this detection period Q0 can be used as the detection signal B180. Further, the detection signal B0 acquired in the detection period Q0 can be used as the detection signal A180. Therefore, in this case, it can be treated in the same manner as when the detection signal A0, the detection signal B0, the detection signal A180, and the detection signal B180 are acquired in the detection period Q0.

Further, the detection signal A90 acquired in the detection period Q1 can be used as the detection signal B270. Further, the detection signal B90 acquired in the detection period Q1 can be used as the detection signal A270. Therefore, in this case, it can be treated in the same manner as when the detection signal A90, the detection signal B90, the detection signal A270, and the detection signal B270 are acquired in the detection period Q1.

Therefore, the case of the 2-tap 2-phase described with reference to FIG. 12 can be handled in the same manner as the case of the 2-tap 4-phase described with reference to FIG.

The value I and the value Q in the equation (1) in the two-tap four-phase described with reference to FIG. 10 are expressed by the following equation (4).

The value I is obtained by subtracting the detection signal B0 from the detection signal A0, and the value Q is obtained by subtracting the detection signal B90 from the detection signal A90. Since the value I and the value Q are acquired, the phase difference θ can be calculated by the equation (1) and the distance D can be calculated by the equation (2) as in the case of the above-mentioned two-tap method.

<About 1 tap 4 phases>
The distance to a predetermined object can be determined by using one tap 51 (the configuration of the pixel 50 including one tap 51 is not shown) using irradiation light having four phase differences or by receiving light with four phase differences. A brief description will be added to the case of distance measurement (hereinafter, appropriately referred to as 1 tap and 4 phases).

FIG. 13 shows the imaging order in the time direction in the one-tap four-phase in the same manner as in FIG. In the detection period Q0, the value C0 in the above equation (1) is acquired. In the detection period Q1, the value C90 in the above equation (1) is acquired. In the detection period Q2, the value C180 in the above equation (1) is acquired. In the detection period Q3, the value C270 in the above equation (1) is acquired.

In the case of the one-tap method, the value I and the value Q in the above equation (1) are expressed as the following equation (5).

Since the value I and the value Q are acquired, the phase difference θ can be calculated by the equation (1) and the distance D can be calculated by the equation (2) as in the case of the above-mentioned two-tap method.

The present technology can be applied to the above-mentioned 2-tap 4-phase, 2-tap 2-phase, and 1-tap 4-phase.

<About flying pixels>
An erroneous detection that occurs near the edge of an object in the environment to be measured will be described. Falsely detected pixels that occur near the edges of an object are sometimes referred to as defective pixels, flying pixels, and the like.

As shown in FIGS. 14 and 15, consider a case where there are two objects (objects) in a three-dimensional environment and the positions of the two objects are measured by the distance measuring device 10. FIG. 14 is a diagram showing the positional relationship between the foreground object 101 and the background object 102 in the xz plane, and FIG. 15 is a diagram showing the positional relationship between the foreground object 101 and the background object 102 in the xy plane.

The xz plane shown in FIG. 14 is a plane when the foreground object 101, the background object 102, and the distance measuring device 10 are viewed from above, and the xy plane shown in FIG. 15 is a plane perpendicular to the xz plane. It is a surface to be located, and is a surface when the foreground object 101 and the background object 102 are viewed from the distance measuring device 10.

With reference to FIG. 14, the foreground object 101 is located on the side closer to the distance measuring device 10 and the background object 102 is located on the side far from the distance measuring device 10 when the distance measuring device 10 is used as a reference. The foreground object 101 and the background object 102 are located within the angle of view of the distance measuring device 10. The angle of view of the distance measuring device 10 is represented by a dotted line 111 and a dotted line 112 in FIG.

One side of the foreground object 101, in FIG. 14, the right side in the figure is an edge 103. Flying pixels may occur near the edge 103.

With reference to FIG. 15, the distance measuring device 10 takes a picture with the foreground object 101 and the background object 102 overlapping each other. In such a case, flying pixels may be generated on the upper side of the foreground object 101 (referred to as the edge 104) and the lower side of the foreground object 101 (referred to as the edge 105).

In this case, the flying pixel is a pixel that is detected as a pixel belonging to the edge portion of the foreground object 101, or is detected as a distance that is neither the foreground object 101 nor the background object 102.

FIG. 16 is a diagram showing a foreground object 101 and a background object 102 by pixels corresponding to the image shown in FIG. The pixel group 121 is a pixel detected from the foreground object 101, and the pixel group 122 is a pixel detected from the background object 102. Pixel 123 and pixel 124 are flying pixels, which are falsely detected pixels.

Pixels 123 and 124 are located on the edge between the foreground object 101 and the background object 102, as shown in FIG. All of these flying pixels may belong to the foreground object 101 or the background object 102, only one may belong to the foreground object 101 and the other may belong to the background object 102.

By detecting the pixel 123 and the pixel 124 as flying pixels and processing them appropriately, for example, as shown in FIG. 17, is modified. With reference to FIG. 17, pixel 123 (FIG. 16) is modified to pixel 123'belonging to pixel group 121 belonging to foreground object 101, and pixel 123 (FIG. 16) belongs to pixel group 122 belonging to background object 102. Corrected to pixel 124'.

In this way, the process of detecting defective pixels such as flying pixels will be described.

<First process related to detection of flying pixels>
The first process related to the detection of flying pixels will be described with reference to FIG. The detection of flying pixels is executed in the filter unit 16 (FIG. 1). Referring to FIG. 1 again, the filter unit 16 is supplied with a depth map and a reliability map from the signal processing unit 13. The filter unit 16 detects flying pixels from a depth map (a collection of pixels).

In step S11, the filter unit 16 sets the pixels to be processed (evaluation target) in the supplied depth map.

In step S12, it is determined whether or not the difference between the distance (depth value) of the pixels to be processed and the distance (depth value) of the surrounding pixels is equal to or greater than the threshold value. With reference to FIG. 16, the flying pixel, for example pixel 123, is located at a distance from the pixel group 121 and the pixel group 122. In other words, the pixel 123 and the pixels around the pixel 123 are distant from each other. Therefore, when the difference in the distance between the pixel 123 and the surrounding pixels (for example, the pixels in the pixel group 121) is calculated, it is assumed that the difference is equal to or more than a predetermined value.

The determination process in step S12 will be further described. A case where a directional derivative around one point corresponding to the pixel to be processed is used to detect a flying pixel will be described as an example.

In the following description, the directional derivative may be examined in multiple directions, but here, for the sake of simplicity, the vertical direction and the horizontal direction will be described as examples. However, it goes without saying that the same principle can be applied to directions other than the vertical and horizontal directions. Further, although the case where the directional derivative is used will be described here as an example, other methods can be applied to the present technology.

Here, "P" is a pixel being evaluated in the depth map, and "a" is a selected direction in the plane. In this case, da (P) is the value of the derivative in the direction "a" at pixel "P". The absolute value of the directional derivative | da (P) | and | da + π (Ρ) | are larger than the predefined threshold of the direction "a", and the signs of da (P) and da + π (Ρ) are the same. If, the pixel is detected as a flying pixel. Note that a + π indicates that the direction is opposite to that of a.

Pixels with noise can also be detected by using the directional function. If the pixel being evaluated has a depth value that is significantly different from the depth values of all pixels adjacent to the pixel being evaluated, and at least one directional derivative is greater than a predefined threshold and If at least two directional derivatives have opposite signs, the pixel is detected as a noisy pixel.

Here, the detection of the noisy pixel will also be described, but when the first process related to the detection of the flying pixel is executed, the detection of the noisy pixel described below is a process that can be omitted as appropriate. ..

For each pixel, noisy pixels and / and flying pixels can be detected in any number of directions. These directions preferably cover the unit circle, a circle with a radius of 1 pixel. Typically, a set of directions {a_i} with i = 1 to n can be used.
a_i ＝ (i-1) × π / n
Is.

The directional derivative can be estimated by a finite difference. In FIG. 19, pixel 150 corresponds to pixels "T" above pixel "P", "L" on the left, "R" on the right, and "B" below, pixel 152, pixel 154, pixel 156. , Pixel "P" being evaluated using pixel 158.

Whether the pixel 150 is a flying pixel and whether the pixel is a noisy pixel is determined in two directions: 0 ° and π / 2 (horizontal and vertical). The values of pixels 152,154,156,158 can be used to do this.

For horizontal and vertical directions, the detection of noisy pixels is
(| RP | <Th and | LP | <Th) or (| TP | <Th and | BP | <Th)
and
sign (RP) ≠ sign (PL) or sign (TP) ≠ sign (PB)
Is used. Let these equations be equation (6). In formula (6), Th represents a threshold.

In equation (6), P represents the depth value of pixel 150, T represents the depth value of pixel 152, L represents the depth value of pixel 154, and R represents the depth value of pixel 156. B represents the depth value of pixel 158. The same applies to the formula (7) shown below.

Further, in the following description, it is described as being smaller than the threshold value or larger than the threshold value, but it may be equal to or more than the threshold value and less than or equal to the threshold value.

For (| RP | <Th and | LP | <Th), the absolute value of the difference between the depth values of pixel 156 and pixel 150 located on the right side of pixel 150 (FIG. 19) is smaller than the threshold value Th, and is on the left side of pixel 150. It is an expression for determining whether the absolute value of the difference between the depth values of the pixels 154 and the pixels 150 located in is smaller than the threshold value Th.

When the judgment is made by this formula, it is also judged whether or not sign (R-P) ≠ sign (P-L). sign (RP) ≠ sign (PL) is an expression that determines whether the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 156 and the value obtained by subtracting pixel 150 from pixel 154 are the same. Is. When these two equations are satisfied, the pixel to be processed is detected as a noisy pixel.

When processing is performed focusing on the vertical direction, (| T-P | <Th and | B-P | <Th) and sign (T-P) ≠ sign (P-B) are used. (| TP | <Th and | BP | <Th) means that the absolute value of the difference between the depth values of the pixel 152 located above the pixel 150 (FIG. 19) and the depth value of the pixel 150 is smaller than the threshold Th and below the pixel 150. It is an expression for determining whether the absolute value of the difference between the depth values of the pixels 158 and the pixels 150 located on the side is smaller than the threshold value Th.

When the judgment is made by this formula, it is also judged whether or not sign (T-P) ≠ sign (P-B). Isn't sign (TP) ≠ sign (PB) the same as the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 152 and the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 158? It is an expression for determining whether or not. When these two equations are satisfied, the pixel to be processed is detected as a noisy pixel.

Flying pixel detection
(| RP |> kTh and | LP |> kTh) or (| TP |> kTh and | BP |> kTh)
and
sign (RP) = sign (PL) or sign (TP) = sign (PB)
Is used. Let these equations be equation (7). In equation (7), Th represents a threshold and k represents a predetermined weighting factor.

For (| RP |> kTh and | LP |> kTh), the absolute value of the difference between the depth values of pixel 156 and pixel 150 located on the right side of pixel 150 (FIG. 19) is multiplied by the threshold Th and the weighting factor k. It is an expression for determining whether or not the absolute value of the difference between the depth values of the pixel 154 and the pixel 150 located on the left side of the pixel 150, which is larger than the value, is larger than the value obtained by multiplying the threshold value Th and the weighting coefficient k.

When the judgment is made by this formula, it is also judged whether or not sign (R-P) = sign (P-L). Does sign (RP) = sign (PL) have the same positive or negative value as the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 156 and the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 154? It is an expression for determining whether or not. When these two equations are satisfied, the pixel to be processed is detected as a flying pixel.

When processing is performed focusing on the vertical direction, (| T-P |> kTh and | B-P |> kTh) and sign (T-P) = sign (P-B) are used. (| TP |> kTh and | BP |> kTh) is the absolute value of the difference between the depth values of pixel 152 and pixel 150 located above pixel 150 (FIG. 19) multiplied by the threshold Th and the weighting factor k. It is an expression for determining whether or not the absolute value of the difference between the depth values of the pixel 158 and the pixel 150, which is larger than the value and is located below the pixel 150, is larger than the value obtained by multiplying the threshold value Th and the weighting coefficient k.

When the judgment is made by this formula, it is also judged whether or not sign (T-P) = sign (P-B). Does sign (TP) = sign (PB) have the same positive or negative value as the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 152 and the value obtained by subtracting the depth value of pixel 150 from the depth value of pixel 158? It is an expression for determining whether or not. When these two equations are satisfied, the pixel to be processed is detected as a flying pixel.

As an alternative to the detection of flying pixels and pixels with noise, the following equation (8) can be used instead of equations (6) and (7).
| LR |> Th and | TB |> Th

For (| LR |> Th and | TB |> Th), the absolute value of the difference between the depth values of the pixel 156 located on the right side and the pixel 154 located on the left side of the pixel 150 to be evaluated is from the threshold value Th. It is an expression for determining whether or not the absolute value of the difference between the depth values of the pixel 152 located above the pixel 150 to be evaluated and the depth value of the pixel 158 located below is larger than the threshold value Th. If this formula is satisfied, the pixel being processed is detected as a flying pixel.

According to this equation (8), since the value between the two pixels surrounding the pixel being evaluated is used, the threshold value Th is higher than the threshold value Th given by the above equations (6) and (7). It may be large.

In step S12 (FIG. 18), the filter unit 16 determines whether or not the pixel to be processed is a flying pixel based on the equation (7). In step S12, a process of determining whether or not the difference between the depth values (distances) of the surrounding pixels is larger than (or more than) the threshold value is performed. Flying pixels can be detected only by the processing in step S12, but pixels that are not flying pixels may be detected as flying pixels.

For example, a plane that is close to the direction parallel to the direction vector of the distance measuring device 10 and exists at a long distance may be detected as a flying pixel. If it is detected as a flying pixel from such a plane in the process of step S12, it will be an erroneous detection. In step S12, the determination process of step S13 is performed as a mechanism for correcting even if an erroneous detection is made.

In step S12, when it is determined that the difference in the distance between the pixel to be processed and the surrounding pixels is equal to or greater than the threshold value (in other words, it is determined to be a flying pixel). , The process proceeds to step S13.

In step S13, it is determined whether or not the difference in reliability or reflectance between the pixel to be processed and the surrounding pixels is equal to or greater than the threshold value. The confidence level is a value calculated by the equation (3). The reflectance is a value calculated by the following equation (9). As described above, the reliability may be the sum of the absolute values of I and Q, for example, an expression other than the expression (3).

reflectance = confidence × depth ²・ ・ ・ (9)
The reflectance is the value obtained by multiplying the reliability by the square of the distance (depth value).

In step S13, it does not matter whether reliability is used or reflectance is used. In addition, some index may be introduced to provide a mechanism for switching between the case of using reliability and the case of using reflectance.

When the determination process in step S13 is performed using the reliability, it is performed based on the following equation (10). The reliability is calculated based on the above equation (3).

(| Rc-Pc |> Thc and | Lc-Pc |> Thc)
Or
(| Tc-Pc |> Thc and | Bc-Pc |> Thc) ・ ・ ・ (10)

In equation (10), Pc represents the reliability of pixel 150, Tc represents the reliability of pixel 152, Lc represents the reliability of pixel 154, Rc represents the reliability of pixel 156, and Bc. Represents the reliability of pixel 158.

For (| Rc-Pc |> Thc and | Lc-Pc |> Thc), the absolute value of the difference between the reliability of pixel 156 located on the right side of pixel 150 (FIG. 19) and the reliability of pixel 150 is the threshold value. It is an expression for determining whether or not the absolute value of the difference between the reliability of pixel 154, which is larger than Thc and is located on the left side of pixel 150, and the reliability of pixel 150 is larger than the threshold Thc. When this formula is satisfied, the pixel to be processed is detected (determined) as a flying pixel.

When processing is performed focusing on the vertical direction, (| Tc-Pc |> Thc and | Bc-Pc |> Thc) are used. (| Tc-Pc |> Thc and | Bc-Pc |> Thc) is the absolute value of the difference between the reliability of pixel 152 located above pixel 150 (FIG. 19) and the reliability of pixel 150, which is the threshold value Thc. It is an expression for determining whether or not the absolute value of the difference between the reliability of the pixel 158, which is larger and located below the pixel 150, and the reliability of the pixel 150 is larger than the threshold Thc. When this formula is satisfied, the pixel to be processed is detected (determined) as a flying pixel.

When the determination process in step S13 is performed using the reflectance, it is performed based on the following equation (11). The reflectance is calculated based on the above equation (9).

(| Rr-Pr |> Thr and | Lr-Pr |> Thr)
Or
(| Tr-Pr |> Thr and | Br-Pr |> Thr) ・ ・ ・ (11)

In equation (11), Pr represents the reflectance of pixel 150, Tr represents the reflectance of pixel 152, Lr represents the reflectance of pixel 154, Rr represents the reflectance of pixel 156, and Br. Represents the reflectance of pixel 158.

For (| Rr-Pr |> Thr and | Lr-Pr |> Thr), the absolute value of the difference between the reflectance of pixel 156 located on the right side of pixel 150 (FIG. 19) and the reflectance of pixel 150 is the threshold value. It is an expression for determining whether or not the absolute value of the difference between the reflectance of pixel 154, which is larger than Thr and is located on the left side of pixel 150, and the reflectance of pixel 150 is larger than the threshold Thr. When this formula is satisfied, the pixel to be processed is detected (determined) as a flying pixel.

When processing is performed focusing on the vertical direction, (| Tr-Pr |> Thr and | Br-Pr |> Thr) are used. (| Tr-Pr |> Thr and | Br-Pr |> Thr) is the absolute value of the difference between the reflectance of pixel 152 located above pixel 150 (FIG. 19) and the reflectance of pixel 150, which is the threshold value Thr. It is an expression for determining whether or not the absolute value of the difference between the reflectance of the pixel 158 located below the pixel 150 and the reflectance of the pixel 150, which is larger than the pixel 150, is larger than the threshold Thr. When this formula is satisfied, the pixel to be processed is detected (determined) as a flying pixel.

The determination process in step S13 may be a determination using reliability or a determination using reflectance. When the reliability is calculated by the signal processing unit 13 (FIG. 1) and supplied to the filter unit 16, the filter unit 16 performs determination processing using the supplied reliability. When the filter unit 16 performs the determination process using the reflectance, the reflectance is calculated using the reliability supplied from the signal processing unit 13, and the determination process is performed using the calculated reflectance.

The threshold value Thc in the formula (10) and the threshold value Thr in the formula (11) are fixed values and preset values. Although it has been described here that it is larger than the threshold value Thc (threshold value Thr), it may be determined whether or not it is equal to or higher than the threshold value Thc (threshold value Thr).

In step S13, it is determined whether or not the difference in reliability or reflectance between the pixel to be processed and the surrounding pixels is larger than the threshold value, and if it is determined that the difference is larger than the threshold value, the process proceeds to step S14. ..

In step S14, the filter unit 16 determines that the pixel to be processed (evaluation target) is a flying pixel. A pixel determined as a flying pixel is changed or deleted to a pixel belonging to the pixel group 121 of the foreground object 101 or the pixel group 122 of the background object 102, as described with reference to FIG. 17, for example. Is corrected.

On the other hand, if it is determined in step S13 that the difference in reliability or reflectance between the pixel to be processed and the surrounding pixels is equal to or less than the threshold value, the process proceeds to step S15.

In step S15, if it is determined in step S12 that it is not a flying pixel, it is detected in step S13 that it may be a flying pixel, but if it is determined that it is not a flying pixel, or in step S14. Comes when confirmed as a flying pixel.

In step S15, it is determined whether or not there are pixels that have not been processed. If it is determined in step S15 that there are pixels that have not been processed, the processing is returned to step S11, and the subsequent processing is repeated.

On the other hand, if it is determined in step S15 that there are no unprocessed pixels, the first process related to the detection of flying pixels is terminated.

In this way, in step S12, a pixel that may be a flying pixel is detected, and in step S13, a determination is made to determine that the pixel is a flying pixel. That is, the flying pixel is detected (determined) in two steps. Therefore, it is possible to reduce erroneous detection of flying pixels, and it is possible to detect flying pixels with improved accuracy.

<Second process related to the detection of flying pixels>
The first process related to the detection of flying pixels has been described by taking the case where the threshold value Thc (Equation 10) and the threshold value Thr (Equation 11) are fixed values as an example, but the second process related to the detection of flying pixels has been described. As an example, a case where the threshold value Thc (Equation 10) and the threshold value Thr (Equation 11) are variable values will be described.

FIG. 20 is a flowchart for explaining a second process related to the detection of flying pixels.

The processing of steps S31 and S32 is the same as the processing of steps S11 and S12 (FIG. 18), and whether or not the pixel to be processed is set and the difference value in the distance between that pixel and the surrounding pixels is larger than the threshold value. By determining whether or not, a pixel that may be a flying pixel is detected.

In step S33, the threshold is calculated. The method of setting the threshold value will be described later. This threshold is the threshold Thc (Equation 10) or the threshold Thr (Equation 11). When the threshold value is set in step S33, the process in step S34 is executed using the set threshold value.

In step S34, it is determined whether or not the difference in reliability or reflectance between the pixel to be processed and the surrounding pixels is larger than the threshold value. The process in step S34 is the same as that in step S13 (FIG. 18), except that the threshold value calculated in step S33 is used as the threshold value Thc (Equation 10) or the threshold value Thr (Equation 11).

The second process related to the detection of flying pixels is the same as the first process except that a threshold value is adaptively set and a determination is made to determine the flying pixel using the threshold value, and steps S34 to S34 to S36 is performed in the same manner as the processes of steps S13 to S15 (FIG. 28).

Here, a method of calculating the threshold value executed in step S33 will be described.

First method of setting the threshold value The threshold value is set to x times the average value of the reliability or reflectance in the vertical and horizontal directions. x can be set to a value such as 0.1.

When the determination in step S34 is performed by the reliability, referring to FIG. 19, the reliability of the pixel 152 in the upward direction of the pixel 150 to be processed, the reliability of the pixel 158 in the downward direction, and the reliability in the left direction. The average value of the reliability of the pixel 154 and the reliability of the pixel 156 to the right is calculated. The threshold Thc is set to x times the average value of the reliability.

When the determination in step S34 is performed by the reflectance, referring to FIG. 19, the reflectance of the pixel 152 in the upward direction of the pixel 150 to be processed, the reflectance of the pixel 158 in the downward direction, and the reflectance in the left direction. The average value of the reflectance of pixel 154 and the reflectance of pixel 156 to the right is calculated. The threshold Thr is set to x times the average value of the reflectance.

When processing is performed using pixels located in either the vertical direction or the horizontal direction as in the equation (10) and the equation (11), the threshold value is set using the pixels in that direction. You may do so. That is, although it has been described here that pixels in the vertical and horizontal directions are used, pixels in the vertical direction or pixels in the horizontal direction may be used to calculate the threshold value.

The same applies to the following setting method, and the description will be continued by taking the case of using pixels on the top, bottom, left, and right as an example, but of course, the threshold value may be calculated using the pixels located on the top, bottom, left, and right. ..

Second setting method of threshold value Set the threshold value to x times the standard deviation of the reliability or reflectance of up, down, left and right. x can be set to a value such as 0.2.

When the determination in step S34 is performed by the reliability, referring to FIG. 19, the reliability of the pixel 152 in the upward direction of the pixel 150 to be processed, the reliability of the pixel 158 in the downward direction, and the reliability in the left direction. The standard deviation of the reliability of pixel 154 and the reliability of pixel 156 to the right is calculated. The threshold Thc is set to x times the standard deviation of the reliability.

When the determination in step S34 is performed by the reflectance, referring to FIG. 19, the reflectance of the pixel 152 in the upward direction of the pixel 150 to be processed, the reflectance of the pixel 158 in the downward direction, and the reflectance in the left direction. The standard deviation of the reflectance of pixel 154 and the reflectance of pixel 156 to the right is calculated. The threshold Thr is set to x times the standard deviation of the reflectance.

Third method of setting the threshold value The threshold value is set to x times the difference between the maximum value and the minimum value of the reliability or reflectance in the vertical and horizontal directions. x can be set to a value such as 0.2.

When the determination in step S34 is performed by the reliability, referring to FIG. 19, the reliability of the pixel 152 in the upward direction of the pixel 150 to be processed, the reliability of the pixel 158 in the downward direction, and the reliability in the left direction. The maximum and minimum values of the reliability of pixel 154 and the reliability of pixel 156 to the right are detected. Then, the difference between the maximum value and the minimum value is calculated. Further, x times the difference value is set to the threshold value Thc.

When the determination in step S34 is performed by the reflectance, referring to FIG. 19, the reflectance of the pixel 152 in the upward direction of the pixel 150 to be processed, the reflectance of the pixel 158 in the downward direction, and the reflectance in the left direction. The maximum and minimum values of the reflectance of pixel 154 and the reflectance of pixel 156 to the right are detected. Then, the difference between the maximum value and the minimum value is calculated. Further, x times the difference value is set to the threshold value Thr.

Fourth method of setting the threshold value The threshold value is set to x times the reliability or reflectance of the pixel to be processed. x can be set to a value such as 0.1.

When the determination in step S34 is performed by the reliability, x times the reliability of the pixel 150 to be processed is set to the threshold value Thc, referring to FIG.

When the determination in step S34 is performed by the reflectance, x times the reflectance of the pixel 150 to be processed is set to the threshold value Thr, referring to FIG.

Fifth setting method of the threshold value The sum of the squares of the distances between the pixel to be processed and the two pixels adjacent to each other is set as the threshold value. The fifth method of setting the threshold value is applied when the determination in step S34 is performed by the reflectance.

Processed pixel 150 (FIG. 19), to set the threshold value using the pixel 152 (FIG. 19) located on top of the pixel 150, the square of the distance of a pixel 150 (a d ₁₅₀₎ (d ₁₅₀ and ²⁾ the sum of the square of the distance of a pixel 152 (a d ₁₅₂₎ (d ₁₅₂ ²⁾ is set to the threshold Thr.
Threshold Thr = (d ₁₅₀ ² ) + (d ₁₅₂ ² ) ・ ・ ・ (12)

Although the case where the pixel above the pixel to be processed is used has been described here as an example, it is of course possible to use the pixel at a position other than the upper side adjacent to the pixel to be processed.

The derivation process of the fifth setting method will be briefly described. In the following description, the distance (depth value) is d, the true value of reliability is c, the measured value of reliability is c', the true value of reflectance is r, and the measured value of reflectance is r'.

The error in reliability and the error in reflectance can be expressed by the following equation (13).

For example, the relationship between the error, the measured value, and the true value of the pixel 150 to be processed and the relationship between the adjacent pixel 152 and the error and the measured value can be represented as shown in FIG. 21.

The upper part of FIG. 21 is a relationship diagram of pixels 150 (processed objects). 'If the centrally true value r ₁₅₀ is, (r _150' measurements r ₁₅₀ of the reflectance of the pixel 150 is present between the ^{_{-d 150 2) (r 150 '}} + d 150 2).

The lower part of FIG. 21 is a relationship diagram of pixels 152 (adjacent pixels). 'If the centrally true value r ₁₅₂ is, (r _152' measurements r ₁₅₂ of the reflectance of the pixel 152 exists between -d ₁₅₂ ²⁾ from ^{_{(r 152 '+ d 152 2}} ).

It is considered that the true value r of the reflectance of the pixel 150 exists in the range A where the relationship diagram of the pixel 150 (processing target) and the relationship diagram of the pixel 152 (adjacent pixels) overlap. The range A exists when the following equation (14) is satisfied.
_{| r 150 '-r 152' |} <d 150 2 + d 152 2 ··· (14)

That is, the absolute value of the difference between the measured value of the reflectance r ₁₅₀ 'and the measured value r ₁₅₂ of the reflectance of the pixel 152' of the pixel 150, second distance d ₁₅₂ of the square and the pixel 152 of the distance d ₁₅₀ of the pixel 150 The range A exists if it is less than the sum of the powers.

When equation (14) is satisfied, it can be assumed that the true value r exists. It can be said that the true value r may not exist when the equation (14) is not satisfied. The time when the formula (14) is not satisfied is, for example, when the formula (15) is satisfied.
_{| r 150 '-r 152' |} > d 150 2 + d 152 2 ··· (15)

When the equation (15) is satisfied, it is a case where the true value r may not exist, and it can be assumed that the pixel to be processed is a defective pixel. In this case, it can be assumed that it may be a flying pixel.

Equation (15) and equation (11) are compared. Here, a part of the equation (11) (an equation when pixels in the vertical direction are used) will be described again.
(| Tr-Pr |> Thr and | Br-Pr |> Thr) ・ ・ ・ (11)

Pay attention to, for example, (| Tr-Pr |> Thr) in equation (11). Tr is a measured value of the reflectance of the pixel 152 (FIG. 19), and corresponds to the _{measured value r 152'.} Pr is a measured value of the reflectance of the pixel 150 (FIG. 19) and corresponds to the _{measured value r 150'.} Therefore, | Tr-Pr | _{is, | r 150 '-r 152'} | become. That is, the left side of the equation (11) can be treated in the same manner as the left side of the equation (15).

Therefore, the threshold value Thr on the right side of the equation (11) can be set to _{(d 150} ² + d ₁₅₂ ^{2) on the right side of the equation (15).}

As described above, as the fifth method of setting the threshold value, the sum of squares of the distances between the pixel to be processed and the two pixels adjacent to each other can be set to the threshold value Thr.

Here, as the threshold value setting method, the first to fifth setting methods have been illustrated, but they are merely examples and are not described to indicate limitation. Therefore, it is possible to set the threshold value by another method.

In this way, by detecting and determining the defective pixel by the determination process twice, it is possible to reduce the erroneous detection of the defective pixel. Therefore, the accuracy of distance measurement can be further improved.

<Example of electronic device configuration>
The distance measuring device 10 described above can be mounted on an electronic device such as a smartphone, a tablet terminal, a mobile phone, a personal computer, a game machine, a television receiver, a wearable terminal, a digital still camera, or a digital video camera.

FIG. 22 is a block diagram showing a configuration example of a smartphone as an electronic device equipped with the distance measuring device 10 as a distance measuring module.

As shown in FIG. 22, in the smartphone 201, the distance measuring module 202, the image pickup device 203, the display 204, the speaker 205, the microphone 206, the communication module 207, the sensor unit 208, the touch panel 209, and the control unit 210 are connected via the bus 211. Is connected and configured. Further, the control unit 210 has functions as an application processing unit 221 and an operation system processing unit 222 by executing a program by the CPU.

The distance measuring device 10 of FIG. 1 is applied to the distance measuring module 202. For example, the distance measuring module 202 is arranged in front of the smartphone 201, and by performing distance measurement for the user of the smartphone 201, the depth value of the surface shape of the user's face, hand, finger, etc. Can be output as.

The image pickup device 203 is arranged in front of the smartphone 201, and takes an image of the user of the smartphone 201 as a subject to acquire an image of the user. Although not shown, the image pickup device 203 may be arranged on the back surface of the smartphone 201.

The display 204 displays an operation screen for performing processing by the application processing unit 221 and the operation system processing unit 222, an image captured by the image pickup apparatus 203, and the like. The speaker 205 and the microphone 206, for example, output the voice of the other party and collect the voice of the user when making a call by the smartphone 201.

The communication module 207 communicates via the communication network. The sensor unit 208 senses speed, acceleration, proximity, etc., and the touch panel 209 acquires a touch operation by the user on the operation screen displayed on the display 204.

The application processing unit 221 performs processing for providing various services by the smartphone 201. For example, the application processing unit 221 can create a face by computer graphics that virtually reproduces the user's facial expression based on the depth supplied from the distance measuring module 202, and can perform a process of displaying the face on the display 204. Further, the application processing unit 221 can perform a process of creating, for example, three-dimensional shape data of an arbitrary three-dimensional object based on the depth supplied from the distance measuring module 202.

The operation system processing unit 222 performs processing for realizing the basic functions and operations of the smartphone 201. For example, the operation system processing unit 222 can perform a process of authenticating the user's face and unlocking the smartphone 201 based on the depth value supplied from the distance measuring module 202. Further, the operation system processing unit 222 performs, for example, a process of recognizing a user's gesture based on the depth value supplied from the distance measuring module 202, and performs a process of inputting various operations according to the gesture. Can be done.

<About recording media>
The series of processes described above can be executed by hardware or software. When a series of processes are executed by software, the programs constituting the software are installed on the computer. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 23 is a block diagram showing a configuration example of the hardware of a computer that executes the above-mentioned series of processes programmatically. In a computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are connected to each other by a bus 504. An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

The input unit 506 includes a keyboard, a mouse, a microphone, and the like. The output unit 507 includes a display, a speaker, and the like. The storage unit 508 includes a hard disk, a non-volatile memory, and the like. The communication unit 509 includes a network interface and the like. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 501 loads the program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the above-described series. Is processed.

The program executed by the computer (CPU 501) can be recorded and provided on a removable recording medium 511 as a package medium or the like, for example. Programs can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. Further, the program can be received by the communication unit 509 and installed in the storage unit 508 via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM 502 or the storage unit 508.

The program executed by the computer may be a program that is processed in chronological order according to the order described in this specification, or may be a program that is processed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

<Example of application to endoscopic surgery system>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the techniques according to the present disclosure may be applied to endoscopic surgery systems.

FIG. 24 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technique according to the present disclosure (the present technique) can be applied.

FIG. 24 shows a surgeon (doctor) 11131 performing surgery on patient 11132 on patient bed 11133 using the endoscopic surgery system 11000. As shown, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical tools 11110 such as an abdominal tube 11111 and an energy treatment tool 11112, and a support arm device 11120 that supports the endoscope 11100. , A cart 11200 equipped with various devices for endoscopic surgery.

The endoscope 11100 is composed of a lens barrel 11101 in which a region having a predetermined length from the tip is inserted into the body cavity of the patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 configured as a so-called rigid mirror having a rigid barrel 11101 is illustrated, but the endoscope 11100 may be configured as a so-called flexible mirror having a flexible barrel. Good.

An opening in which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101 to be an objective. It is irradiated toward the observation target in the body cavity of the patient 11132 through the lens. The endoscope 11100 may be a direct endoscope, a perspective mirror, or a side endoscope.

An optical system and an image sensor are provided inside the camera head 11102, and the reflected light (observation light) from the observation target is focused on the image sensor by the optical system. The observation light is photoelectrically converted by the image pickup device, and an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated. The image signal is transmitted to the camera control unit (CCU: Camera Control Unit) 11201 as RAW data.

The CCU11201 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and comprehensively controls the operations of the endoscope 11100 and the display device 11202. Further, the CCU 11201 receives an image signal from the camera head 11102, and performs various image processing on the image signal for displaying an image based on the image signal, such as development processing (demosaic processing).

The display device 11202 displays an image based on the image signal processed by the CCU 11201 under the control of the CCU 11201.

The light source device 11203 is composed of, for example, a light source such as an LED (light emitting diode), and supplies irradiation light for photographing an operating part or the like to the endoscope 11100.

The input device 11204 is an input interface to the endoscopic surgery system 11000. The user can input various information and input instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) by the endoscope 11100.

The treatment tool control device 11205 controls the drive of the energy treatment tool 11112 for cauterizing, incising, sealing a blood vessel, or the like of a tissue. The pneumoperitoneum device 11206 uses a gas in the pneumoperitoneum tube 11111 to inflate the body cavity of the patient 11132 for the purpose of securing the field of view by the endoscope 11100 and securing the work space of the operator. To send. The recorder 11207 is a device capable of recording various information related to surgery. The printer 11208 is a device capable of printing various information related to surgery in various formats such as text, images, and graphs.

The light source device 11203 that supplies the irradiation light to the endoscope 11100 when photographing the surgical site can be composed of, for example, an LED, a laser light source, or a white light source composed of a combination thereof. When a white light source is configured by combining RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high accuracy. Therefore, the light source device 11203 adjusts the white balance of the captured image. It can be carried out. Further, in this case, the laser light from each of the RGB laser light sources is irradiated to the observation target in a time-divided manner, and the drive of the image sensor of the camera head 11102 is controlled in synchronization with the irradiation timing to support each of RGB. It is also possible to capture the image in a time-divided manner. According to this method, a color image can be obtained without providing a color filter on the image sensor.

Further, the drive of the light source device 11203 may be controlled so as to change the intensity of the output light at predetermined time intervals. By controlling the drive of the image sensor of the camera head 11102 in synchronization with the timing of changing the light intensity to acquire an image in a time-divided manner and synthesizing the image, so-called high dynamic without blackout and overexposure. Range images can be generated.

Further, the light source device 11203 may be configured to be able to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, by utilizing the wavelength dependence of light absorption in body tissue to irradiate light in a narrow band as compared with the irradiation light (that is, white light) in normal observation, the surface layer of the mucous membrane. So-called narrow band imaging, in which a predetermined tissue such as a blood vessel is photographed with high contrast, is performed. Alternatively, in the special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiating with excitation light. In fluorescence observation, the body tissue is irradiated with excitation light to observe the fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and the body tissue is injected. It is possible to obtain a fluorescence image by irradiating excitation light corresponding to the fluorescence wavelength of the reagent. The light source device 11203 may be configured to be capable of supplying narrow band light and / or excitation light corresponding to such special light observation.

FIG. 25 is a block diagram showing an example of the functional configuration of the camera head 11102 and CCU11201 shown in FIG. 24.

The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera head control unit 11405. CCU11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and CCU11201 are communicatively connected to each other by a transmission cable 11400.

The lens unit 11401 is an optical system provided at a connection portion with the lens barrel 11101. The observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and incident on the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

The image sensor constituting the image pickup unit 11402 may be one (so-called single plate type) or a plurality (so-called multi-plate type). When the image pickup unit 11402 is composed of a multi-plate type, for example, each image pickup element may generate an image signal corresponding to each of RGB, and a color image may be obtained by synthesizing them. Alternatively, the image pickup unit 11402 may be configured to have a pair of image pickup elements for acquiring image signals for the right eye and the left eye corresponding to 3D (dimensional) display, respectively. The 3D display enables the operator 11131 to more accurately grasp the depth of the biological tissue in the surgical site. When the image pickup unit 11402 is composed of a multi-plate type, a plurality of lens units 11401 may be provided corresponding to each image pickup element.

Further, the imaging unit 11402 does not necessarily have to be provided on the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101 immediately after the objective lens.

The drive unit 11403 is composed of an actuator, and moves the zoom lens and the focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. As a result, the magnification and focus of the image captured by the imaging unit 11402 can be adjusted as appropriate.

The communication unit 11404 is configured by a communication device for transmitting and receiving various types of information to and from the CCU11201. The communication unit 11404 transmits the image signal obtained from the image pickup unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

Further, the communication unit 11404 receives a control signal for controlling the drive of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head control unit 11405. The control signal includes, for example, information to specify the frame rate of the captured image, information to specify the exposure value at the time of imaging, and / or information to specify the magnification and focus of the captured image, and the like. Contains information about the condition.

The imaging conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by the user, or may be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. Good. In the latter case, the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are mounted on the endoscope 11100.

The camera head control unit 11405 controls the drive of the camera head 11102 based on the control signal from the CCU 11201 received via the communication unit 11404.

The communication unit 11411 is composed of a communication device for transmitting and receiving various types of information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

Further, the communication unit 11411 transmits a control signal for controlling the driving of the camera head 11102 to the camera head 11102. Image signals and control signals can be transmitted by telecommunications, optical communication, or the like.

The image processing unit 11412 performs various image processing on the image signal which is the RAW data transmitted from the camera head 11102.

The control unit 11413 performs various controls related to the imaging of the surgical site and the like by the endoscope 11100 and the display of the captured image obtained by the imaging of the surgical site and the like. For example, the control unit 11413 generates a control signal for controlling the drive of the camera head 11102.

Further, the control unit 11413 causes the display device 11202 to display the captured image in which the surgical unit or the like is reflected, based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image by using various image recognition techniques. For example, the control unit 11413 detects the shape, color, and the like of the edge of an object included in the captured image to remove surgical tools such as forceps, a specific biological part, bleeding, and mist when using the energy treatment tool 11112. Can be recognized. When displaying the captured image on the display device 11202, the control unit 11413 may superimpose and display various surgical support information on the image of the surgical unit by using the recognition result. By superimposing and displaying the surgical support information and presenting it to the surgeon 11131, it is possible to reduce the burden on the surgeon 11131 and to allow the surgeon 11131 to proceed with the surgery reliably.

The transmission cable 11400 connecting the camera head 11102 and CCU11201 is an electric signal cable corresponding to electric signal communication, an optical fiber corresponding to optical communication, or a composite cable thereof.

Here, in the illustrated example, the communication is performed by wire using the transmission cable 11400, but the communication between the camera head 11102 and the CCU11201 may be performed wirelessly.

<Example of application to mobiles>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on a moving body of any kind such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 26 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technique according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 26, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (Interface) 12053 are shown.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating braking force of the vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives inputs of these radio waves or signals and controls a vehicle door lock device, a power window device, a lamp, and the like.

The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image pickup unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or a character on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects information in the vehicle. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (Advanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generating device, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040. It is possible to perform coordinated control for the purpose of automatic driving that runs autonomously without depending on the operation.

Further, the microprocessor 12051 can output a control command to the body system control unit 12030 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio-image output unit 12052 transmits an output signal of at least one of audio and an image to an output device capable of visually or audibly notifying the passenger of the vehicle or the outside of the vehicle. In the example of FIG. 26, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 27 is a diagram showing an example of the installation position of the imaging unit 12031.

In FIG. 27, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as, for example, the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The imaging units 12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The imaging unit 12105 provided on the upper part of the windshield in the vehicle interior is mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 27 shows an example of the photographing range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the imaging units 12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining, it is possible to extract as the preceding vehicle a three-dimensional object that is the closest three-dimensional object on the traveling path of the vehicle 12100 and that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, 0 km / h or more). it can. Further, the microprocessor 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform coordinated control for the purpose of automatic driving or the like in which the vehicle travels autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, electric poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microprocessor 12051 identifies obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

As used herein, the term "system" refers to an entire device composed of a plurality of devices.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

The present technology can also have the following configurations.
(1)
A first determination unit that determines whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When the first determination unit determines that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reliability of the first pixel and the second pixel. It is provided with a second determination unit for determining whether or not the difference in reliability of is larger than the second threshold value.
When the second determination unit determines that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel is a defective pixel. A distance measuring device that determines that.
(2)
The distance measuring device according to (1) above, wherein the second threshold value is a fixed value or a variable value.
(3)
The distance measuring device according to (1) or (2), wherein the second threshold value is set to a value obtained by multiplying the average value of the reliabilitys of a plurality of the second pixels by a predetermined value.
(4)
The distance measuring device according to (1) or (2), wherein the second threshold value is set to a value obtained by multiplying the standard deviation of the reliability of a plurality of the second pixels by a predetermined value.
(5)
The second threshold value is set to a value obtained by multiplying the difference between the maximum value and the minimum value of the reliability of the plurality of the second pixels by a predetermined value, according to the above (1) or (2). Distance measuring device.
(6)
The distance measuring device according to (1) or (2), wherein the second threshold value is set to a value obtained by multiplying the reliability of the first pixel by a predetermined value.
(7)
A first determination unit that determines whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When the first determination unit determines that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reflectance of the first pixel and the second pixel It is provided with a second determination unit for determining whether or not the difference in reflectance of is larger than the second threshold value.
When the second determination unit determines that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel is a defective pixel. A distance measuring device that is determined to be.
(8)
The distance measuring device according to (7) above, wherein the reflectance is a value obtained by multiplying the reliability by the square of the depth value.
(9)
The distance measuring device according to (7) or (8), wherein the second threshold value is a fixed value or a variable value.
(10)
The distance measurement according to any one of (7) to (9) above, wherein the second threshold value is set to a value obtained by multiplying the average value of the reflectances of the plurality of the second pixels by a predetermined value. apparatus.
(11)
The distance measurement according to any one of (7) to (9) above, wherein the second threshold value is set to a value obtained by multiplying the standard deviation of the reflectances of the plurality of the second pixels by a predetermined value. apparatus.
(12)
The second threshold value is set to a value obtained by multiplying the difference between the maximum value and the minimum value of the reflectance of the plurality of the second pixels by a predetermined value. The distance measuring device described in.
(13)
The distance measuring device according to any one of (7) to (9), wherein the second threshold value is set to a value obtained by multiplying the reflectance of the first pixel by a predetermined value.
(14)
The second threshold value is set in any one of (7) to (9) above, which is set to the sum of the square of the depth value of the first pixel and the square of the depth value of the second pixel. Distance measuring device.
(15)
The distance measuring device that measures the distance is
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reliability of the first pixel and the reliability of the second pixel is the second. Determine if it is greater than the threshold of
When it is determined that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel. ..
(16)
The distance measuring device that measures the distance is
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reflectance of the first pixel and the reflectance of the second pixel is the second. Determine if it is greater than the threshold of
When it is determined that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel. ..
(17)
On the computer
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reliability of the first pixel and the reliability of the second pixel is the second. Determine if it is greater than the threshold of
If it is determined that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel includes a step of determining that the first pixel is a defective pixel. A program to execute processing.
(18)
The computer
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reflectance of the first pixel and the reflectance of the second pixel is the second. Determine if it is greater than the threshold of
When it is determined that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel includes a step of determining that the first pixel is a defective pixel. A program for executing processing.

10 Distance measuring device, 11 lens, 12 light receiving unit, 13 signal processing unit, 14 light emitting unit, 15 light emitting control unit, 16 filter unit, 31 photodiode, 41 pixel array unit, 42 vertical drive unit, 43 column processing unit, 44 Horizontal drive unit, 45 system control unit, 46 pixel drive line, 47 vertical signal line, 48 signal processing unit, 50 pixels, 51 taps, 61 photodiode, 62 transfer transistor, 63 FD unit, 64 reset transistor, 65 amplification transistor, 66 Selective transistor, 101 Foreground object, 102 Background object, 103, 104, 105 Edge, 111, 112 Dot line, 121 pixel group, 122 pixel group, 123, 124 pixels, 150, 152, 154, 156, 158 pixels

Claims (18)

A first determination unit that determines whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When the first determination unit determines that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reliability of the first pixel and the second pixel. It is provided with a second determination unit for determining whether or not the difference in reliability of is larger than the second threshold value.
When the second determination unit determines that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel is a defective pixel. A distance measuring device that determines that. The distance measuring device according to claim 1, wherein the second threshold value is a fixed value or a variable value. The distance measuring device according to claim 1, wherein the second threshold value is set to a value obtained by multiplying the average value of the reliabilitys of the plurality of the second pixels by a predetermined value. The distance measuring device according to claim 1, wherein the second threshold value is set to a value obtained by multiplying the standard deviation of the reliability of the plurality of the second pixels by a predetermined value. The distance measuring device according to claim 1, wherein the second threshold value is set to a value obtained by multiplying the difference between the maximum value and the minimum value of the reliability of the plurality of the second pixels by a predetermined value. The distance measuring device according to claim 1, wherein the second threshold value is set to a value obtained by multiplying the reliability of the first pixel by a predetermined value. A first determination unit that determines whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When the first determination unit determines that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the reflectance of the first pixel and the second pixel It is provided with a second determination unit for determining whether or not the difference in reflectance of is larger than the second threshold value.
When the second determination unit determines that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel is a defective pixel. A distance measuring device that is determined to be. The distance measuring device according to claim 7, wherein the reflectance is a value obtained by multiplying the reliability by the square of the depth value. The distance measuring device according to claim 7, wherein the second threshold value is a fixed value or a variable value. The distance measuring device according to claim 7, wherein the second threshold value is set to a value obtained by multiplying the average value of the reflectances of the plurality of the second pixels by a predetermined value. The distance measuring device according to claim 7, wherein the second threshold value is set to a value obtained by multiplying the standard deviation of the reflectances of the plurality of the second pixels by a predetermined value. The distance measuring device according to claim 7, wherein the second threshold value is set to a value obtained by multiplying the difference between the maximum value and the minimum value of the reflectance of the plurality of the second pixels by a predetermined value. The distance measuring device according to claim 7, wherein the second threshold value is set to a value obtained by multiplying the reflectance of the first pixel by a predetermined value. The distance measuring device according to claim 7, wherein the second threshold value is set to the sum of the square of the depth value of the first pixel and the square of the depth value of the second pixel. The distance measuring device that measures the distance is
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reliability of the first pixel and the reliability of the second pixel is the second. Determine if it is greater than the threshold of
When it is determined that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel. .. The distance measuring device that measures the distance is
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reflectance of the first pixel and the reflectance of the second pixel is the second. Determine if it is greater than the threshold of
When it is determined that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel is determined to be a defective pixel. .. On the computer
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reliability of the first pixel and the reliability of the second pixel is the second. Determine if it is greater than the threshold of
If it is determined that the difference between the reliability of the first pixel and the reliability of the second pixel is larger than the second threshold value, the first pixel includes a step of determining that the first pixel is a defective pixel. A program to execute processing. The computer
It is determined whether or not the difference between the depth values of the first pixel in the depth map and the depth value of the second pixel adjacent to the first pixel is larger than the first threshold value.
When it is determined that the difference between the distances between the first pixel and the second pixel is larger than the first threshold value, the difference between the reflectance of the first pixel and the reflectance of the second pixel is the second. Determine if it is greater than the threshold of
When it is determined that the difference between the reflectance of the first pixel and the reflectance of the second pixel is larger than the second threshold value, the first pixel includes a step of determining that the first pixel is a defective pixel. A program for executing processing.

Images