JP2020038163A - Distance image generation device, and distance image generation method - Google Patents

Abstract

To provide a distance measurement technique capable of enhancing the accuracy of a distance image or distance information.SOLUTION: A distance image generation device is a device for generating a distance image having a plurality of pixels: (20, 21) calculates distance to each pixel and the intensity of reflected light by receiving the reflected light when modulation light is emitted to an object; (22) calculates the intensity of disturbance light of each pixel by receiving the disturbance light when the modulation light is not emitted to the object; (23) sets a threshold determined in accordance with the intensity of the disturbance light in each pixel; and (24) compares the threshold with the intensity of the reflected light in each pixel to generate a distance image by correcting corresponding distance about a pixel in which the intensity of the reflected light falls below the threshold.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for generating a distance image.

  As a technique for generating a distance image, for example, a technique using a TOF (Time Of Flight) method is known. The TOF method is a method of irradiating a target object with light modulated by a constant frequency, detecting reflected light, and measuring a distance based on a time until the reflected light returns. When distance measurement is performed using reflected light as in the TOF method, disturbance light due to sunlight or illumination is detected together with the modulated light. Therefore, the component of the modulated light from the received light is used to suppress the influence thereof. It is desired to take out only. Techniques for suppressing the effects of sunlight and disturbance light are described in, for example, Japanese Patent No. 5271463 (Patent Document 1) and Japanese Patent No. 5452200 (Patent Document 2).

  In the inventions described in the above-mentioned known documents, the effects of light shot noise, dark current noise, and kTC noise have not been particularly considered. Here, the light shot noise is noise generated when light is incident on the light receiving unit, and affects the distance measurement accuracy when the intensity of disturbance light is extremely large. Further, dark current noise and kTC noise are noises generated even when no light is received, and affect the distance measurement accuracy when the intensity of reflected light due to modulated light is small.

  Regarding the above points, for example, Japanese Patent Application Laid-Open No. 2012-154719 (Patent Document 3) focuses on the fact that the intensity of reflected light is inversely proportional to the square of distance and proportional to the reflectance of an object. A technique is described in which the threshold value of the amount of received light is changed according to the threshold value. Specifically, for example, the amount of light received when an object with a reflectance of 3% is photographed and the amount of light received when an object with a reflectivity of 80% is photographed are plotted on the horizontal axis as a distance. An object is recognized as an object only when the amount of light falls within a range sandwiched between two characteristic lines, and is recognized as not an object in other cases. According to this principle, if the characteristic line is lower than the characteristic line (lower threshold) in the case of the reflectance of 3%, it is regarded as erroneously detected due to the influence of kTC noise or the like, and exceeds the characteristic line (upper threshold) in the case of the reflectivity of 80%. In this case, it can be said that this is equivalent to that an erroneous detection is considered due to the influence of light shot noise or the like.

  However, when the above technique is applied, there is room for improvement in that the distance and reflectivity of an object that can be measured are limited. Further, since there is no distinction between disturbance light and reflected light, threshold processing is insufficient, and there is still room for improvement in that respect.

Japanese Patent No. 5271643 Japanese Patent No. 5452200 JP 2012-154719 A

  A specific mode according to the present invention has an object to provide a distance measurement technique capable of improving the accuracy of a distance image or distance information.

[1] A distance image generating device according to one embodiment of the present invention is a device for (a) generating a distance image having a plurality of pixels, and (b) generating a distance image when an object is irradiated with modulated light. The distance of each pixel and the intensity of the reflected light are obtained by receiving the reflected light, and (c) the disturbance light when the modulated light is not applied to the object is received by the pixel. Determining the intensity of the disturbance light, (d) setting a threshold value determined according to the intensity of the disturbance light for each of the pixels, and (e) comparing the threshold value with the intensity of the reflected light for each of the pixels. A distance image generation device that generates the distance image by correcting the distance corresponding to a pixel whose reflected light intensity is lower than the threshold.
[2] The distance image generation method according to one aspect of the present invention is a method for (a) generating a distance image having a plurality of pixels, and (b) generating a distance image when a target is irradiated with modulated light. Calculating the distance of each pixel and the intensity of the reflected light by receiving the reflected light; and (c) receiving the disturbance light when the object is not irradiated with the modulated light. Determining the intensity of the disturbance light, (d) setting a threshold determined according to the intensity of the disturbance light for each of the pixels, and (e) determining the threshold and the reflection light for each of the pixels. Comparing the intensities and correcting the distance corresponding to the pixels for which the intensity of the reflected light is lower than the threshold to generate the distance image.

  According to the above configuration, a distance measurement technique capable of improving the accuracy of the distance image or the distance information can be obtained.

FIG. 1 is a block diagram illustrating a configuration of a range image generation device according to an embodiment. FIG. 2 is a diagram for describing a configuration of a light receiving element provided in the light receiving unit. FIG. 3 is a diagram for explaining the contents of the threshold function. FIG. 4 is a diagram for describing a specific example of the threshold function. FIG. 5 is a diagram for explaining an occurrence probability distribution of the noise intensity I _noise . FIG. 6 is a flowchart illustrating an operation procedure of the range image generation device.

  FIG. 1 is a block diagram illustrating a configuration of a range image generation device according to an embodiment. The illustrated depth image generation device 1 irradiates the target object 2 with modulated light using the TOF method, and communicates with the target object 2 based on reflected light (hereinafter, referred to as “modulated reflected light”) generated by the modulated light. Is to generate a distance image indicating the distance between. The distance image referred to here is an image of information indicating a distance between the object 2 and the surface. This distance image (distance information) is output to a higher-level device (not shown). The distance image generation device 1 includes a light source unit 10, a light receiving unit 11, a distance measurement mode control unit 12, a control unit 13, and a memory 14. Note that the distance measurement mode control unit 12 and the control unit 13 in the present embodiment correspond to the “control unit” in the present invention.

  The light source unit 10 includes, for example, a plurality of semiconductor light emitting elements (LEDs), and generates modulated light for irradiating the object 2. This modulated light is modulated at a constant frequency, for example, a sine wave or a half-wave rectified wave.

  The light receiving section 11 receives reflected modulated light or disturbance light from the object 2 and generates an electric signal corresponding to the light intensity. The light receiving unit 11 includes, for example, a plurality of light receiving elements arranged in a matrix.

  The ranging mode control unit 12 controls the light source unit 10 and the light receiving unit 11 to operate synchronously in order to perform measurement in a plurality of different modes. The distance measurement mode control unit 12 is configured using, for example, an FPGA (Field-Programmable Gate Array).

  The control unit 13 controls the overall operation of the distance image generation device 1 and generates a distance image based on the electric signal generated by the light receiving unit 11. The control unit 13 is realized by executing a predetermined operation program in a computer system including, for example, a CPU, a ROM, a RAM, and the like. The control unit 13 includes a distance calculation unit 20, a reflected light intensity calculation unit 21, a disturbance light intensity calculation unit 22, a threshold setting unit 23, and a distance image generation unit 24 as functional blocks configured by executing an operation program. It is configured.

  The memory 14 is for storing information relating to a threshold function required for generating a distance image in the control unit 13. As the memory 14, for example, a nonvolatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) is used. The details of the threshold function stored in the memory 14 will be described later.

  The distance calculation unit 20 obtains a distance for each pixel included in the distance image based on reflected light received by the plurality of light receiving elements of the light receiving unit 11 when the target object 2 is irradiated with the modulated light.

  The reflected light intensity calculation unit 21 calculates the reflected light of each pixel included in the distance image based on the reflected light received by the plurality of light receiving elements of the light receiving unit 11 when the target object 2 is irradiated with the modulated light. Find the strength.

  The disturbance light intensity calculation unit 22 calculates the intensity of the disturbance light for each pixel of the range image based on the disturbance light received by the plurality of light receiving elements of the light receiving unit 11 when the target object 2 is not irradiated with the modulated light. Ask.

  The threshold setting unit 23 uses a threshold function stored in the memory 14 to set a threshold determined according to the intensity of disturbance light for each pixel of the distance image.

  The distance image generation unit 24 compares the threshold value set by the threshold value setting unit 23 with the intensity of the reflected light for each pixel, and corrects the distance corresponding to the pixel for the pixel whose intensity of the reflected light is lower than the threshold value. Then, for a pixel whose reflected light intensity is equal to or greater than the threshold, a distance image is generated using the distance corresponding to the pixel as it is.

  FIG. 2 is a diagram for describing a configuration of a light receiving element provided in the light receiving unit. As described above, the light receiving unit 11 includes a plurality of light receiving elements. Here, the configuration of one light receiving element will be described, but the other light receiving elements have the same configuration. As shown in FIGS. 2A and 2B, the light receiving element 30 includes two photodiodes 31A and 31B. Each of the photodiodes 31A and 31B generates an electric charge according to the intensity of incident light by a photovoltaic effect. The photodiode 31A has two transfer gates G0 and G180 for transferring charges. Similarly, the photodiode 31B has two transfer gates G90 and G270 for transferring charges.

  When a distance image is generated, it is necessary for each light receiving element 30 to determine the light intensity I of the reflected modulated light (hereinafter, sometimes simply referred to as “light intensity I”). In this case, the ranging mode control unit 12 controls the light emitting unit 10 to emit the modulated light, and performs synchronous control so that the reflected light is received by the light receiving unit 11 in synchronization with the modulated light.

  At this time, the reflected light intensity calculation unit 21 of the control unit 13 integrates the output from the transfer gate G0 of the photodiode 31A between the modulated light phase of 0 ° and 180 °. This integrated value is defined as S0. Further, the reflected light intensity calculation unit 21 integrates the output from the transfer gate G180 of the photodiode 31A within the phase of the modulated light from 180 ° to 360 °. This integral value is defined as S180. Further, the reflected light intensity calculation unit 21 integrates the output from the transfer gate G90 of the photodiode 31B between the modulated light phase of 90 ° and 270 °. This integrated value is defined as S90. In addition, the reflected light intensity calculation unit 21 integrates the output from the transfer gate G270 of the photodiode 31B in the phase of the modulated light from 270 ° to 540 °. This integrated value is defined as S270.

The reflected light intensity calculation unit 21 calculates the light intensity I of the reflected modulated light using the integrated value as described below. Here, the light intensity of the reflected modulated light from which the influence of disturbance light has been removed is obtained.
I = {(S0-S180) ² + (S90-S270) ² }

Further, the distance calculation unit 20 of the control unit 13 obtains the phase difference Δφ between the modulated light and the reflected modulated light using the above-described integral value as follows.
Δφ = arctan {(S90-S270) / (S0-S180)}

Then, the distance calculation unit 20 obtains a distance L between the pixel corresponding to each light receiving element 30 and the object 2 as follows. Note that c is the speed of light and f is the modulation frequency.
L = Δφ × (c / 4πf)

  On the other hand, when an image indicating the intensity of disturbance light (hereinafter, referred to as “disturbance light image”) is generated, the light intensity B of disturbance light (hereinafter, simply referred to as “light intensity B”) is generated in each light receiving element 30. In some cases). In this case, the ranging mode control unit 12 performs control so that the modulated light is not emitted from the light emitting unit 10 (light quantity 0), and the light receiving unit 11 receives the disturbance light. In addition, the distance measurement mode control unit 12 turns off one of the two transfer gates provided in each of the photodiodes 31A and 31B of the light receiving element 30, and enables the charge to be read only from the other gate. For example, as shown in FIG. 2B, the ranging mode control unit 12 turns on the transfer gates G0 and G90 and turns off the transfer gates G180 and G270.

  At this time, the disturbance light intensity calculation unit 22 of the control unit 13 integrates the output from the transfer gate G0 of the photodiode 31A over a certain period. This integral value is defined as B1. Further, the disturbance light intensity calculator 22 integrates the output from the transfer gate G90 of the photodiode 31B for a certain period. This integral value is defined as B2.

The disturbance light intensity calculation unit 22 obtains the light intensity B of the disturbance light as described below using the above integrated value.
B = B1 + B2

  By using the light intensity I of the reflected modulated light and the light intensity B of the disturbance light obtained as described above and a predetermined threshold function, each light receiving element 30 of the light receiving unit 11 is based on the output of the light receiving element 30. It is possible to determine whether or not the information on the distance L obtained for each pixel can be adopted as appropriate. Hereinafter, the threshold function will be described in detail.

FIG. 3 is a diagram for explaining the contents of the threshold function. In FIG. 3, the horizontal axis corresponds to the light intensity B of the disturbance light, and the vertical axis corresponds to the light intensity I of the reflected modulated light. In the light receiving element 30 shown in FIG. 2, the amount of charge output from each transfer gate includes kTC noise and dark current noise. When the light intensity of the reflected modulated light is very small, the charge amount of each transfer gate fluctuates independently due to kTC noise or dark current noise, which affects the measured value of the phase difference Δφ. On the other hand, when the light intensity of the disturbance light is large, the light shot noise affects the measured value of the phase difference Δφ. Therefore, even when the modulated light and ambient light is not present at all, the noise intensity I ₀ due to the noise is output from the light receiving element 30. The noise intensity I ₀ takes a random value for each imaging frame. When disturbance light is present, the noise intensity increases under the influence of light shot noise.

Here, it is generally known that the light shot noise generated in the photodiode increases in proportion to the square root of the light intensity B of the disturbance light. Based on this, the largest noise intensity I _th, which can occur when reflected modulated light is 0 can be expressed as follows using the light intensity B of the ambient light. However, a is a _{constant, I 0max} is the maximum possible value of noise intensity _{I 0} occurring at random.
I _th = I _0max + a√B

The curve shown in FIG. 3 shows the threshold function represented by the above relational expression. In the range image generation device 1 of the present embodiment, the threshold setting unit 23 of the control unit 13 inputs the light intensity B of the disturbance light to the threshold function for each imaging frame to thereby obtain the noise intensity I _th (threshold) Is determined dynamically. Then, distance image generation unit 24 of the control unit 13, when the light intensity I of the reflected modulated light is greater than this noise intensity I _th, the information obtained distance L based on the output of the corresponding light receiving element 30 Is appropriate and used. The reflection when the light intensity I of the modulated light is below the noise intensity I _th determines that inappropriate information obtained distance L based on the output of the corresponding light receiving element 30 is appropriately corrected I do.

  FIG. 4 is a diagram for describing a specific example of the threshold function. Also in FIG. 4, the horizontal axis corresponds to the intensity B of the disturbance light, and the vertical axis corresponds to the light intensity I of the reflected modulated light. The threshold function shown in FIG. 4 was obtained as follows. In a state where the modulated light is not emitted from the light source 10 (light intensity = 0), the cloudy sky is imaged, and the light intensity I of the reflected modulated light detected at 5,670 pixels (the light receiving elements 30) and the disturbance light The relationship with the light intensity B was plotted. When an image of a cloudy sky is taken, the light intensity of the disturbance light differs depending on the pixel. Here, each is represented by the number of electrons (charge amount), the unit on the horizontal axis is the number of electrons generated in the photodiode per 100 ns, and the unit on the vertical axis is the number of electrons generated per 100 ns at the frequency of 5 MHz. This is the number of electrons obtained when the data is distributed to each of the four transfer gates. It can be understood that the light intensity I of the reflected modulated light is obtained as a value larger than 0 even though the light intensity of the modulated light is set to 0 and a cloudy sky that can be regarded as an infinity point is photographed. Also, the light intensity I of the reflected modulated light increases as the light intensity B of the disturbance light increases. This phenomenon can be explained by kTC noise, dark current noise, and light shot noise as described above.

The threshold function when I _th = 0.223 + 0.021 ° B using each measurement point in the figure is shown by a characteristic line in the figure. Using this characteristic line noise intensity I _th indicated by a threshold, for the pixels light intensity I of the modulated light is obtained as the following I _th (the light receiving element 30), the incorrect information of the distance L at that pixel (Distance measurement impossible).

Here, a method of determining the threshold function will be considered. _Assuming that the noise intensity when the light intensity of the modulated light is 0 and the light intensity B of the disturbance light is constant is I _noise , I _noise takes a random value for each imaging frame, and its distribution is a normal distribution. FIG. 5 shows an I _noise occurrence probability distribution when the average value μ = 0.5 and the standard deviation σ = 0.05. This indicates the probability of the amount of noise generated for each imaging frame when B ≒ 500 in the data shown in FIG.

  Ideally, all light intensities I must be judged to be inappropriate (distance measurement is impossible). However, if the threshold value is too large, the distance L when the light intensity of the reflected modulated light is slightly present is increased. Is also eliminated. For this reason, the threshold function for determining the threshold can be appropriately adjusted according to the purpose of use of the distance image generation device 1 and the like.

For example, when the light intensity of the modulated light is 0 and B is 500, if the threshold function is determined so as to satisfy I _th = μ + 3σ, the probability that the determination of improperness (distance measurement is impossible) can be accurately performed is as follows. It is about 99.865%. Similarly, when the threshold function is determined so as to satisfy I _th = μ + 4σ, the probability that the determination of inappropriateness (distance measurement is impossible) can be made accurately is about 99.99715%. Actually, when B = 500, a threshold function was determined so as to satisfy I _th = μ + 4σ, and a distance image taken under an environment in which the light intensity B of the disturbance light varied depending on the pixel (the light receiving element 30) was verified. It was confirmed that information of pixels that could not be measured (for example, an area without an object such as the sky) was accurately excluded, and that an object having a low reflectance could be appropriately measured.

  FIG. 6 is a flowchart illustrating an operation procedure of the range image generation device. Note that the processing order may be changed as long as no inconsistency occurs in the processing, and the processing order is not limited to the illustrated processing order.

  The ranging mode control unit 12 controls the light source 10 and the light receiving unit 11 in a ranging mode for measuring the light intensity I of the reflected modulated light. The reflected light intensity calculation unit 21 of the control unit 13 acquires the integrated values S0, S90, S180, and S270 corresponding to the output of the transfer gate of each light receiving element 30 (Step S11).

  Next, the distance measurement mode control unit 12 controls the light source 10 and the light receiving unit 11 in a distance measurement mode for measuring the light intensity B of the disturbance light intensity. The disturbance light intensity calculation unit 22 of the control unit 13 acquires the integral values B1 and B2 corresponding to the output of the transfer gate of each light receiving element 30 (Step S12).

  Next, the reflected light intensity calculator 21 calculates the light intensity I of the reflected modulated light (Step S13). Further, the disturbance light intensity calculator 22 calculates the light intensity B of the disturbance light (Step S14). Further, the distance calculation unit 20 calculates the distance L for each pixel (Step S15).

Next, the threshold setting unit 23 uses the threshold function read out from the memory 14, and calculates the noise intensity I _th is a threshold based on the light intensity B of the ambient light for each pixel (step S16). Then, distance image generation unit 24 determines whether or larger or not than the noise intensity I _th is the light intensity I is the threshold of the reflected modulated light obtained in step S13 (step S17).

If the light intensity I of the reflected modulated light is greater than the noise intensity I _th threshold (Step S17; YES), the distance image generation unit 24 holds the distance L of the pixel as appropriate (step S18) . On the other hand, when the light intensity I of the reflected modulated light is below the noise intensity I _th threshold (Step S17; NO), the distance image generation unit 24, an error processing the distance L of the pixel as objectionable ( Step S19). Specifically, for example, a correction process for outputting a design maximum value as the distance L of the pixel is performed, or a correction process for discarding information on the distance L is performed.

  The distance image generation unit 24 determines whether or not processing has been completed for all pixels (step S20). If the processing has not been completed (step S20; NO), the process returns to step S17, and thereafter. Is repeated. When the processing is completed (step S20; YES), the distance image generation unit 24 generates a distance image using the information of the distance L of each pixel, and outputs the generated distance image to the host device (step S21).

  According to the above-described embodiment, it is possible to obtain a distance measurement technique capable of improving the accuracy of a distance image or distance information.

  It should be noted that the present invention is not limited to the contents of the above-described embodiment, and can be implemented with various modifications within the scope of the present invention. For example, in the above-described embodiment, the case where the present invention is applied to the distance image generating apparatus using the TOF method has been exemplified. However, the present invention is not limited to this, and distance measurement is performed using light emitted from a light source (light emitting unit). The present invention can be applied to all the range image generating apparatuses to be performed. There is no limitation on the waveform of light emitted from the light source. Further, in the above-described embodiment, the case where the light receiving unit having two transfer gates is used for each of the two photodiodes has been exemplified. However, as long as the light intensity of the reflected modulated light and the light intensity of the disturbance light can be obtained. In, the configuration of the light receiving unit is not limited.

  1: distance image generation device, 2: target object, 10: light source unit, 11: light receiving unit, 12: distance measurement mode control unit, 13: control unit, 14: memory, 20: distance calculation unit, 21: reflected light intensity Calculation unit, 22: disturbance light intensity calculation unit, 23: threshold setting unit, 24: distance image generation unit, 30: light receiving element, 31A, 31B: photodiode, G0, G90, G180, G270: transfer gate

Claims (6)

An apparatus for generating a range image having a plurality of pixels,
By receiving the reflected light when the modulated light is applied to the object, the distance of each pixel and the intensity of the reflected light are obtained,
By obtaining the intensity of the disturbance light for each pixel by receiving the disturbance light when the modulated light is not irradiated to the object,
A threshold determined according to the intensity of the disturbance light is set for each of the pixels,
Comparing the threshold and the intensity of the reflected light for each of the pixels, and generating the distance image by correcting the corresponding distance for a pixel whose intensity of the reflected light is less than the threshold,
Range image generation device. An apparatus for generating a range image having a plurality of pixels,
A plurality of light receiving elements corresponding to each of the plurality of pixels;
A control unit that generates the distance image based on outputs of the plurality of light receiving elements,
The control unit includes:
Determine the distance of each pixel and the intensity of the reflected light based on the reflected light received by the plurality of light receiving elements when the object is irradiated with the modulated light,
Determine the intensity of the disturbance light for each pixel based on the disturbance light received by the plurality of light receiving elements when the modulated light is not irradiated to the object,
A threshold determined according to the intensity of the disturbance light is set for each of the pixels,
For each of the pixels, the threshold value is compared with the intensity of the reflected light, and the intensity of the reflected light is lower than the threshold value.
Range image generation device. An apparatus for generating a range image having a plurality of pixels,
A plurality of light receiving elements corresponding to each of the plurality of pixels;
A control unit that generates the distance image based on outputs of the plurality of light receiving elements,
The control unit includes:
A distance calculation unit that calculates a distance for each pixel based on reflected light received by the plurality of light receiving elements when the target object is irradiated with the modulated light,
A reflected light intensity calculation unit that calculates the intensity of the reflected light for each pixel based on the reflected light received by the plurality of light receiving elements when the modulated light is applied to the object,
A disturbance light intensity calculation unit that calculates the intensity of the disturbance light for each of the pixels based on the disturbance light received by the plurality of light receiving elements when the modulated light is not applied to the object,
A threshold setting unit that sets a threshold determined according to the intensity of the disturbance light for each of the pixels,
For each pixel, the threshold value is compared with the intensity of the reflected light, and the intensity of the reflected light is less than the threshold value For pixels whose intensity is less than the threshold value, a corresponding distance is corrected and the distance image generation unit generates the distance image. ,
A distance image generation device comprising: The threshold is determined in proportion to the square root of the intensity of the reflected light,
The distance image generation device according to claim 1. The correction of the distance is performed by a process of discarding the information of the distance or a process of overwriting the information of the distance with an arbitrary value.
The distance image generation device according to claim 1. A method for generating a range image having a plurality of pixels, comprising:
Obtaining the distance of each pixel and the intensity of the reflected light by receiving the reflected light when the target object is irradiated with the modulated light,
Obtaining the intensity of the disturbance light for each pixel by receiving disturbance light when the modulated light is not radiated to the object;
Setting a threshold determined according to the intensity of the disturbance light for each of the pixels,
Comparing the threshold and the intensity of the reflected light for each of the pixels, and correcting the corresponding distance for a pixel whose intensity of the reflected light is less than the threshold to generate the distance image,
And a distance image generation method.

Images