KR102351152B1 - Method and apparatus for correcting depth information error due to multiple reflective path of three dimensional camera using time of flight sensor - Google Patents

Abstract

The present invention relates to a method for correcting a depth information error due to a multi-reflective path of a three-dimensional (3D) camera using a time of flight (ToF) sensor and an apparatus thereof. According to the present invention, the method comprises: a pixel information acquisition step of acquiring, by a pixel information acquisition unit, light intensity values and depth values of all pixels included in an image captured by a ToF sensor; a pixel classification step of comparing, by a pixel classification unit, a value obtained by multiplying the reciprocal of the squared value of the light intensity value and depth value by a threshold value corresponding to reflectivity and a scale value for adjusting the threshold value to classify all pixels into a normal pixel in which a depth information error does not occur and a noise pixel in which the depth information error occurs; a noise removing step of removing, by a noise removing unit, noise of the noise pixel; and a depth information output step of synthesizing, by a depth information output unit, a depth value of the normal pixel acquired by the depth information acquisition unit and a depth value of the noise pixel from which the noise is removed by the noise removing unit to output depth information of all pixels. Accordingly, the method can effectively remove a depth information error caused by a multi-reflection path in a 3D camera using a ToF sensor.

Description

Method and apparatus for correcting depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor

The present invention relates to a method and apparatus for correcting a depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor. More specifically, the present invention provides a method and apparatus for correcting depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor, which can effectively remove a depth information error caused by multiple reflection paths in a three-dimensional camera using a ToF sensor is about

ToF (Time of Flight) is a method of calculating the distance by measuring flight time, that is, the time that light is emitted and reflected. .

These ToF sensors are a major technological means that can be applied to a wide variety of fields such as quadcopters, autonomous vehicles, motion recognition control, virtual reality, games, 3D modeling, and human robot interaction.

On the other hand, in the case of a ToF sensor that measures distance information by detecting the intensity of reflected light using a light source, the position of the light source is located as close to the sensor as possible.

For this reason, when an object is close to the sensor, there is a problem in that it is difficult to accurately recognize the distance information of the object due to the light that is diffusely reflected from the object and the boundary of the object. technical means are required.

Republic of Korea Patent Publication No. 10-2018-0032989 (published on April 02, 2018, title: ToF (time of flight) photographing apparatus and method for reducing depth distortion by multiple reflections) Republic of Korea Patent Publication No. 10-2013-0092157 (Published date: August 20, 2013, title: Depth map correction apparatus and method, and stereoscopic image conversion apparatus and method using the same)

It is an object of the present invention to effectively remove a depth information error caused by multiple reflection paths in a three-dimensional camera using a ToF sensor.

The method for correcting depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor according to the present invention for solving this technical problem is the light intensity value of all pixels included in the image captured by the ToF sensor by the pixel information acquisition unit and A pixel information obtaining step of obtaining a depth value, a pixel classification unit compares a value obtained by multiplying the reciprocal of the square of the light intensity value and the depth value by a threshold value corresponding to the reflectivity and a scale value for adjusting the threshold value A pixel classification step of classifying pixels into a normal pixel in which the depth information error does not occur and a noise pixel in which the depth information error occurs, a noise removing step in which a noise removing unit removes noise of the noise pixel, and a depth information output unit obtaining the pixel information and a depth information output step of outputting depth information of all pixels by synthesizing the depth value of the normal pixel obtained by the unit and the depth value of the noise pixel from which the noise has been removed by the noise removing unit.

The apparatus for correcting depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor according to the present invention comprises: a pixel information obtaining unit for obtaining light intensity values and depth values of all pixels included in an image photographed by the ToF sensor; By comparing a value obtained by multiplying a reciprocal of the square of the light intensity value and the depth value by a threshold value corresponding to the reflectivity and a scale value for adjusting the threshold value, all pixels are compared with a normal pixel in which a depth information error does not occur and the A pixel classifier for classifying a noise pixel having a depth information error, a noise removing unit for removing the noise of the noise pixel, and the depth value of the normal pixel obtained by the pixel information obtaining unit and the noise removing unit and a depth information output unit outputting depth information of all pixels by synthesizing the depth values of the removed noise pixels.

According to the present invention, it is possible to effectively remove a depth information error caused by multiple reflection paths in a 3D camera using a ToF sensor.

1 is a view showing an apparatus for correcting a depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor according to an embodiment of the present invention;
2 is a diagram illustrating a method for correcting a depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor according to an embodiment of the present invention;
3 is a diagram illustrating a depth image in which a depth information error due to multiple reflection paths does not occur together with a light intensity image according to an embodiment of the present invention;
4 is a diagram illustrating a depth image in which a depth information error due to multiple reflection paths occurs together with a light intensity image according to an embodiment of the present invention;
5 is a diagram illustrating a depth image from which a depth information error due to multiple reflection paths is removed together with a light intensity image according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention may take various forms. It can be implemented with the above and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another, for example without departing from the scope of the inventive concept, a first component may be termed a second component and similarly a second component A component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that other components may exist in between. will be. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described herein exists, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in the dictionary should be interpreted as having meanings consistent with the meanings in the context of the related art, and are not to be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. .

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing an apparatus for correcting a depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor according to an embodiment of the present invention, and FIG. 2 is a three-dimensional camera using a ToF sensor according to an embodiment of the present invention. It is a diagram showing a method of correcting depth information error due to multiple reflection paths.

Hereinafter, a method embodiment will be described in order to avoid duplication of description, but it should be noted that the description part of the method embodiment may also be applied to an apparatus embodiment.

1 and 2, the method for correcting depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor according to an embodiment of the present invention includes a pixel information acquisition step (S10), a pixel classification step (S20), It is configured to include a noise removal step (S30) and a depth information output step (S40).

In the pixel information obtaining step (S10), a process of obtaining, by the pixel information obtaining unit 10, a light intensity value and a depth value of all pixels included in the captured image is performed.

Although it is a well-known technique, the process of obtaining the pixel information by calculating the light intensity value and the depth value of all pixels by the pixel information acquisition unit 10 will be described as an example.

If the luminous flux of the light source provided in the 3D camera using the ToF sensor is C=2.997*10 ⁸ m/s and the modulation frequency for the light source is f, the distance that can be expressed with respect to the modulation frequency (f) The range is expressed as R(f)=C/2f, and the distance information at this time, ie, depth information, is expressed as Depth=(phase * R(f))/2. phase is the phase of the light.

If it is assumed that the number that can equally distribute the phases of 2π is N, then I and Q can be obtained using the image information from 0 to N divided by N obtained according to each phase, and using this, the phase, Depth and Amplitude can be calculated using Equations 1 to 5 below.

[Formula 1]

[Equation 2]

[Equation 3]

[Equation 4]

[Equation 5]

In the pixel classification step (S20), the pixel classification unit 20 for each pixel constituting the entire pixel, a threshold value ( threshold) and a value multiplied by a scale value (scale_value) for adjusting the threshold, and classifying all pixels into a normal pixel in which a depth information error does not occur and a noise pixel in which a depth information error occurs is performed. .

For example, in the pixel classification step ( S20 ), the pixel classifying unit 20 may 1) select a pixel whose light intensity value is smaller than a value obtained by multiplying a threshold value and a scale value by the reciprocal of a depth value squared among all pixels. It may be configured to classify pixels as pixels, and 2) to classify pixels whose light intensity value is equal to or greater than a value obtained by multiplying a threshold value and a scale value by a reciprocal of a depth value squared among all pixels as normal pixels.

For example, the threshold and scale values may be configured to be set and reset by a user.

A process of classifying all pixels into normal pixels and noise pixels by determining whether depth noise has occurred in each of the pixels constituting all pixels will be described in more detail and exemplarily as follows.

In a 3D camera using a ToF sensor, as the object exposed to the light source of constant intensity used for distance measurement becomes farther away from the light source, the intensity of light reflected from the object is measured to be smaller. In general, the intensity of light according to the distance can be approximated as a value inversely proportional to the square of the distance.

Equation 6 below can be used in the pixel classification step (S20) to determine whether or not the pixel is noisy according to the above principle. It can be configured to

[Equation 6]

In Equation 6, Amplitude is the light intensity value of a pixel, Depth is a depth value that is a distance information value of the pixel, threshold is a threshold value according to reflectivity, scale_value is a scale value for adjusting the size of the threshold, threshold and scale_value are input by the user is the value

When the condition of Equation 6 is satisfied, the pixel classifying unit 20 determines that noise due to multiple reflection paths is included in the depth value of the corresponding pixel. That is, the corresponding pixel is classified as a noise pixel.

Conversely, when the condition of Equation 6 is not satisfied, the pixel classifying unit 20 determines that noise due to multiple reflection paths is not included in the depth value of the corresponding pixel. That is, the corresponding pixel is classified as a normal pixel.

An example of how the threshold and scale_value are set by the user is as follows.

For example, since the amplitude value according to the output distance is different depending on the type of light source, the control method, and the object, the maximum range of the threshold is set through the ToF camera to be actually used.

Using Equation 6, the maximum value of the threshold can be expressed as Equation 7.

[Equation 7]

For example, the sequence for setting the range may be performed as follows.

First, after locating the subject at a close distance (eg, 30cm to 40cm), the output amplitude and depth values are substituted into amplitude_close and depth_close of Equation 7, respectively.

Next, the scale_value value is set so that the calculation result of Equation 7 to which amplitude_close and depth_close are substituted is smaller than the maximum threshold value that the user can input.

In the noise removing step ( S30 ), the noise removing unit 30 removes the noise of the noise pixel.

For example, in the noise removing step S30, the noise removing unit 30 may be configured to remove the noise of the noise pixel in such a way that the depth value of the noise pixel obtained by the pixel information obtaining unit 10 is replaced with 0. can

In the depth information output step S40 , the depth information output unit 40 determines the depth value of the normal pixel acquired by the pixel information acquisition unit 10 and the depth value of the noise pixel from which the noise is removed by the noise removal unit 30 . A process of synthesizing and outputting depth information of all pixels is performed.

Hereinafter, an embodiment of the present invention will be described by comparing an image in which a depth information error due to multiple reflection paths occurs and an image in which it does not.

In FIG. 3 , a depth image in which a depth information error due to multiple reflection paths does not occur is exemplarily shown together with a light intensity image.

4 is a diagram exemplarily showing a depth image in which a depth information error due to multiple reflection paths occurs together with a light intensity image according to an embodiment of the present invention.

Referring to FIG. 4 , based on each pixel, the light intensity value (Amplitude) is inversely proportional to the square of the depth value (Depth). It measures large, and if it is farther away, it measures small.

Referring to the image of the depth information error due to multiple reflection paths when the palm approaches the 3D camera using the ToF sensor in FIG. 4 , the light intensity image on the left (the lower the light intensity value is, the darker the actual finger is) It can be seen that only the located portion is output with a high value.

On the other hand, in the depth image before correction on the right, it can be seen that the same depth value (distance value) as that of the finger is output even in the area where the actual finger is not displayed.

In FIG. 4 , the profile line is indicated by a dotted line, the light intensity value corresponding to the profile line is displayed in a graph form at the lower part of the light intensity image, and the pre-correction depth value corresponding to the profile line is displayed at the lower part of the pre-correction depth image. This is shown in graph form.

5 is a diagram exemplarily showing a depth image from which a depth information error due to multiple reflection paths is removed together with an intensity image, according to an embodiment of the present invention.

Referring to FIG. 5 , in both the light intensity image on the left and the depth image after correction on the right, depth information is output only to the region where the real finger is located, and it can be seen that, unlike in FIG. 4 , the depth error part between the fingers is all removed. have.

In FIG. 5, the profile line is indicated by a dotted line, the light intensity value corresponding to the profile line is displayed in the form of a graph at the lower part of the light intensity image, and the post-correction depth value corresponding to the profile line is displayed at the lower part of the post-correction depth image. This is shown in graph form.

As described in detail above, according to the present invention, it is possible to effectively remove a depth information error caused by multiple reflection paths in a 3D camera using a ToF sensor.

1: Depth information error correction device due to multiple reflection paths of a 3D camera using a ToF sensor
10: pixel information acquisition unit
20: pixel classification unit
30: noise removal unit
40: depth information output unit
S10: pixel information acquisition step
S20: Pixel classification stage
S30: Noise Removal Step
S40: depth information output step

Claims (6)

A method of correcting a depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor, the method comprising:
A pixel information obtaining step of obtaining, by the pixel information obtaining unit, light intensity values and depth values of all pixels included in the image captured by the ToF sensor;
A pixel classification unit compares a value obtained by multiplying the reciprocal of the square of the light intensity value and the depth value by a threshold value corresponding to the reflectivity and a scale value for adjusting the threshold value, and selects all the pixels as a normal value in which a depth information error does not occur. a pixel classification step of classifying a pixel and a noise pixel in which the depth information error occurs;
a noise removing step of removing the noise of the noise pixel by a noise removing unit; and
The depth information output unit outputs the depth information of all pixels by synthesizing the depth value of the normal pixel obtained by the pixel information obtaining unit and the depth value of the noise pixel from which the noise is removed by the noise removing unit. comprising steps;
In the pixel classification step,
The pixel classification unit classifies a pixel whose light intensity value is smaller than a value obtained by multiplying the threshold value and the scale value by a reciprocal of a square of the depth value among all the pixels as the noise pixel;
In the pixel classification step,
The pixel classification unit classifies pixels, among all the pixels, of which the light intensity value is equal to or greater than a value obtained by multiplying the inverse of the square of the depth value by the threshold value and the scale value, as the normal pixel, a three-dimensional camera using a ToF sensor. Depth information error correction method due to multiple reflection paths.
delete delete According to claim 1,
In the noise removal step,
The method for correcting a depth information error due to multiple reflection paths of a three-dimensional camera using a ToF sensor, characterized in that the noise removing unit substitutes 0 for the depth value of the noise pixel obtained by the pixel information obtaining unit.
According to claim 1,
The depth information error correction method due to multiple reflection paths of a three-dimensional camera using a ToF sensor, characterized in that the threshold value and the scale value can be set and reset by a user.
A device for correcting a depth information error due to multiple reflection paths of a 3D camera using a ToF sensor, comprising:
a pixel information obtaining unit which obtains light intensity values and depth values of all pixels included in the image captured by the ToF sensor;
A value obtained by multiplying the reciprocal of the square of the light intensity value and the depth value by a threshold value corresponding to the reflectivity and a scale value for adjusting the threshold value is compared to compare all pixels with a normal pixel in which a depth information error does not occur and the a pixel classifier for classifying noise pixels with a depth information error;
a noise removing unit for removing the noise of the noise pixel; and
and a depth information output unit outputting depth information of all pixels by synthesizing the depth value of the normal pixel obtained by the pixel information obtaining unit and the depth value of the noise pixel from which the noise has been removed by the noise removing unit;
The pixel classifying unit classifies a pixel in which the light intensity value is smaller than a value obtained by multiplying the threshold value and the scale value by a reciprocal of the square of the depth value among all pixels as the noise pixel, and the light intensity value among all pixels Depth information error correction apparatus due to multiple reflection paths of a three-dimensional camera using a ToF sensor, which classifies pixels having a value equal to or greater than a value obtained by multiplying a value obtained by multiplying a value obtained by multiplying a value obtained by multiplying the threshold value and the scale value by an inverse of the square of the depth value, as the normal pixel.

Images