KR101896307B1 - Method and apparatus for processing depth image - Google Patents

Abstract

A method and apparatus for processing depth images is provided. Using depth images generated using different cumulative times produces a depth image with reduced noise and motion blur. Based on the noise of the depth image and the motion blur, the cumulative time for the depth image to be generated later can be determined.

Description

METHOD AND APPARATUS FOR PROCESSING DEPTH IMAGE < RTI ID = 0.0 >

The following embodiments relate to a method and apparatus for processing a depth image, and more particularly, to a method and apparatus for processing a depth image related to the cumulative time used to generate a depth image.

Time-of-Flight (ToF) Depth A camera is a camera that provides 2.5-dimensional (D) information about a scene. Here, 2.5D means that depth information is provided only for the visible surface.

The ToF depth camera includes a light emitting diode (LED) that irradiates an InfraRed (IR) signal and a sensor that detects an IR signal. The IR signal emitted from the LED is reflected on the object and reflected back to the sensor. The sensor detects an IR signal reflected and returned to the object. Here, the IR signal may mean IR light.

The travel time of the IR signal is calculated based on the phase difference between the IR signal illuminated from the ToF depth camera and the IR signal detected by the ToF depth camera. The ToF depth image consists of the result of converting the signal travel time into distance as an image. In other words, the ToF depth image is an image representing the distance between the camera and the object by converting the travel time of the IR signal into distance.

The phase difference of the IR signal can be calculated indirectly by using the result of measuring the intensity of the IR signal returned to the sensor for two or four phases. Here, the intensity of the IR signal may indicate the intensity of the IR light.

The integration time means a time obtained by summing the times for measuring the intensity of the IR signal for each of the phases. During the cumulative time, when an object to be measured or a ToF depth camera moves, a motion blur occurs in the ToF depth image photographed by the ToF depth camera.

One embodiment can provide a method and apparatus for generating a depth image with reduced noise and motion blur by using a plurality of depth images generated using different cumulative times.

One embodiment can provide a method and apparatus for determining an accumulation time based on noise and motion blur of a depth image.

In one aspect, the method includes determining at least one spatially neighboring pixel of each of the pixels of the input depth image, calculating a weight of each of the at least one spatially neighboring pixel, and calculating a depth of each of the at least one spatially neighboring pixel And generating an output depth image by updating a depth value of each of the pixels based on the weight value and the weight value.

The input depth image may be an intermediate input depth image.

Wherein the determining step comprises the steps of: identifying a first correspondence between a preceding input depth image and a trailing depth image; detecting a motion vector based on the identified first correspondence; Estimating a second corresponding relationship between a depth image of one of the preceding input depth image and the subsequent input depth image and the intermediate input depth image, and estimating a second corresponding relationship between the pixels of the intermediate input depth image And determining the at least one spatiotemporal neighbor pixel of the at least one spatially neighboring pixel.

The preceding input depth image may be an input depth image temporally preceding the intermediate input depth image. The trailing input depth image may be an input depth image traced temporally to the intermediate input depth image.

The cumulative time of the preceding input depth image and the cumulative time of the subsequent input depth image may be shorter than the cumulative time of the intermediate input depth image.

The identifying step may include calculating an optical flow between a preceding infrared intensity image and a trailing infrared intensity image and identifying the first correspondence based on the optical flow.

The preceding infrared intensity image may correspond to the preceding input depth image.

The trailing infrared intensity image may correspond to the trailing depth image.

The identifying step may further include reducing a noise of each of the preceding infrared intensity image and the following infrared intensity image by applying a bilateral filter to each of the preceding infrared intensity image and the following infrared intensity image .

The weight of each of the at least one neighboring pixel may increase as the noise of each of the at least one neighboring pixel is smaller.

The weight of each of the at least one neighboring pixel may increase as the motion blur generated in each of the at least one neighboring pixel is smaller.

According to another aspect, there is provided a method of generating a depth image, comprising: calculating a level of noise of a depth image; calculating a level of motion blur of the depth image; and generating a depth image based on at least one of the level of the noise and the level of the motion blur The depth image processing method including the step of determining the cumulative time in the depth image processing step.

Wherein if the level of the noise of the depth image is greater than the level of noise of the previous depth image, the accumulation time may increase, and if the level of the motion blur of the depth image is less than the level of the depth image, The cumulative time may be reduced if it is greater than the level of the motion blur of the previous depth image.

According to another aspect, there is provided a method of generating a depth image, comprising: receiving a receiving depth image and determining at least one spatially neighboring pixel of each of the pixels of the input depth image; calculating a weight of each of the at least one spatially neighboring pixel; And a processing unit for generating an output depth image by updating the depth value of each of the pixels based on the depth value of each of the temporal and spatial neighboring pixels and the weight value.

Wherein the processing unit is operable to identify a first correspondence between a preceding input depth image and a trailing depth image and to detect a motion vector based on the identified first correspondence relationship, Wherein the second input image can estimate a second corresponding relationship between a depth image of one of the input depth image and the subsequent input depth image and the intermediate input depth image, At least one spatially neighboring pixel may be determined.

The processing unit may calculate an optical flow between a leading infrared intensity image and a trailing infrared intensity image and identify the first correspondence based on the optical flow.

The processing unit may reduce the noise of each of the preceding infrared intensity image and the following infrared intensity image by applying a bilateral filter to each of the preceding infrared intensity image and the following infrared intensity image.

The processing unit may increase the weight of each of the at least one neighboring pixel as the noise of each of the at least one neighboring pixel is smaller.

The processing unit may increase the weight of each of the at least one neighboring pixel as the motion blur generated in each of the at least one neighboring pixels is smaller.

According to another aspect, there is provided a method of calculating a depth image, the method comprising: calculating at least one of a receiver receiving a depth image and a level of noise and a level of motion blur of a depth image; and based on at least one of the level of the noise and the level of the motion blur, A depth image processing apparatus including a processing unit for determining an accumulation time in the generation of an image can be provided.

Wherein the processing unit can increase the cumulative time if the level of the noise of the depth image is greater than the level of noise of the previous depth image and the level of the motion blur of the depth image is less than the previous depth of the depth image If the level of the motion blur of the image is larger than the level of the motion blur, the cumulative time can be reduced.

1 is a block diagram of a depth image processing apparatus according to an exemplary embodiment of the present invention.
2 is a block diagram of a depth image processing method according to an exemplary embodiment.
3 is a flow chart of a method for determining a spatio-temporal neighbor pixel according to an example.
4 is a flowchart of a method for estimating a correspondence relationship between input depth images based on an IR intensity image according to an example.
FIG. 5 illustrates a method of determining a noise level of a pixel for each pixel of an input depth image based on an IR intensity image according to an exemplary embodiment.
FIG. 6 illustrates a method of determining a motion blur level of a pixel of an input depth image based on a spatio-temporal pattern analysis according to an exemplary embodiment.
FIG. 7 illustrates the regularity between phase information according to an example.
FIG. 8 illustrates a method of determining a motion blur level of a pixel of an input depth image based on phase information according to an example.
9 is a flowchart of a method of calculating the weights of each of the temporal and spatial neighboring pixels according to an example.
10 is a flowchart of a depth image processing method for adjusting an accumulation time of a depth image according to an exemplary embodiment.

In the following, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

1 is a block diagram of a depth image processing apparatus according to an exemplary embodiment of the present invention.

The depth image processing apparatus 100 may include a camera 110, a receiving unit 120, and a processing unit 130.

The camera 110 may be a depth camera. The camera 110 may be a ToF depth camera. The camera 110 can generate an input depth image by photographing a scene. The camera 110 can continuously generate input depth images with the passage of time.

The receiving unit 120 may receive the input depth image from the camera 110.

The processing unit 130 may generate an output depth image based on the input depth image.

2 is a block diagram of a depth image processing method according to an exemplary embodiment.

In step 210, the camera 110 may generate a plurality of input depth images by photographing the scene. The camera 110 may generate a plurality of input depth images by successively photographing the scene.

The cumulative time of a plurality of input depth images may be different from each other. For example, when photographing a plurality of input depth images, the camera 110 may photograph the input depth images for different cumulative times.

An input depth image taken during a relatively shorter cumulative time may have more noise and have less motion blur than an input depth image taken during a relatively longer cumulative time. That is, the characteristics of the captured input depth image may be changed according to the cumulative time used to capture the input depth image. The cumulative time used to capture the input depth image can be approximated by the cumulative time of the input depth image. The motion blur may occur according to the relative movement between the object captured by the camera 110 and the camera 110.

In step 220, the receiving unit 120 may receive a plurality of input depth images from the camera 110. [

At step 230, the processing unit 130 may determine at least one spatio-temporal neighboring pixel of each of the pixels of the input depth image. Here, the input depth image may be one input depth image corresponding to the output depth image, which will be described later, among the plurality of input depth images. The input depth image may be an intermediate input depth image.

The plurality of input depth images may include a preceding input depth image, an intermediate input depth image, and a posterior input depth image. The intermediate input depth image may be an input depth image corresponding to the output depth image to be generated by the processing unit 130. The preceding input depth image may be at least one input depth image temporally preceding the intermediate input depth image among the plurality of input depth images. The trailing input depth image may be at least one input depth image that temporally traces the intermediate input depth image among the plurality of input depth images.

The leading input depth image and the trailing input depth image may be short cumulative time input depth images. The intermediate input depth image may be a long cumulative time input depth image. That is, the cumulative time of the preceding input depth image and the cumulative time of the subsequent input depth image may be shorter than the cumulative time of the intermediate input depth image. The cumulative time of the preceding input depth image and the cumulative time of the subsequent input depth image may be the same. Because of the cumulative time difference, the preceding input depth image and the posterior input depth image may each be a depth image with more noise and less motion blur than the intermediate input depth image.

For example, if the depth of the output image generated by the processor 130, the depth image corresponding to the input depth image at time t, the intermediate input depth image may be the input depth image at time t. The preceding input depth image may be an input depth image at time t- 1. The trailing input depth image may be an input depth image at time t + 1. The preceding input depth image and the subsequent input depth image may each be a plurality of input depth images.

A pixel to be processed among the pixels of the intermediate input depth image may be named a target pixel. Spatially neighboring pixels of the target pixel may refer to pixels neighboring the target pixel of the intermediate input depth image. The temporally neighboring pixels of the target pixel may refer to pixels corresponding to the target pixel in the preceding input depth image or the following input depth image. Spatiotemporal neighboring pixels of a target pixel may refer to pixels spatially and / or temporally neighboring the target pixel.

At step 240, the processing unit 130 may calculate a weight for each of the at least one spatiotemporal neighboring pixel, for each of the pixels of the intermediate input depth image.

In step 250, the processing unit 130 may generate an output depth image corresponding to the intermediate input depth image based on the depth values and weights of each of the at least one temporal neighbor pixels. The processing unit 130 may generate the output depth image by updating the depth value of each of the pixels of the intermediate input depth image based on the depth value and the weight of each of the at least one spatiotemporal neighboring pixels.

3 is a flow chart of a method for determining a spatio-temporal neighbor pixel according to an example.

Step 230 described above with reference to FIG. 2 may include steps 310, 320, 330, and 340.

The processing unit 130 may determine at least one spatially neighboring pixel of each of the intermediate input depth images based on motion estimation. Motion estimation can be seen as a process of identifying the correspondence between two depth images. The correspondence between the two depth images may mean a correspondence between the pixels of the two depth images. That is, the correspondence relationship may include information on matching between one pixel of the first depth image and one pixel of the second depth image.

The two input depth images may be 1) a preceding input depth image and a following input depth image, 2) a preceding input depth image and an intermediate input depth image, or 3) a middle input depth image and a following depth image have. In the following, a method for determining at least one spatially neighboring pixel of each of intermediate input depth images based on motion estimation between a preceding input depth image and a subsequent input depth is illustrated illustratively.

In step 310, the processing unit 130 may identify the correspondence between the leading input depth image and the trailing input depth image. Here, the corresponding relationship between the leading input depth image and the trailing input depth image may be named as the first corresponding relationship.

In step 320, the processing unit 130 may detect a motion vector based on the identified first correspondence. Here, the motion vector may be a vector representing a motion from a first pixel in a preceding input depth image to a second pixel in a subsequent input depth image corresponding to the first pixel.

In step 330, the processing unit 130 may estimate the correspondence between the depth image of one of the preceding input depth image and the subsequent input depth image and the intermediate input depth image, based on the detected motion vector. Here, the corresponding relationship between the depth image of one of the preceding input depth image and the subsequent input depth image and the intermediate input depth image may be named as the second corresponding relationship.

For example, the processing unit 130 may reduce the size of the motion vector detected in step 320 to ½ so that the corresponding relationship between the preceding input depth image and the intermediate depth image, or the correspondence between the intermediate depth image and the trailing depth image, can do.

The processing unit 130 may determine the location of each of the at least one temporal and spatial neighboring pixel of a pixel of the medium depth image based on the inferred correspondence.

At step 340, the processing unit 130 may determine at least one spatially neighboring pixel of each of the pixels of the intermediate input depth image based on the estimated second correspondence.

4 is a flowchart of a method for estimating a correspondence relationship between input depth images based on an IR intensity image according to an example.

Step 310 described above with reference to FIG. 3 may include steps 410, 420, and 430.

The processing unit 130 may estimate the first correspondence based on the IR intensity images.

The IR intensity image may be an image generated by accumulating IR light reflected back from the object and accumulating all the IR images without discriminating the phases. Each of the IR intensity images may correspond to one input depth image of a plurality of input depth images have. Here, the IR intensity image and the input depth image correspond to each other, which means that the IR intensity image and the input depth image are captured at the same time.

The IR intensity image may be photographed with the corresponding input depth image by the camera 110 or another sensor included in the depth image processing apparatus 100. The IR intensity image and the input depth image corresponding to each other may have the same viewpoint and / or resolution.

The preceding IR intensity image may be an IR intensity image corresponding to the preceding input depth image. The trailing IR intensity image may be an IR intensity image corresponding to the trailing input depth image. The information for one pixel may be in the input depth image and the IR intensity image. For example, information about the depth value of a pixel at a particular coordinate may be in the input depth image, and information about the IR intensity may be within the IR intensity. Furthermore, the pixels in the input depth image having the same position and the pixels in the IR intensity image can be regarded as the same pixel.

In step 410, the processing unit 130 may reduce the noise of each of the leading IR intensity image and the trailing IR intensity image by applying a bilater filter to each of the leading IR intensity image and the trailing IR intensity image.

In step 420, the processing unit 130 may calculate an optical flow between the leading IR intensity image and the trailing IR intensity image. By applying the bilateral filter in step 410, the optical flow can be estimated more accurately.

At step 430, the processing unit 130 may identify a first correspondence between the preceding input depth image and the following input depth image based on the computed optical flow. By the calculated optical flow, information on which pixel of one depth image has moved to another pixel of another depth image can be expressed in the form of a vector. Therefore, the correspondence between the two input images can be estimated by the movement information between pixels.

FIG. 5 illustrates a method of determining a noise level of a pixel for each pixel of an input depth image based on an IR intensity image according to an exemplary embodiment.

The weights calculated in step 240 described above with reference to FIG. 2 may be determined based on the level of noise of each of the at least one neighboring pixel. The processing unit 130 can estimate the level of noise of the pixel from the IR intensity image and estimate the level of noise of the pixel based on the change of the depth value of the pixels within the spatio-temporal block . Here, the level of noise may be a value indicating the degree of noise.

The graph of FIG. 4 can show the correlation between IR reflectivity and depth noise. In the graph of Fig. 4, the x- axis can indicate the IR reflectivity and the y- axis can show the depth noise. The position on the x- axis can indicate the brightness value of the digital image measured by the camera 110 sensor. The unit of the y- axis may be mm.

The IR reflectivity can indicate the intensity of the pixel in the IR intensity image. The depth noise may indicate the level of noise of the pixel of the input depth image corresponding to the pixel. The input depth image may be an input depth image of at least one of the intermediate input depth image, the preceding input depth image, and the following input depth image.

That is, in the graph of FIG. 4, a point may represent one pixel. That is, the points of the graph can represent actual data for one pixel. The pixel may represent a pixel in the IR intensity image and / or a pixel in the input depth image corresponding to the pixel. The x- coordinate of the pixel can indicate the IR reflectivity of the pixel, and the y- coordinate can indicate the depth noise of the pixel.

Also, in the graph of FIG. 4, a line may represent a correlation between IR reflectivity and depth noise according to the distribution of pixels. The correlation may be a regression curve obtained based on the measured data.

The noise of the pixels of the input depth image can be determined by the number of electrons generated in the sensor by the reflected IR light. The more electrons generated, the less the noise of a pixel can be. The smaller the number of generated electrons, the more the noise of a pixel can increase. Here, the electrons generated in the sensor by the reflected IR light may mean the number of electrons generated in the sensor with respect to the pixel when the sensor senses the reflected IR light. In general, the correlation between noise and the number of electrons can be modeled as a Poisson distribution.

When the modeling according to the Poisson distribution is applied to the correlation between the number of noise and the number of electrons, the processing unit 130 can calculate the noise of the pixel of the input depth image based on Equation (1) below.

Here, E _{N 1} can represent the energy according to the noise of the pixel of the input depth image. k may be a constant. N _electron can represent the number of electrons generated in a pixel. As shown in Equation (1), the levels of noise of each of the pixels of the input depth image may be different from each other.

The noise of a pixel can vary depending on time. The noise of the corresponding series of pixels in a plurality of input depth images may vary over time. The processing unit 130 may calculate the noise of the pixel by measuring a change in the depth value of pixels in a spatio-temporal block with respect to time and space. Here, the block for time and space may mean a window.

The processing unit 130 may calculate the noise of a pixel of the input depth image based on Equation (2).

Here, E _N2 can represent the energy according to the noise of the pixel of the input depth image.

D can represent the depth value of the pixel. D ( a , b , c ) can represent the depth value of a pixel in row b and row j of an input depth image with index a . Here, the index can represent the time at which the input depth image is generated. An input depth image having index a may mean an input depth image generated at time a .

Height and Width can represent the height and width of the block over time and space, respectively. i and j may represent the vertical and horizontal positions of one of the pixels included in the block for the space-time.

k can represent the index of the intermediate input depth image. k- 1 may represent the index of the preceding input depth image. k +1 can represent the index of the posterior input depth image. The cumulative times of the intermediate input depth image, the preceding input depth image, and the posterior input depth image may all be the same.

FIG. 6 illustrates a method of determining a motion blur level of a pixel of an input depth image based on a spatio-temporal pattern analysis according to an exemplary embodiment.

A specific pattern as shown in FIG. 6 may appear at the position where the motion blur occurs in the input depth image.

In Figure 6, generated in the input depth image at time k first pattern generated in the input depth image from -1 610, time k the second pattern 620 and the time k +1 generated in the input depth image in the Three patterns 630 are shown respectively. Here, the first pattern 610, the second pattern 620, and the third pattern 630 may each be a pattern estimated by motion blur. The input depth image at time k -1, the input depth image at time k , and the input depth image at time k +1 may be the preceding input depth image, the intermediate input depth image, and the posterior input depth image, respectively.

P ( k- 1), P ( k ) and P ( k + 1) may be vectors representing a path through which a pattern estimated by motion blur moves. Vd ( k -1), Vd ( k ), and Vd ( k +1) may indicate the directions of the first pattern 610, the second pattern 620, and the third pattern 630, respectively. In addition, a vector 640 from the first pattern 610 to the third pattern 630 is shown.

Patterns generated in a plurality of input depth images may have directionality. The processing unit 130 verifies whether or not the patterns estimated as motion blur generated in the plurality of input depth images generated according to the time move in the direction coinciding with the moving direction of the pattern of the motion blur, It can be determined whether or not the patterns estimated to be motion blur are actually patterns representing motion blur.

The pattern estimated as motion blur may be a pattern having adjacent white and black surfaces. Generally, a pattern of a motion blur is a white surface on the front side and a black side on the rear side with respect to the moving direction. Accordingly, if the patterns estimated by the motion blur are white on the front and black on the basis of the moving direction of the patterns estimated by the motion blur over time, the processing unit 130 determines that the patterns estimated by the motion blur are actually motion It can be judged that it represents blur.

The processing unit 130 can calculate the energy according to the motion blur based on Equation (3) below. If the energy value according to the motion blur is below the reference value, the patterns estimated as motion blur are actually motion blur Patterns. ≪ / RTI >

Here, E _MB can represent energy according to motion blur. E _MB can have a larger value as the angle of Vp ( k ) and the angle of Vd ( k ) are different from each other. That is, the more similar the angle of Vp ( k ) and the angle of Vd ( k ), the greater the probability that the vectors estimated by motion blur are actually motion blur vectors.

The processing unit 130 may calculate the level of motion blur of the input depth image by measuring the width and intensity of the motion blur patterns when it is determined that the patterns estimated by the motion blur are actually motion blur patterns. Here, the level of the motion blur of the input depth image may indicate the level of the motion blur for each of the pixels in the input depth image. The level of motion blur may be a value indicating the degree of motion blur.

FIG. 7 illustrates the regularity between phase information according to an example.

To generate an input depth image, the camera 110 may measure four phase information. Here, the phase information may be represented by Q1, Q2, Q3, and Q4.

Q1, Q2, Q3 and Q4 may be, each information of the 0 ° phase of the Q ₀ of information, 180 ° phase Q ₁₈₀ of information, 90 ° of the phase information and the Q ₉₀ Q ₂₇₀ 270 ° phase of the. For example, Q1 may be phase information when the phase difference between the irradiated IR light and the returned IR light is 0 DEG. Q3 may be phase information when the phase difference between the irradiated IR light and the returned IR light is 180 DEG. The phase information may reflect the intensity of the IR light that is reflected back.

Q1, Q2, Q3 and Q4 can have regularity with each other on the principle of measurement of phase information.

The graph of Fig. 7 can show the relationship between Q1-Q2 and Q3-Q4 of the pixels of the depth image. The x-axis of the graph can represent the difference between the value of Q1 minus the value of Q2. The y-axis of the graph can represent the difference between the value of Q3 minus the value of Q4. Q1-Q2 and Q3-Q4 may represent the difference between the brightness values of the two phase images, respectively.

The phase image may be a 2D image. Thus, the difference between the brightness values of the phase images may have 2D values. Here, the values of 2D may correspond to pixels, respectively. Also, Q1-Q2 and Q3-Q4 may mean brightness values for one pixel in the phase image, respectively.

Each point in the graph may correspond to a pixel in the input depth image. That is, the pixels in the input depth image may have the regularity indicated by the distribution of the positions of the points in the graph of FIG. The regularity can be expressed by the following equation (4).

Here, k may be a constant.

That is, for a pixel in an input depth image such as an intermediate depth-depth image, regularity may mean that the sum of the absolute value of the difference between Q1 and Q2 and the absolute value of the difference between Q3 and Q4 of the pixel has a constant value.

The closer the particular pixel of the intermediate input depth image is to the diamond shape represented by the points in the graph of FIG. 7, the more the phase information of that particular pixel can be regarded as being accurately measured. On the other hand, the farther the specific pixel is from the diamond, the phase information of the particular pixel may be regarded as being incorrectly measured.

The above-described regularity can be maintained when no motion blur occurs in the pixels of the input depth image. That is, a pixel to which the above-described regularity is not applied can be regarded as a pixel where motion blur occurs.

FIG. 8 illustrates a method of determining a motion blur level of a pixel of an input depth image based on phase information according to an example.

The graph of FIG. 8 can show the relationship between Q1-Q2 and Q3-Q4. The x-axis of the graph can represent the difference between the value of Q1 minus the value of Q2. The y-axis of the graph can represent the difference between the value of Q3 minus the value of Q4.

In Fig. 7, the relationship between Q1-Q2 and Q3-Q4 of specific pixels to which the regularity is not applied is indicated by the symbol 'X'.

The symbol 'x' is far from the rhombus consisting of dots. Therefore, the processing unit 130 can determine that the regularity between the phase information is not maintained for pixels corresponding to the symbol 'X'. Also, the processing unit 130 can determine that a motion blur has occurred in the pixel corresponding to the symbol 'X'.

For the pixels in the input depth image, the processing unit 130 calculates the sum of the absolute value of the difference between Q1 and Q2 of the pixel and the absolute value of the difference between Q3 and Q4, and calculates the motion blur level of the pixel based on the calculated sum You can decide. The processing unit 130 can determine that the higher the absolute value of the difference between the constants k indicating the regularity between the calculated sum and phase information, the higher the motion blur level of the pixel.

9 is a flowchart of a method of calculating the weights of each of the temporal and spatial neighboring pixels according to an example.

Step 240 described above with reference to FIG. 2 may include steps 910, 920, 930, 940, and 950.

In step 910, the processing unit 130 may calculate the level of noise of each of the at least one neighboring pixel. A method of calculating the level of noise has been described above in detail with reference to FIG.

In step 920, the processing unit 130 may calculate the level of motion blur for each of at least one neighboring pixel. A method of calculating the level of motion blur has been described in detail above with reference to Figs.

In step 930, the processing unit 130 may compute the degree of similarity of the depth between the pixels of the input depth image and the at least one temporal / spatial neighboring pixel of the pixel. Here, the degree of similarity of depth may be larger as the depth value of the pixel of the input depth image and the depth value of at least one space-time neighboring pixel of the pixel are closer to each other. That is, the similarity of the depth may be inversely proportional to the absolute value of the difference between the depth value of the pixel of the input depth image and the depth value of the at least one space-time neighboring pixel of the pixel.

At step 940, the processing unit 130 may compute the degree of similarity between the intensity of the IR intensity image and the at least one spatiotemporal neighboring pixel of the pixel. Here, the degree of similarity of the intensity may be larger as the intensity value of the pixel of the IR intensity image and the intensity value of the at least one space-time neighboring pixel of the pixel are closer to each other. That is, the similarity of the intensity may be inversely proportional to the absolute value of the difference between the intensity value of the pixel of the IR intensity image and the intensity value of the at least one spatio-temporal neighboring pixel of the pixel. The IR intensity images of step 930 and the input depth image of step 930 may be images corresponding to each other. In addition, the pixels of the input depth image of step 930 and the pixels of the IR intensity image of step 940 may correspond to each other and may be considered to represent different information of the same pixel.

The pixel i may be a pixel of an input depth image such as an intermediate input depth image or a pixel of an IR intensity image. Pixel j may be a neighbor pixel of one of the at least one spatio temporal neighbor pixel of pixel i .

In step 950, the processing unit 130 determines the following: 1) the level of noise of neighboring pixels, 2) the level of motion blur of neighboring pixels, 3) the similarity of the depths of pixels of the input depth image and neighboring pixels, and 4) The weight of the neighboring pixels may be calculated based on at least one of the pixel of the neighboring pixels and the similarity between neighboring pixels.

For example, the processing unit 130 may increase the weight of each of the at least one neighboring pixel as the noise of each of the at least one neighboring pixel becomes smaller. The processing unit 130 may increase the weight of each of the at least one neighboring pixel as the motion blur generated in each of the at least one neighboring pixel becomes smaller. The processing unit 130 may increase the weight of each of the at least one neighboring pixel as the depth similarity between at least one of the neighboring pixels and the pixels of the input depth image is lower. The processing unit 130 may increase the weight of each of the at least one neighboring pixel as the intensity similarity degree between each of the at least one neighboring pixel and the pixel of the IR intensity image is lower.

The processing unit 130 may calculate a weight based on Equation (5) below.

Here, i may represent a pixel of the input depth image and a pixel of the IR intensity image corresponding to the pixel of the input depth image. j may represent a neighboring pixel of one of at least one neighboring pixel of pixel i . k can represent a k- th input depth image among N input depth images. The kth input depth image may be an input depth image of pixel i . That is, k may represent an index of an input depth image including a pixel i .

W ₁ (k, i, m, j) may be a k-th input depth of a pixel i of an image to be used for filtering, 1) m of the second input depth image, and 2) the weight of the pixel i spatial and temporal neighboring pixels j of . Here, the k- th input depth image may be an intermediate input depth image. The m- th input depth image may be an input depth image of one of the preceding input depth image, the intermediate input depth image, and the posterior input depth image.

? can be a constant of the Gaussian distribution.

E _N may be the energy according to the noise of pixel j . E _N may be E _{N 1} of the above-mentioned equation (1). E _N may be E _{N 2} in the above-described equation (2). The processing unit 130 may calculate E _N based on one or more of E _{N 1} and E _{N 2} .

E _M can be the energy according to the motion blur of pixel j . E _M can be E _{MB in} Equation (3). Processor 130 may on the basis of the calculated E _MB E _M.

E _{S 1} may be energy according to the depth similarity between pixel i and pixel j .

E _{S 2} may be an energy according to intensity similarity between pixel i and pixel j .

C ₁ , C ₂ , C ₃ and C ₃ may be weights of E _N , E _M , E _{S 1} and E _{S 2} , respectively, and may have a value of zero or more.

After the weights of at least one neighboring pixel of each of the pixels of the input depth image are computed, the processing unit may generate an output depth image at step 250.

The depth value of each of the pixels of the output depth image may be a sum of products of the weight and depth values of at least one neighbor pixel of the pixels of the output depth image. The processing unit 130 may multiply the sum of the products of the weights and depth values of at least one neighboring pixel of the pixel with respect to the pixels of the input depth image by a depth value of the pixel of the output depth image corresponding to the pixel of the input depth image As shown in FIG.

The processing unit 130 may calculate the depth value of each of the pixels of the output depth image based on Equation (6).

Here, D _raw ( m , j ) may be the depth value of the pixel j of the m- th input depth image.

N ( k , j ) may be a set of at least one neighboring pixel of pixel j of the k- th input depth image.

D _filtered ( k , i ) may be the depth value of pixel i of the kth output depth image. The kth output depth image may be an output depth image corresponding to the kth input depth image.

The output depth image relative to the input depth image may be a filtered depth image by the weight and depth values of at least one neighboring pixel. Thus, the output depth image can be named a filtered depth image.

10 is a flowchart of a depth image processing method for adjusting an accumulation time of a depth image according to an exemplary embodiment.

In step 1010, the camera 110 may generate a plurality of input depth images by photographing the scene. The camera 110 may generate a plurality of input depth images by successively photographing the scene.

The cumulative time of a plurality of input depth images may be different from each other. That is, the camera 110 may photograph the input depth images for different cumulative times when photographing a plurality of input depth images.

In step 1020, the receiving unit 120 may receive a plurality of input depth images from the camera 110.

In step 1030, the processing unit 130 may calculate the level of noise in the input depth image. The input depth image may be the current input depth image among the plurality of input depth images. A method of calculating the level of noise has been described above in detail with reference to FIG. Here, the level of noise of the input depth image may mean the level of noise of each of the pixels of the input depth image or the average level of the noise.

In step 1040, the processing unit 130 may calculate the level of motion blur of the input depth image. A method of calculating the level of motion blur has been described in detail above with reference to Figs. Here, the level of the motion blur of the input depth image may mean the levels of the motion blur of each of the pixels of the input depth image or the average level of the motion blur.

Thus, by performing one or more of steps 1030 and 1040, the processing unit 130 may calculate one or more of the level of noise and the level of motion blur.

At step 1050, the processing unit 130 may determine the cumulative time in the generation of the depth image based on at least one of the level of the calculated noise and the level of the calculated motion blur.

The processing unit 130 may determine the cumulative time in the generation of the depth image based on Equation (7).

T ( k ) may be the cumulative time for the k-th input depth image. T (k +1) may be a cumulative time for the k th +1 input depth image, for example, the input depth image is taken next. A may be a variation of the cumulative time determined based on one or more levels of the level of noise of the k-th input depth image and the level of motion blur. alpha may be a constant value that adjusts the degree of change of the cumulative time.

k , i , j , sigma , E _N and E _M may have the same meanings as in Equation (6). C ₅ and C ₆ may be weights of E _N and E _M , respectively, and may have a value of zero or more.

For example, the processing unit 130 may increase the cumulative time if the level of the noise of the input depth image is larger than the level of the noise of the previous depth image. The processing unit 130 may reduce the cumulative time if the level of the motion blur of the input depth image is larger than the level of the motion blur of the previous depth image of the input depth image. The increased or decreased cumulative time may be applied to the generation of a subsequent depth image of the input depth image. That is, the processing unit 130 may increase the cumulative time used to generate the input depth image if the noise increases in the input depth images generated by the camera 110 over time. Also, the processing unit 130 may reduce the cumulative time used to generate the input depth image, if the motion blur increases within the input depth images generated by the camera 110 over time.

If the cumulative time is too short, the noise size becomes larger than the distance information of the input depth image. When the size of the noise increases, the quality of the input depth image deteriorates, and it may become difficult to measure the depth of a distant object to which a relatively insufficient amount of light is given as compared with a nearby object. Adjustment of the cumulative time described above can minimize the noise and motion blur that occur in the input image.

The technical contents described above with reference to Figs. 1 to 9 can be directly applied. Steps 1010, 1020, 1030 and 1040 may correspond to steps 210, 220, 910 and 920, respectively. A more detailed description will be omitted below.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

100: Depth image processing device
110: camera
120: Receiver
130:

Claims (19)

Determining at least one temporal and spatial neighboring pixel of each of the pixels of the input depth image based on a preceding input depth image, an intermediate input depth image corresponding to the input depth image, and a following input depth image;
Calculating a weight of each of said at least one spatially neighboring pixel; And
Generating an output depth image by updating a depth value of each of the pixels based on the depth value and the weight of each of the at least one spatiotemporal neighboring pixels
Lt; / RTI >
Wherein the preceding input depth image is an input depth image temporally preceding the intermediate input depth image, the trailing input depth image is an input depth image traced temporally to the intermediate input depth image,
The cumulative time of the preceding input depth image and the cumulative time of the following input depth image are shorter than the cumulative time of the intermediate input depth image
Depth image processing method. The method according to claim 1,
Wherein the determining comprises:
Identifying a first corresponding relationship between the preceding input depth image and the subsequent input depth image;
Detecting a motion vector based on the identified first correspondence;
Estimating a second corresponding relationship between the depth image of one of the preceding input depth image or the trailing depth image and the intermediate depth image based on the detected motion vector; And
Determining the at least one spatiotemporal neighbor pixel of each of the pixels of the intermediate input depth image based on the estimated second correspondence
/ RTI > delete 3. The method of claim 2,
Wherein the identifying comprises:
Calculating an optical flow between a leading infrared intensity image and a trailing infrared intensity image; And
Identifying the first correspondence based on the optical flow
Lt; / RTI >
Wherein the preceding infrared intensity image corresponds to the preceding input depth image and the trailing infrared intensity image corresponds to the trailing input depth image. 5. The method of claim 4,
Wherein the identifying comprises:
A step of reducing the noise of each of the preceding infrared intensity image and the following infrared intensity image by applying a bilateral filter to each of the preceding infrared intensity image and the following infrared intensity image,
The depth image processing method further comprising: The method according to claim 1,
Wherein the weight of each of the at least one neighboring pixel is increased as the noise of each of the at least one neighboring pixels is smaller. The method according to claim 1,
Wherein the weight of each of the at least one neighboring pixel is increased as the motion blur generated in each of the at least one neighboring pixels is smaller. Calculating at least one of a level of noise of the input depth image and a level of motion blur; And
Determining an accumulated time in generation of the input depth image based on at least one of the level of the noise and the level of the motion blur
Lt; / RTI >
Wherein the input depth image is an intermediate input depth image, the preceding input depth image is an input depth image temporally preceding the intermediate input depth image, the posterior input depth image is an input depth image traced temporally to the intermediate input depth image, ,
Wherein the accumulation time of the preceding input depth image and the accumulation time of the subsequent input depth image are shorter than the accumulation time of the intermediate input depth image,
Depth image processing method. 9. The method of claim 8,
Wherein the cumulative time is increased if the level of the noise of the input depth image is greater than the level of noise of the preceding input depth image and the level of the motion blur of the input depth image is greater than the level of the preceding input depth Wherein the cumulative time is reduced if the level of motion blur is greater than the level of motion blur of the image. A computer-readable recording medium embodying a program for performing the method of any one of claims 1, 2, and 4 to 9. A receiving unit for receiving an input depth image; And
Determining at least one temporal and spatial neighboring pixel of each of the pixels of the input depth image based on a preceding input depth image, an intermediate input depth image corresponding to the input depth image, and a following input depth image, A processor for generating an output depth image by calculating a weight of each pixel and updating the depth value of each of the pixels based on the depth value of each of the at least one temporal and spatial neighboring pixels and the weight,
Lt; / RTI >
Wherein the preceding input depth image is an input depth image temporally preceding the intermediate input depth image, the trailing input depth image is an input depth image traced temporally to the intermediate input depth image,
Wherein the accumulation time of the preceding input depth image and the accumulation time of the subsequent input depth image are shorter than the accumulation time of the intermediate input depth image,
Depth image processing device. 12. The method of claim 11,
Wherein the processing unit identifies a first corresponding relationship between the preceding input depth image and the trailing input depth image, detects a motion vector based on the identified first correspondence relationship, and based on the detected motion vector, Estimating a second corresponding relationship between a depth image of one of the depth input image and the trailing input depth image and the intermediate input depth image, and based on the estimated second correspondence, Determining one spatio-temporal neighbor pixel,
Depth image processing device. delete 13. The method of claim 12,
Wherein the processor is configured to calculate an optical flow between a leading infrared intensity image and a trailing infrared intensity image, identify the first correspondence based on the optical flow,
Wherein the preceding infrared intensity image corresponds to the preceding input depth image and the trailing infrared intensity image corresponds to the trailing input depth image. 15. The method of claim 14,
Wherein the processing unit reduces a noise of each of the preceding infrared intensity image and the following infrared intensity image by applying a bilateral filter to each of the preceding infrared intensity image and the following infrared intensity image. 12. The method of claim 11,
Wherein the processing unit increases the weight of each of the at least one neighboring pixel as the noise of each of the at least one neighboring pixels is smaller. 12. The method of claim 11,
Wherein the processing unit increases the weight of each of the at least one neighboring pixel as the motion blur generated in each of the at least one neighboring pixels is smaller. A receiving unit for receiving an input depth image; And
Calculating at least one of a level of noise of the input depth image and a level of motion blur and determining an accumulation time in generation of the input depth image based on at least one of the level of the noise and the level of the motion blur Processing unit
Lt; / RTI >
Wherein the input depth image is an intermediate input depth image, the preceding input depth image is an input depth image temporally preceding the intermediate input depth image, the posterior input depth image is an input depth image traced temporally to the intermediate input depth image, ,
Wherein the accumulation time of the preceding input depth image and the accumulation time of the subsequent input depth image are shorter than the accumulation time of the intermediate input depth image,
Depth image processing device. 19. The method of claim 18,
Wherein the processing unit increases the accumulation time if the level of the noise of the input depth image is greater than the level of noise of the preceding input depth image, and the level of the motion blur of the input depth image is higher than the level of the motion of the preceding input depth image And the cumulative time is decreased if the level of blur is greater than the level of blur.

Images