KR20130082054A - Method and apparatus for recovering depth value of depth image - Google Patents

Abstract

PURPOSE: A method and apparatus for restoring a depth value of a depth image determines the number of modes of corresponding pixels within a first depth image and a second depth image, and restores at least one depth value of a depth image of the first depth image and the second depth image, based on the determined NoM. CONSTITUTION: A receiving unit receives a first depth image and a second depth image (330). A processing unit determines NoM (Number of Mods) of modes of corresponding pixels within the first depth image and second depth image (340). The processing unit restores a depth value of at least one depth image of the first depth image and the second depth image based on the determined NoM (350). The processing unit uses the first depth image and the second depth image of which the depth values are restored, and renews each restored depth value of the first depth image and the second depth image (360). [Reference numerals] (310) Generate a first depth image; (320) Generate a second depth image; (330) Receive the depth images; (340) Decide a NoM; (342) Decide a NoM for each pixel of the first depth image; (344) Decide a NoM for each pixel of the second depth image; (350) Recover the depth value of the depth images; (360) Update the recovered depth value; (362) Update the recovered depth value of the first depth image; (364) Update the recovered depth value of the second depth image; (AA) Start; (BB) End

Description

METHOD AND APPARATUS FOR RECOVERING DEPTH VALUE OF DEPTH IMAGE}

The following embodiments relate to a method and apparatus for reconstructing a depth value in a depth image, and more particularly, to a method and apparatus for calculating a depth value of a pixel in a depth image photographed by a ToF camera.

A time-of-flight (ToF) depth camera is a camera that provides 2.5D (D) information of the scene. Here, 2.5D means that depth information is provided only for the visible surface.

The ToF depth camera includes a light emitting diode (LED) for irradiating infrared (InfraRed) light and a sensor for detecting light. The ToF depth camera calculates the distance between the ToF depth camera and the object by measuring how long the IR light emitted from the LED hits the object, reflects, and returns to the sensor.

The irradiated IR light is modulated at the frequency f . In other words, the IR light is light having a frequency f . ToF t _{time-of-flight} is calculated indirectly by measuring the intensity of the light returning to the sensor for two phases or four phases.

The method may further include receiving a first depth image and a second depth image, and determining a number of mods (NoM) of modes of respective pixels in the first depth image and the second depth image. Reconstructing a depth value of at least one depth image of the first depth image and the second depth image based on the determined NoM, wherein the corresponding pixels represent the same three-dimensional point. The depth image processing method may be provided.

The determining may include determining a NoM of each pixel in the first depth image and minimizing a Markov Random Field (MRF) energy for the first depth image and for the second depth image. And determining the NoM of each of the pixels in the second depth image to minimize MRF energy.

The MRF energy for the first depth image may be determined based on the data cost of each of the pixels in the first depth image and the discontinuity cost of two adjacent pixels in the first depth image.

The MRF energy for the second depth image may be determined based on the data cost of each of the pixels in the second depth image and the discontinuity cost of two adjacent pixels in the second depth image.

The discontinuity cost may be a cost reflecting the constraint that the NoMs of the two pixels are the same when three-dimensional distances of two spatially neighboring pixels in one depth image are similar to each other.

The three-dimensional distance of the pixel may be a distance between a camera that generates a depth image including the pixel and a three-dimensional point corresponding to the pixel.

The first pixel in the first depth image and the second pixel in the second depth image may be pixels corresponding to each other.

The determined data cost of the first pixel that minimizes the distance between the first three-dimensional point reconstructed by applying NoM to the first pixel and the second three-dimensional point reconstructed by applying NoM to the second pixel. It can be calculated by calculating the NoM and the determined NoM of the second pixel.

When the value of the calculated data cost is greater than the threshold, the value of the data cost may be changed to the threshold by a robust potential function.

The depth image processing method may further include updating the reconstructed depth value of the first depth image by using the second depth image in which the depth value is reconstructed.

The updating may include the depth value based on the restored depth value of the second depth image such that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other. The reconstructed depth value of the first depth image may be updated by repeatedly minimizing MRF energy for the reconstructed first depth image.

The depth image processing method may further include generating the first depth image and generating the second depth image.

The first depth image and the second depth image may be generated by photographing the scene at different positions and / or directions.

According to another aspect, a receiver for receiving a first depth image and a second depth image, a value of a number of mods (NoM) of respective modes of pixels corresponding to the first depth image and the second depth image And a processor configured to reconstruct a depth value of at least one depth image of the first depth image and the second depth image based on the determined value of NoM, wherein the corresponding pixels have the same three-dimensional point ( A depth image processing apparatus may be provided.

The processor may determine the NoM of each of the pixels in the first depth image that minimizes Markov Random Field (MRF) energy for the first depth image, and minimizes the MRF energy for the second depth image. The NoM of each of the pixels in the second depth image may be determined.

The processor may determine the MRF energy for the first depth image based on the data cost of each pixel in the first depth image and the discontinuity cost of two adjacent pixels in the first depth image, and the second depth The MRF energy for the second depth image may be determined based on the data cost of each pixel in the image and the discontinuity cost of two adjacent pixels in the second depth image.

The processing unit minimizes the distance between the first three-dimensional point reconstructed by applying NoM to the first pixel and the second three-dimensional point reconstructed by applying NoM to the second pixel, wherein the determined NoM of the first pixel is minimized. And calculating the data cost by calculating the determined NoM of the second pixel.

The processor may change the value of the data cost to the threshold when the value of the calculated data cost is greater than the threshold by using a robust potential function.

The processor may update the reconstructed depth value of the first depth image by using the second depth image in which the depth value is reconstructed.

The processor may further include a first depth in which the depth value is restored based on a restored depth value of the second depth image such that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other. By repeatedly minimizing the MRF energy for the image, the reconstructed depth value of the first depth image may be updated.

The depth image processing apparatus may further include a first camera that generates the first depth image by photographing a scene, and a second camera that generates the second depth image by photographing the scene.

And the first camera and the second camera photograph the scene at different positions and / or directions.

1 is a structural diagram of a depth image generating apparatus according to an exemplary embodiment.
2 illustrates a distance calculation method using a plurality of ToF depth cameras according to an example.
3 is a flowchart illustrating a method of restoring a depth value of a depth image, according to an exemplary embodiment.
4 illustrates a change in distance of a 3D point according to a NoM value according to an example.
5 illustrates depth images reconstructed by using a data cost function according to an example.
6 illustrates a role of discontinuity cost according to an example.
7 illustrates depth images to which a depth value update is applied according to an example.

In the following, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

1 is a structural diagram of a depth image generating apparatus according to an exemplary embodiment.

The depth image processing apparatus 100 may include a first camera 110, a second camera 120, a receiver 130, and a processor 140.

The first depth image may be generated by photographing a scene of the first camera 110. The second camera 120 may generate a second depth image by capturing the scene. The first camera 110 and the second camera 120 may generate a first depth image and a second depth image, respectively, by photographing the scene at different positions and / or directions. The first depth image and the second depth image may be generated by photographing the scene at different positions and directions.

The first camera 110 and the second camera 120 may each be a ToF depth camera. The first camera 110 and the second camera 120 may be a stereo type ToF depth camera.

The receiver 130 may receive the captured first depth image and the second depth image.

The processor 140 may restore the depth value of the first depth image and the depth value of the second depth image. Here, the depth value of the depth image may mean a depth value of each pixel in the depth image. Restoration of the depth value may mean changing an incorrect or insufficient depth value to an accurate or more accurate depth value.

The depth value of a pixel in the depth image represents a distance between a camera photographing the depth image and a 3D point of a scene corresponding to the pixel. Thus, the closer the distance and depth value of the pixel are, the more accurate the depth value of the pixel can be considered.

2 illustrates a distance calculation method using a plurality of ToF depth cameras according to an example.

The plurality of ToF depth cameras may be the first camera 110 and the second camera 120.

The processor 140 may calculate a distance between the ToP depth camera and the object based on Equation 1 below.

Here, d may be a distance between the ToP depth camera and the object. c may be the speed of light. t _{time-of-flight} may be the _time from when the light is irradiated from the LED of the ToP depth camera to hit the object, reflect, and return to the sensor of the ToF depth camera. Hereinafter, the light irradiated from the LED is irradiated with the irradiated light, and the light reflected by the object and returned to the sensor is outlined as the reflected light. The light irradiated from the LED may be light modulated at a frequency f , and may be IR light.

t _{time-of-flight} may be indirectly calculated by measuring the intensity of light returning to the sensor for two phases or four phases, for example by the processor 140.

2 may represent a way to calculate the distance by using the intensity of light measured in two different phases. Here, the two different phases may be phases of 0 ° and phases of 180 °, respectively.

The signal received by the sensor can be expressed as Equation 2 below. Here, the received signal may be reflected light received by the sensor.

는 조사 광 및 반사 광 간의 위상차일 수 있다.Here, A may represent the amplitude of the received signal. t can represent time.

May be a phase difference between the irradiation light and the reflected light.

The processor 140 may calculate t _{time-of-flight} based on Equation 3 below.

The received signal may be mixed with sine wave sin (2 pi ft ) and cosine wave cos (2 pi ft ) by the first mixer 210 and the second mixer 220. The signals generated by the mixing may pass through a first low pass filter 212 and a second low pass filter 214, respectively. As the signals generated by the mixing pass through the first low pass filter 212 and the second low pass filter 214, the first correlated signal B and the second correlated signal C may be generated. Here, the sine wave sin (2 pi ft ) and the cosine wave cos (2 pi ft ) may be files corresponding to the first and third phases, respectively.

B and C may be calculated based on Equation 4 below.

는 각각 하기의 수학식 5에 기반하여 계산될 수 있다.In addition, A and

Each can be calculated based on Equation 5 below.

d _max may represent the maximum distance that can be measured without ambiguity. For example, d _max may be a distance corresponding to one period of the phase difference. d _max may be named the non-ambiguous distance range or the maximum acquisition distance.

When the phase difference between the irradiation light and the reflected light is larger than 2π, a value of 0 or more and 2π or less smaller than the actual phase difference between the irradiation light and the reflected light can be measured as the phase difference.

d may be calculated based on Equation 6 below.

에 2πn을 더한 값인

+2πn를 이용하여 d+nd _max를 계산할 수 있다. 여기서, n은 0 이상의 정수일 수 있다.In order to accurately calculate the distance d , the processing unit 140 measures the phase difference

Plus 2 πn

We can calculate d + nd _max using +2 π n . Here, n may be an integer of 0 or more.

The value of n may not be determined by the depth measurement method of a single depth camera. Therefore, the maximum distance that can be measured by the depth camera may be d _max , and the processor 140 may incorrectly calculate a depth value for an object farther than d _max from the depth camera as a shorter distance value than the actual value. . The depth value of the object may be a depth value of a pixel representing the object among pixels in the depth image.

That is, the depth value of the depth image captured by the first camera 110 or the second camera 120 due to ambiguity may be an incorrect depth value in which the nd _max portion is lost. The processor 140 may restore the depth value by adding an nd _max portion to the depth value of the depth image generated by the imaging.

The reconstructed depth value described above with reference to FIG. 1 may be a far reconstructed depth value by determining a value of n. Also, the reconstructed depth image may be a depth image having a far reconstructed depth value by determining a value of n for each depth value of each pixel in the depth image.

The above description with reference to Equations 1 to 6 may be applied to each of the signals received by the plurality of cameras.

3 is a flowchart illustrating a method of restoring a depth value of a depth image, according to an exemplary embodiment.

In operation 310, the first camera 110 may generate a first depth image by capturing a scene.

In operation 320, the second camera 120 may generate a second depth image by capturing the scene.

In operations 310 and 320, the first camera 110 and the second camera 120 may photograph the same scene at different positions and / or directions. That is, the first depth image and the second depth image may be generated by photographing the scene at different positions and / or directions, respectively. The first depth image and the second depth image may have different views of the scene.

In operation 330, the receiver 130 may receive a first depth image from the first camera 110 and may receive a second depth image from the second camera 120.

In operation 340, the processor 140 may determine a NoM of each of the corresponding pixels in the first depth image and the second depth image. Here, the corresponding pixels may represent the same three-dimensional point. That is, when an object in the scene is photographed by each of the first camera 110 and the second camera 120, the pixels in the first depth image and the pixels in the second depth image representing the object are pixels corresponding to each other. .

The determination of NoM is described in detail with reference to FIG. 4 below.

Step 340 may include step 342 and step 344.

In operation 342, the processor 140 may determine a NoM of each pixel in the first depth image that minimizes Markov Random Field (MRF) energy for the first depth image. The Markov random field energy is described in detail with reference to FIG. 5 below.

In operation 344, the processor 140 may determine a NoM of each pixel in the second depth image that minimizes the MRF energy for the second depth image.

In operation 350, the processor 150 may restore a depth value of at least one depth image of the first depth image and the second depth image based on the determined NoM.

In operation 360, the processor 160 reconstructs the reconstructed depth value of the first depth image and the second depth image by using the first depth image in which the depth value is restored and the second depth image in which the depth value is restored. You can update the depth value. Step 360 may be regarded as performing optimization on the first depth image and the second depth image.

Step 360 can include step 362 and step 364.

In operation 362, the processor 160 may update the restored depth value of the first depth image by using the second depth image in which the depth value is restored.

In operation 364, the processor 160 may update the restored depth value of the second depth image by using the first depth image in which the depth value is restored.

4 illustrates a change in distance of a 3D point according to a NoM value according to an example.

The first pixel may be one of the pixels in the first depth image, and the second pixel may be one of the pixels in the second depth image. The first pixel and the second pixel may be pixels corresponding to each other.

In operation 330 described above with reference to FIG. 3, the depth value of the first pixel of the first depth image is d ₁ . The depth value of the second pixel of the second depth image is d ₂ . As described above with reference to FIG. 2, d ₁ may be a value smaller than 0 or more d _max ( f ₁ ), and d ₂ may be smaller than 0 or more d _max ( f ₂ ).

d _max ( f ₁ ) may be d _max of a signal detected by a sensor of the first camera 110 when the light emitted from the first camera 110 is modulated at a frequency f ₁ . Here, d _max of the signal may be d _max of the pixel in the depth image. Also, d _max ( f ₂ ) may be d _max of a signal detected by a sensor of the second camera 120 when the light emitted from the second camera 120 is modulated at a frequency f ₂ . Here, d _max of the signal may be d _max of the pixel in the depth image. That is, each of the pixels in the first depth image may have a depth value equal to or greater than 0 and smaller than d _max ( f ₁ ), regardless of how far each point represented by the pixels actually falls from the first camera 110. have. The depth value of the pixel of the first depth image may be the remaining value when the value of the actual distance between the first camera 110 and the point indicated by the pixel is divided by d _max ( f ₁ ).

x ₁ may represent the first pixel. Also, x ₂ may represent a second pixel. X ₁ may represent a 3D point corresponding to x ₁ . X ₂ may represent a 3D point corresponding to x ₂ .

When the depth value of x ₁ generated by the first camera 110 is d ₁ , the distance between the first camera 110 and X ₁ may be d ₁ + n ₁ d _max ( f ₁ ). n ₁ may be NoM of x ₁ and X ₁ . n ₁ may be an integer of 0 or more. In addition, when the depth value of x ₂ generated by the second camera 120 is d ₂ , the distance between the second camera 120 and X ₂ may be d ₂ + n ₂ d _max ( f ₂ ). n ₂ may be NoM of x ₂ and X ₂ . n ₂ may be an integer of 0 or more.

X ₁ ( k ₁ ) may represent a 3D point corresponding to x ₁ when the value of n ₁ is k ₁ . That is, X ₁ ( k ₁ ) is determined by the position of the first camera 110, the direction of the first camera 110, d ₁ and d _max ( f ₁ ) when the NoM of x ₁ is k _1. 3D points corresponding to x ₁ may be represented. X ₂ ( k ₂ ) may represent a 3D point corresponding to x ₂ when the value of n ₂ is k ₂ . That is, X ₂ ( k ₂ ) is determined by the position of the second camera 120, the direction of the second camera 120, d ₂ and d _max ( f ₂ ) when the NoM of x ₂ is k _2. 3D points corresponding to x ₂ may be represented.

The symbol '≡' can be used to indicate the distance of the 3D point. Equation ' X ₁ ( k ₁ ) ≡ d ₁ ' may indicate that the distance between the first camera 110 and X ₁ ( k ₁ ) is d ₁ . Equation ' d ₂ ≡ X ₂ ( k ₂ )' may indicate that the distance between the second camera 120 and X ₂ ( k ₂ ) is d ₂ .

In FIG. 4, the positions of X ₁ (0), X ₁ (1) and X ₁ (2) are shown respectively, and the positions of X ₂ (0), X ₂ (1) and X ₂ (2) are shown respectively. It became.

The processor 140 may restore X ₁ , which is a 3D point corresponding to x ₁ , based on x ₁ . Here, the restoration may mean determining the NoM of x ₁ . That is, reconstruction may mean determining one of X ₁ (0) to X ₁ ( m ₁ ) as a 3D point corresponding to x ₁ . Here, m ₁ may be the maximum value of k ₁ .

If x ₁ and x ₂ correspond to each other, the 3D point reconstructed from x ₁ and the 3D point reconstructed from x ₂ may be points that are at the same location or within a certain distance.

In FIG. 4, the trajectory of X ₁ corresponding to x ₁ as the value of n ₁ is changed and the trajectory of X ₂ corresponding to x ₂ as the value of n ₂ is changed as a straight line. In FIG. 4, when NoM of X ₁ and NoM of X ₂ are each 2, X ₁ and X ₂ may intersect at one point. Therefore, according to FIG. 4, when n ₁ and n ₂ are each 2, it can be seen that X ₁ and X ₂ are correctly restored. That is, the 3D point reconstructed from x ₁ may be X ₁ (2), and the 3D point reconstructed from x ₂ may be X ₂ (2).

The processor 140 may calculate an optimal NoM of each of X ₁ and X ₂ based on Equation 7 below.

Here, R may be a rotation matrix indicating a rotation relationship between the first camera 110 and the second camera 120. R may be a rotation matrix representing rotation in the direction from the first camera 110 to the second camera 120. R may be a rotation matrix of 3 × 3. R ^T may be a transpose matrix of R.

T may be a 3D translation vector representing a parallel movement relationship between the first camera 110 and the second camera 120. T may be a 3D deformation vector representing parallel movement from the position of the first camera 110 to the position of the second camera 120.

R and T may be predetermined values given by extrinsic calibration between the first camera 110 and the second camera 120, respectively.

In Equation 7, m may be a NoM of 3D point X ₁ corresponding to one pixel x ₁ in the first depth image captured by the first camera 110. n may be the NoM of the 3D point X ₂ corresponding to one pixel x ₂ in the second depth image captured by the second camera 120.

X ₂ ( n ) may represent a 3D point when X ₁ ( m ) is projected on the second depth image, and NoM of the pixel projected on the second depth image is n . Here, the pixel in which X ₁ ( m ) is projected onto the second depth image may be X ₂ . X ₁ ( m ) may represent a 3D point when X ₂ ( n ) is projected on the first depth image, and NoM of the pixel projected on the first depth image is m . Here, the pixel in which X ₂ ( n ) is projected on the second depth image may be X ₁ .

m ^* and n ^* may be, NoM optimum value to minimize the distance between the 3D point generated by reconstructing a 3D point, and X ₂ generated by reconstructing the X _1, respectively.

The functions of Equation 7 described above are called data cost functions.

5 illustrates depth images reconstructed by using a data cost function according to an example.

In operation 340 described above with reference to FIG. 3, the processor 140 may apply the first and second reconstructed depth images 510 and the second reconstructed data by applying the above-described data cost function to the first and second depth images. The depth image 520 may be generated.

If the viewpoint of the first camera 110 and the viewpoint of the second camera 120 are the same, the first depth image photographed by the first camera 110 and the photographing by the second camera 120 are taken. There is no occlusion between the captured second depth images. Thus, for each of all the pixels in one depth image, a corresponding pixel may be present in the other depth image.

However, since the first camera 110 and the second camera 120 have different positions and / or directions with respect to the scene, for all pixels in one depth image, a corresponding pixel exists in the other depth image. You can't. That is, occlusion may exist between the first depth image and the second depth image. Also, due to noise present in each of the first depth image and the second depth image, the value of NoM calculated by using the data cost function may not always be accurate.

The first reconstructed depth image 510 and the second reconstructed depth image 520 shown may illustrate that the reconstructed depth image may not always be accurate due to the aforementioned occlusion and noise.

For each of the first depth image and the second depth image, MRF energy may be defined as shown in Equation 8 below. The processor 140 may use the MRF energy function defined according to Equation 8 below to determine the NoM of each pixel in the first depth image and the NoM of each pixel in the second depth image.

Here, E ₁ may be MRF energy for the first depth image. E ₂ may be the MRF energy for the second depth image.

i and j may each be an index of a pixel in the first depth image or the second depth image.

N may be a set of all neighboring pixel-pairs in the first depth image or the second depth image. That is, ( i , j ) may represent an ordered pair of indices of two neighboring pixels in the first depth image or the second depth image. The neighboring pixels of the pixel at index i may be four or eight.

m _i may be the NoM of the pixel at which the index of the first depth image is i . m _j may be the NoM of the pixel at which the index of the first depth image is j . n _i may be the NoM of the pixel at which the index of the second depth image is i . n _j may be the NoM of the pixel at which the index of the second depth image is j .

D ₁ may be a data cost for the first depth image. D ₂ may be a data cost for the second depth image. D ₁ ( m _i ) may be a data cost for NoM of a pixel whose index of the first depth image is i . D ₂ ( n _i ) may be a data cost for NoM of a pixel whose index of the second depth image is i .

V ₁ may be a discontinuity cost of the first depth image. V ₂ may be a discontinuity cost of the second depth image. V ₁ ( m _i , m _j ) may be the discontinuity cost of the NoM of the pixel at which the index of the first depth image is i and the NoM of the pixel at the index of j of the first depth image. V ₂ ( n _i , n _j ) may be the discontinuity cost of the NoM of the pixel at which the index of the second depth image is i and the NoM of the pixel at the index of j of the second depth image.

The discontinuity cost may be a cost reflecting the constraint that the NoMs of the two pixels are the same when the three-dimensional distances of each of two spatially neighboring pixels in one depth image are similar to each other. Here, the 3D distance of the pixel may be a distance between the camera generating the depth image including the pixel and the 3D point corresponding to the pixel.

λ may be a balancing coefficient that is a value that fits a unit between the discontinuity cost and the data cost or a unit between the discontinuity cost and the MRF energy. λ may have a constant value.

That is, the processor 140 may determine the MRF energy for the first depth image based on the data cost of each pixel in the first depth image and the discontinuity cost of adjacent pixels in the first depth image, and in the second depth image. The MRF energy for the second depth image may be determined based on the data cost of each of the pixels and the cost of discontinuity of adjacent pixels in the second depth image.

The processor 140 may calculate the NoM of each pixel of the first depth image for minimizing the MRF energy for the first depth image, and all pixels of the second depth image for minimizing the MRF energy for the second depth image. Each NoM can be calculated.

D ₁ ( m _i ) and D ₂ ( n _i ) may be defined according to Equation 9 below. The processor 140 may use a function defined according to Equation 9 below to calculate a data cost for each of M _i and N _i .

x _{1 i} may represent a pixel whose index of the first depth image is i . x _{2 i} may represent a pixel whose index of the second depth image is i . x _{1 k} may represent a pixel whose index of the first depth image is k . x _{2 k} may represent a pixel whose index of the second depth image is k . x _{1 i} and x _{2 k} may be the first pixel and the second pixel described above with reference to FIG. 4, respectively. That is, x _{1 i} and x _{2 k} may be pixels corresponding to each other. In addition, x _{1 k} and x _{2 i} may be the first pixel and the second pixel described above with reference to FIG. 4, respectively. That is, x _{1 k} and x _{2 i} may be pixels corresponding to each other.

X _{1 i} (m _i) when one NoM yi m _i of x _{i 1,} may represent a three-dimensional point corresponding to a, X _{1 i} calculated on the basis of m _i. X _{1 k} (m _k) is when NoM yi m _k of the _k x _1, can represent the three-dimensional point corresponding to a, x _{1 k} calculated on the basis of m _k. X _{2 i} (n _i) may represent a three-dimensional point corresponding to the time of NoM x _{2 i} n _i day, calculated on the basis of n _i, x _{2 i.} X _{2 k} (n _k) may represent the three-dimensional point corresponding to a, x _{2 k} calculated by the NoM of ₂ x _k based on, when the n _k n _k.

δ may be a function that gives a limit of the final distance error. δ may be a robust potential function. In Equation 9, data costs D ₁ and D ₂ may each represent a form to which a robust latent function δ is applied. δ may serve to replace a larger calculated value with a threshold when the distance between two 3D points is calculated larger than a predetermined threshold by occlusion or noise.

In Equation 9, when the value of the calculated data cost is larger than the threshold, the value of the data cost may be changed to the threshold by the robust potential function. The processor 140 may change the value of the data cost to a threshold when the value of the calculated data cost is greater than the threshold by using the robust potential sum.

As shown in Equation (9), the processor 140 may generate a first distance between the first 3D point reconstructed by applying NoM to the first pixel and the second 3D point reconstructed by applying NoM to the second pixel. The data cost can be calculated by calculating the determined NoM of the pixel and the determined NoM of the second pixel.

V ₁ ( m _i , m _j ) and V ₂ ( n _i , n _j ) may be defined according to Equation 10 below. The processor 140 may use a function defined according to Equation 10 below to calculate the discontinuity cost of each of the ordered pairs ( m _i , m _j ) and ( n _i , n _j ).

Here, τ may be a threshold value. For example, when the distance between two 3D points X _{1 i} and X _{1 j} is greater than or equal to the threshold τ, the value of the discontinuity cost V ₁ ( m _i , m _j ) may be zero. τ may have a constant value.

ν may be a function or constant for determining V ₁ ( m _i , m _j ) based on the distance between X _{1 i} and X _{i j} . ν may be a function or constant for determining V ₂ ( m _i , m _j ) based on the distance between X _{2 i} and X _{2 j} . ν may be a function for generating an output value inversely proportional to the distance between the input values X _{1 i} and X _{i j} or the distance between X _{2 i} and X _{2 j} . That is, ν is If the distance between X _{1 i} and X _{i j} or the distance between X _{2 i} and X _{2 j} is short and the values of m _i and m _j are different from each other, V ₁ ( m _i , m _j ) or V ₂ ( n _i , n _j ) can be a function that has a large value.

6 illustrates a role of discontinuity cost according to an example.

The discontinuity cost V is the value of the NoM values of two neighboring pixels to maintain the similarity when there are two spatially neighboring pixels in one depth image and the three-dimensional distances of the two neighboring pixels are similar to each other. May also indicate constraint constraints to be the same.

Pixel x _{1 a} and pixel x _{1 b} may be neighboring pixels in the first depth image. Pixel x _{2 a} and pixel x _{2 b} may be neighboring pixels in the second depth image.

x _{1 a} and x _{2 a} may correspond to each other. x _{2 a} and x _{2 b} may correspond to each other.

In Figure 6, the 3D point of 3D points corresponding to the trajectory (620), x _{2 a} of the X _{1 b} 3D points corresponding to the trajectory (610), x _{1 b} of X _{1 a} which corresponds to x _{1 a} X The trajectory 630 of _{2 a} and the trajectory 640 of X _{2 b} , which is a 3D point corresponding to x _{2 b} , are shown.

X _{1 a} , X _{1 b} , X _{2 a} , and X _{2 b} when NoM is 0, 1, and 2 in a direction from a near point to a far point from the first camera 110 or the second camera 120. Are represented by circles or triangles, respectively.

According to FIG. 6, the values of NoMs of all of X _{1 a} , X _{1 b} , X _{2 a} , and X _{2 b} are two. Therefore, the reconstructed depths of neighboring pixels in one depth image may also have consistency.

Reconstruction of the depth value of the depth image using the MRF energy described above with reference to FIG. 6 may be performed separately for each of the depth images. Therefore, in operation 360 described above with reference to FIG. 3, the first depth image in which the depth value is restored and the second depth image in which the depth value is restored may be inconsistent with each other.

In order to resolve the above inconsistency, in step 360, the processor 160 may update the reconstructed depth value of the other depth image by using the depth image of one of the depth images from which the two depth values are reconstructed. Can be. That is, the processor 160 may update the depth values of the first depth image and the second depth image in which the depth values are restored by the operation 350, using the depth image in which the other depth values are restored.

The processor 140 may obtain consistent depth images by updating depth values of both the first depth image in which the depth value is restored and the second depth image in which the depth value is restored.

When the depth values of the first depth image and the second depth image are reconstructed, the processor 140 may generate a first depth image based on Equation 11 below to make the first depth image consistent with the second depth image. The restored depth value of the one depth image may be updated.

Here, F ₁ ( m _i ) may be a data cost for updating the depth value of the first depth image.

m _i may be the NoM of pixel x _{1 i} whose index of the first depth image is i . n _k * may be NoM of pixels x _{2 k} having an index k of the second depth image from which the depth value is reconstructed. n _i * may be the NoM of the pixel at which the index of the second depth image determined in step 340 is k . Pixel x _{1 i} and pixel x _{2 k} may be pixels corresponding to each other representing the same 3D point.

X _{2 k} ( n _k *) may represent a remotely restored 3D point corresponding to X _{1 i} ( m _i ).

The processor 140 may update the depth value of the pixel by determining the NoM m _i of the pixel to minimize F ₁ ( m _i ) for the pixel whose index of the first depth image is i . F ₁ ( m _i ) may be a data cost for a pixel of the first depth image calculated based on the depth value of the second depth image in which the depth value is reconstructed.

When the reconstructed depth value of the second depth image is correct, F ₁ ( m _i ) of Equation 11 may replace D ₁ ( m _i ). However, the reconstructed depth value of the second depth image may not necessarily be guaranteed to be accurate. In operation 362, the processor 140 restores the depth value based on the restored depth value of the second depth image such that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other. It is possible to repeatedly minimize the MRF energy for the first depth image. The processor 140 may update the restored depth value of the first depth image by repeatedly minimizing the MRF energy.

The processor 140 may repeatedly minimize the MRF energy based on Equation 12 below.

Here, T may be the maximum number of iterations. T may be an integer of 0 or more. t may be the current number of iterations. t may be an integer of 0 or more and T or less. Equation 11 may represent blending of D ₁ ( m _i ) and F ₁ ( m _i ).

D ₁ ^t ( m _i ) may be the data cost for the NoM of pixel x _{1 i} whose index of the first depth image at the t- th iteration is i . The processor 140 may update the depth value of x _{1 i} by determining NoM m _i of x _{1 i} to minimize D ₁ ^t ( m _i ).

When the value of t is 0, the processor 140 may use D ₁ ( m _i ) described above with reference to Equation 9 to restore the depth value of the first depth image. When the value of t is 1 or more, the processor 140 may use F ₁ ( m _i ) described above with reference to Equation 11 together with D ₁ ( m _i ) to restore the depth value of the first depth image. .

The above description of updating the depth value may also be applied to the second depth image in which the depth value is restored.

The processor 140 may update the reconstructed depth value of the second depth image based on Equation 13 below to make the second depth image consistent with the first depth image.

Here, F ₂ ( n _i ) may be a data cost for updating the depth value of the second depth image.

n _i may be the NoM of pixel x _{2 i} whose index of the second depth image is i . m _k ^* may be NoM of pixels x _{1 k} having an index k of the first depth image from which the depth value is restored. m _k ^* may be the NoM of the pixel at which the index of the first depth image determined in step 340 is k . Pixel x _{2 i} and pixel x _{1 k} may be pixels corresponding to each other representing the same 3D point.

X _{1 k} ( m _k ^* ) may represent a remotely restored 3D point corresponding to X _{2 i} ( n _i ).

The processor 140 may update the depth value of the pixel by determining the NoM n _i of the pixel to minimize F ₂ ( n _i ) with respect to the pixel whose index of the second depth image is i . F ₂ ( n _i ) may be a data cost for a pixel of the second depth image calculated based on the depth value of the first depth image in which the depth value is reconstructed.

In operation 364, the processor 140 restores the depth value based on the restored depth value of the first depth image such that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other. It is possible to repeatedly minimize the MRF energy for the second depth image.

The processor 140 may repeatedly minimize the MRF energy based on Equation 14 below.

D ₂ ^t ( m _i ) may be the data cost for the NoM of pixel x _{2 i} whose index of the second depth image at the t- th iteration is i . The processor 140 may update the depth value of x _{2 i} by determining NoM m _i of x _{2 i} to minimize D ₂ ^t ( m _i ).

When the value of t is 0, the processor 140 may use D ₂ ( m _i ) described above with reference to Equation 9 to restore the depth value of the second depth image. When the value of t is 1 or more, the processor 140 may use F ₂ ( m _i ) described above with reference to Equation 13 together with D ₂ ( m _i ) to restore the depth value of the second depth image. .

In operation 360, the processor 140 repeatedly minimizes the MRF energy function based on Equation 15 below to ensure that the first depth image and the second depth image are consistent with each other. The depth value of the depth image may be updated.

Here, E ₁ ^t and E ₂ ^t may represent MRF energy for the first depth image and MRF energy for the second depth image in the t- th iteration, respectively.

7 illustrates depth images to which a depth value update is applied according to an example.

The first updated depth image 710 is a depth generated by updating the depth value of the first reconstructed depth image 510 using the depth value of the second reconstructed depth image 520 described above with reference to FIG. 5. It is a video.

The second updated depth image 720 is a depth generated by updating the depth value of the second reconstructed depth image 520 using the depth value of the first reconstructed depth image 510 described above with reference to FIG. 5. It is a video.

By the update described above with reference to FIG. 7, the first updated depth image 710 and the second updated depth image are compared with the consistency between the first reconstructed depth image 510 and the first reconstructed depth image 520. Coherence between 720 may be further improved.

The above-described embodiments may restore the depth values of the depth images by using the depth images photographed by the plurality of cameras at the same time. In addition, the above-described embodiments may restore the depth values of the depth images even in a situation where a plurality of cameras move and an object moves independently of the movement of the plurality of cameras. In this case, the reconstructed depth value may be a depth value of a far distance more than the maximum acquisition distance.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

110: First camera
120: Second camera
130:
140: processing unit

Claims (17)

Receiving a first depth image and a second depth image;
Determining a number of mods (NoM) of modes of each of the corresponding pixels in the first depth image and the second depth image; And
Restoring a depth value of at least one depth image of the first depth image and the second depth image based on the determined NoM
Lt; / RTI >
And the corresponding pixels represent the same three-dimensional point. The method of claim 1,
Wherein the determining comprises:
Determining a NoM of each of the pixels in the first depth image that minimizes Markov Random Field (MRF) energy for the first depth image; And
Determining a NoM of each of the pixels in the second depth image that minimizes MRF energy for the second depth image
Depth image processing method comprising a. The method of claim 2,
MRF energy for the first depth image is determined based on the data cost of each pixel in the first depth image and the discontinuity cost of two adjacent pixels in the first depth image,
MRF energy for the second depth image is determined based on the data cost of each of the pixels in the second depth image and the discontinuity cost of two adjacent pixels in the second depth image,
The discontinuity cost is a cost reflecting the constraint that the NoMs of the two pixels are the same when the three-dimensional distances of the two spatially neighboring pixels in one depth image are similar to each other,
And the three-dimensional distance of the pixel is a distance between the camera that generated the depth image including the pixel and the three-dimensional point corresponding to the pixel. The method of claim 3,
A first pixel in the first depth image and a second pixel in the second depth image are pixels corresponding to each other;
The determined data cost of the first pixel that minimizes the distance between the first three-dimensional point reconstructed by applying NoM to the first pixel and the second three-dimensional point reconstructed by applying NoM to the second pixel. And calculating the NoM and the determined NoM of the second pixel. 5. The method of claim 4,
And when the value of the calculated data cost is greater than a threshold, the value of the data cost is changed to the threshold by a robust potential function. The method of claim 1,
Updating the reconstructed depth value of the first depth image by using the second depth image in which the depth value is reconstructed
Depth image processing method further comprising. The method according to claim 6,
Wherein the updating comprises:
A first depth from which the depth value is restored based on the restored depth value of the second depth image such that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other And updating the reconstructed depth value of the first depth image by repeatedly minimizing MRF energy for the image. The method of claim 1,
Generating the first depth image; And
Generating the second depth image
Further comprising:
The depth image processing method of claim 1, wherein the first depth image and the second depth image are generated by photographing a scene from at least one of different positions and different directions. A computer readable recording medium having recorded thereon a program for performing the method of any one of claims 1 to 8. A receiver configured to receive a first depth image and a second depth image; And
A value of the number of mods (NoM) of the respective pixels in the first depth image and the second depth image is determined, and the first depth image and the first depth image are determined based on the determined value of NoM. A processor to restore the depth value of at least one depth image of the two depth image
Lt; / RTI >
And the corresponding pixels represent the same three-dimensional point. The method of claim 10,
The processor determines a NoM of each of the pixels in the first depth image that minimizes Markov Random Field (MRF) energy for the first depth image, and minimizes the MRF energy for the second depth image. And a depth image processing apparatus for determining the NoM of each pixel in the second depth image. 12. The method of claim 11,
The processor determines an MRF energy for the first depth image based on a data cost of each pixel in the first depth image and a discontinuity cost of two adjacent pixels in the first depth image, and determines the MRF energy for the second depth image. Determine MRF energy for the second depth image based on the data cost of each of the pixels in the pixel and the cost of discontinuity of two adjacent pixels in the second depth image,
The discontinuity cost is a cost reflecting the constraint that the NoMs of the two pixels are the same when the three-dimensional distances of the two spatially neighboring pixels in one depth image are similar to each other,
And the three-dimensional distance of the pixel is a distance between the camera generating the depth image including the pixel and a three-dimensional point corresponding to the pixel. The method of claim 12,
Wherein,
The determined NoM and the first of the first pixel to minimize the distance between the first three-dimensional point reconstructed by applying NoM to the first pixel and the second three-dimensional point reconstructed by applying NoM to the second pixel And calculating the data cost by calculating the determined NoM of two pixels. The method of claim 13,
And the processor is configured to change the value of the data cost to the threshold when the value of the calculated data cost is greater than the threshold by using a robust potential function. The method of claim 10,
And the processor is configured to update the reconstructed depth value of the first depth image by using the second depth image in which the depth value is reconstructed. 16. The method of claim 15,
The processor may further include a first depth in which the depth value is restored based on a restored depth value of the second depth image such that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other. And updating the reconstructed depth value of the first depth image by repeatedly minimizing MRF energy for the image. The method of claim 10,
A first camera generating the first depth image by photographing a scene; And
A second camera generating the second depth image by photographing the scene
Further comprising:
And the first camera and the second camera photograph the scene at different positions and / or directions.

Images