KR101874486B1 - Method and apparatus for recovering depth value of depth image - Google Patents

Abstract

A method and apparatus for reconstructing a depth value of a depth image is provided. The number of modes of each of the corresponding pixels in the plurality of depth images is determined. The corresponding pixels are pixels representing the same three-dimensional point. The number of modes is determined to a value that minimizes the Markov random field energy. When the depth value of one depth image is restored, the depth value of another depth image is updated by using the restored depth value.

Description

METHOD AND APPARATUS RECOVERING DEPTH VALUE OF DEPTH IMAGE BACKGROUND OF THE INVENTION [0001]

The following embodiments relate to a method and apparatus for restoring a depth value in a depth image, and more particularly, a method and apparatus for calculating a depth value of a pixel in a depth image photographed by a ToF camera.

Time-of-Flight (ToF) Depth A camera is a camera that provides 2.5-dimensional (D) information about a scene. Here, 2.5D means that depth information is provided only for the visible surface.

The ToF depth camera includes a light emitting diode (LED) for irradiating infrared (IR) light and a sensor for detecting light. The ToF depth camera calculates the distance between the ToF depth camera and the object by measuring the time that the IR light emitted from the LED is reflected on the object and reflected back to the sensor.

The illuminated IR light is modulated at a frequency f . That is, the IR light is light having the frequency f . ToF t _{time-of-flight} is indirectly calculated by measuring the intensity of the light returning to the sensor for either two phases or four phases.

According to one aspect, there is provided a method comprising: receiving a first depth image and a second depth image; determining a number of modes (Number of Mods: NoM) of each of the corresponding pixels in the first depth image and the second depth image And restoring a depth value of at least one of the first depth image and the second depth image based on the determined NoM, wherein the corresponding pixels represent the same three-dimensional point , A depth image processing method can be provided.

Wherein the determining comprises: determining NoM of each of the pixels in the first depth image that minimizes Markov Random Field (MRF) energy for the first depth image; Determining NoM of each of the pixels in the second depth image that minimizes MRF energy.

The MRF energy for the first depth image may be determined based on the data cost of each of the pixels in the first depth image and the discontinuity cost of two adjacent pixels in the first depth image.

The MRF energy for the second depth image may be determined based on the data cost of each of the pixels in the second depth image and the discontinuity cost of two adjacent pixels in the second depth image.

The discontinuity cost may be a cost reflecting the constraint that the NoMs of the two pixels are equal to each other if the three-dimensional distances of each of two spatially adjacent pixels in one depth image are similar to each other.

The three-dimensional distance of the pixel may be the distance between the three-dimensional point corresponding to the camera and the pixel that generated the depth image including the pixel.

The first pixel in the first depth image and the second pixel in the second depth image may correspond to each other.

Wherein the data cost is determined by minimizing the distance between the restored first three-dimensional point by applying NoM to the first pixel and the restored second three-dimensional point by applying NoM to the second pixel, NoM and the determined NoM of the second pixel.

If the value of the calculated data cost is greater than the threshold value, the value of the data cost may be changed to the threshold value by a robust potential function.

The depth image processing method may further include updating the reconstructed depth value of the first depth image using the second depth image whose depth value is reconstructed.

Wherein the updating is based on the restored depth value of the second depth image so that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other, The reconstructed depth value of the first depth image can be updated by repeatedly minimizing the MRF energy for the reconstructed first depth image.

The depth image processing method may further include generating the first depth image and generating the second depth image.

The first depth image and the second depth image may be generated by photographing the scene at different positions and / or directions from each other.

According to another aspect of the present invention, there is provided an image processing apparatus including a receiver for receiving a first depth image and a second depth image, a value of a number of modes (Number of Mods: NoM) of corresponding pixels in the first depth image and the second depth image, And a processor for restoring a depth value of at least one depth image of the first depth image and the second depth image based on the determined value of NoM, point of the depth image processing apparatus.

The processor may determine the NoM of each of the pixels in the first depth image that minimizes the Markov Random Field (MRF) energy for the first depth image, and minimize the MRF energy for the second depth image Of the pixels in the second depth image.

Wherein the processing unit is able to determine the MRF energy for the first depth image based on a data cost of each of the pixels in the first depth image and a discontinuity cost of two adjacent pixels in the first depth image, The MRF energy for the second depth image may be determined based on the data cost of each of the pixels in the image and the discontinuity cost of two adjacent pixels in the second depth image.

Wherein the processing unit is configured to minimize the distance between the restored first three-dimensional point by applying NoM to the first pixel and the restored second three-dimensional point by applying NoM to the second pixel, And calculating the data cost by calculating the determined NoM of the second pixel.

The processor may use a robust potential function to change the value of the data cost to the threshold value if the calculated data cost is greater than a threshold value.

The processing unit may update the reconstructed depth value of the first depth image using the second depth image whose depth value is reconstructed.

Wherein the processing unit is configured to reconstruct the first depth image based on the reconstructed depth value of the second depth image so that the reconstructed depth value of the first depth image and the reconstructed depth value of the second depth image are consistent with each other, The reconstructed depth value of the first depth image can be updated by repeatedly minimizing the MRF energy for the image.

The depth image processing apparatus may further include a first camera that generates the first depth image by photographing the scene, and a second camera that generates the second depth image by photographing the scene.

Wherein the first camera and the second camera capture the scene in different positions and / or directions from each other.

1 is a block diagram of a depth image generating apparatus according to an exemplary embodiment of the present invention.
FIG. 2 illustrates a distance calculation method using a plurality of ToF depth cameras according to an example.
3 is a flowchart illustrating a depth value restoration method according to an exemplary embodiment of the present invention.
4 illustrates a distance change of the 3D point according to the NoM value according to an example.
FIG. 5 illustrates reconstructed depth images using a data cost function according to an example.
Figure 6 illustrates the role of the discontinuity cost according to one example.
FIG. 7 shows depth images to which a depth value update according to an example is applied.

In the following, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

1 is a block diagram of a depth image generating apparatus according to an exemplary embodiment of the present invention.

The depth image processing apparatus 100 may include a first camera 110, a second camera 120, a receiving unit 130, and a processing unit 140.

The first depth image can be generated by taking a scene of the first camera 110. The second camera 120 can generate the second depth image by photographing the scene. The first camera 110 and the second camera 120 may respectively generate the first depth image and the second depth image by photographing the scene in different positions and / or directions from each other. The first depth image and the second depth image can be generated by photographing the scene at different positions and directions.

The first camera 110 and the second camera 120 may each be a ToF depth camera. The first camera 110 and the second camera 120 may be stereotyped ToF depth cameras.

The receiving unit 130 may receive the first and second depth images.

The processing unit 140 may restore the depth value of the first depth image and the depth value of the second depth image. Here, the depth value of the depth image may mean the depth value of each pixel in the depth image. Restoration of depth values may mean changing an inaccurate or insufficient depth value to an accurate or more accurate depth value.

The depth value of the pixel in the depth image represents the distance between the 3D image (3D) point of the scene corresponding to the camera and the image of the depth image. Thus, the closer the distance and the depth value of the pixel are, the more the depth value of the pixel can be regarded as more accurate.

FIG. 2 illustrates a distance calculation method using a plurality of ToF depth cameras according to an example.

The plurality of ToF depth cameras may be the first camera 110 and the second camera 120.

The processing unit 140 can calculate the distance between the ToP depth camera and the object based on Equation (1) below.

Where d may be the distance between the ToP depth camera and the object. c can be the speed of light. t _{time-of-flight} may be the _time from when the light is illuminated from the LED of the ToP depth camera to the object and reflected back to the sensor of the ToF depth camera. Hereinafter, light irradiated from the LED is irradiated with light, light reflected by the object and reflected by the object is outlined as reflected light. The light emitted from the LED may be light modulated at a frequency f and may be an IR light.

t _{time-of-flight} can be indirectly calculated, for example, by the processing unit 140 by measuring the intensity of light returning to the sensor for two or four phases.

FIG. 2 may represent a manner of calculating distances by using the intensity of light measured at two different phases. Here, the two different phases may be 0 ° phase and 180 ° phase, respectively.

The signal received by the sensor can be expressed by the following equation (2). Here, the received signal may be reflected light received by the sensor.

는 조사 광 및 반사 광 간의 위상차일 수 있다.Where A can represent the amplitude of the received signal. t can time out.

May be a phase difference between the irradiation light and the reflected light.

The processing unit 140 may calculate t _{time-of-flight} based on Equation (3) below.

The received signal may be mixed with a sine wave sin (2? Ft ) and a cosine wave cos (2? Ft ) by a first mixer 210 and a second mixer 220. The signals generated by the mixing may pass through a first low pass filter 212 and a second low pass filter 214, respectively. A first correlated signal B and a second correlated signal C may be generated by passing the signals generated by the mixing through the first low pass filter 212 and the second low pass filter 214, respectively. Here, the sine wave sin (2? Ft ) and the cosine wave cos (2? Ft ) are files corresponding to the first phase and the third phase, respectively.

B and C can be calculated based on Equation (4) below.

는 각각 하기의 수학식 5에 기반하여 계산될 수 있다.In addition, A and

Can be calculated based on Equation (5) below.

d _max can represent the maximum distance that can be measured without ambiguity. For example, d _max may be a distance corresponding to one period of the phase difference. d _max can be named a non-ambiguous distance range or a maximum acquisition distance.

When the phase difference between the irradiation light and the reflected light is larger than 2 ?, a value of 0 to 2? Smaller than the actual phase difference between the irradiation light and the reflected light can be measured as a phase difference.

d may be calculated based on Equation (6) below.

에 2πn을 더한 값인

+2πn를 이용하여 d+nd _max를 계산할 수 있다. 여기서, n은 0 이상의 정수일 수 있다.In order to calculate the distance d accurately, the processing unit 140 calculates the distance d

Lt; 2 >

D + nd _max can be calculated using +2 πn . Here, n may be an integer of 0 or more.

The value of n may not be determined by the depth measurement method of a single depth camera. Thus, the maximum distance that can be measured by the depth camera may be d _max , and the processing unit 140 may incorrectly calculate the depth value for an object farther than d _max from the depth camera as a shorter distance value than the actual . Here, the depth value for the object may be the depth value of the pixel representing the object among the pixels in the depth image.

That is, the depth value of the depth image photographed by the first camera 110 or the second camera 120 due to ambiguity may be an erroneous depth value in which the nd _max part is lost. The processing unit 140 may restore the depth value by adding the nd _max portion to the depth value of the depth image generated by the photographing.

The restored depth value described above with reference to FIG. 1 may be a remotely reconstructed depth value by determining the value of n. The reconstructed depth image may be a depth image having a reconstructed depth value by determining the value of n with respect to the depth value of each of the pixels in the depth image.

The above description with reference to equations (1) to (6) can be applied to each of the signals received by the plurality of cameras.

3 is a flowchart illustrating a depth value restoration method according to an exemplary embodiment of the present invention.

In step 310, the first camera 110 may generate a first depth image by photographing the scene.

In step 320, the second camera 120 may generate a second depth image by capturing the scene.

In steps 310 and 320, the first camera 110 and the second camera 120 may photograph the same scene in different positions and / or directions. That is, the first depth image and the second depth image can be generated by photographing the scenes at different positions and / or directions, respectively. The first depth image and the second depth image may have different viewpoints for the scene.

The receiving unit 130 may receive the first depth image from the first camera 110 and may receive the second depth image from the second camera 120 in step 330. [

In step 340, the processing unit 140 may determine the NoM of each of the corresponding pixels in the first depth image and the second depth image. Here, the corresponding pixels may represent the same three-dimensional point. That is, when an object in the scene is imaged by each of the first camera 110 and the second camera 120, the pixels in the first depth image representing the object and the pixels in the second depth image are pixels corresponding to each other .

The determination of NoM will be described in detail below with reference to Fig.

Step 340 may include steps 342 and 344.

In step 342, the processing unit 140 may determine the NoM of each of the pixels in the first depth image that minimizes the Markov Random Field (MRF) energy for the first depth image. The Markov random field energy is described in detail below with reference to FIG.

In step 344, the processing unit 140 may determine the NoM of each of the pixels in the second depth image that minimizes the MRF energy for the second depth image.

In step 350, the processing unit 150 may restore the depth value of at least one of the first depth image and the second depth image based on the determined NoM.

In operation 360, the processing unit 160 reconstructs the reconstructed depth value of the first depth image and the reconstructed depth value of the second depth image using the reconstructed first depth image and the reconstructed second depth image, The depth value can be updated. Step 360 may be considered as performing the optimization for the first depth image and the second depth image.

Step 360 may include steps 362 and 364. [

In operation 362, the processing unit 160 may update the reconstructed depth value of the first depth image using the reconstructed second depth image.

In step 364, the processing unit 160 may update the restored depth value of the second depth image using the first depth image whose depth value is restored.

4 illustrates a distance change of the 3D point according to the NoM value according to an example.

The first pixel may be one of the pixels in the first depth image, and the second pixel may be one of the pixels in the second depth image. The first pixel and the second pixel may be pixels corresponding to each other.

In step 330 described above with reference to FIG. 3, the depth value of the first pixel of the first depth image is d ₁ . And the depth value of the second pixel of the second depth image is d ₂ . As described above with reference to FIG. 2, d ₁ may be a value less than d _max ( f ₁ ) greater than or equal to zero, and d ₂ may be a value less than d _max ( f ₂ ) greater than or equal to zero.

d _max ( f ₁ ) may be the d _max of the signal sensed by the sensor of the first camera 110 when the light emitted from the first camera 110 is modulated at the frequency f ₁ . Where the d _max of the signal may be the d _max of the pixel in the depth image. Also, d _max ( f ₂ ) may be the d _max of the signal sensed by the sensor of the second camera 120 when the light emitted from the second camera 120 is modulated at the frequency f ₂ . Where the d _max of the signal may be the d _max of the pixel in the depth image. That is, each of the pixels in the first depth image has a depth value that is greater than or equal to zero and less than d _max ( f ₁ ), regardless of how far the point represented by each of the pixels actually is from the first camera 110 have. The depth value of the pixel of the first depth image may be the remaining value obtained by dividing the value of the actual distance between the first camera 110 and the point indicated by the pixel by d _max ( f ₁ ).

x ₁ may represent the first pixel. Also, x ₂ may represent the second pixel. X ₁ may represent a 3D point corresponding to x ₁ . X ₂ may represent a 3D point corresponding to x ₂ .

When the depth value of x ₁ generated by the first camera 110 is d ₁ , the distance between the first camera 110 and X ₁ may be d ₁ + n ₁ d _max ( f ₁ ). n ₁ may be x ₁ and No ₁ of X ₁ . n ₁ may be an integer equal to or greater than zero. Also, when the depth value of x ₂ generated by the second camera 120 is d ₂ , the distance between the second camera 120 and X ₂ may be d ₂ + n ₂ d _max ( f ₂ ). n ₂ may be x ₂ and No ₂ of X ₂ . n ₂ may be an integer of 0 or more.

X ₁ ( k ₁ ) may represent a 3D point corresponding to x ₁ when the value of n ₁ is k ₁ . That is, X ₁ ( k ₁ ) is determined by the position of the first camera 110, the direction of the first camera 110, d ₁ and d _max ( f ₁ ) when No ₁ of x ₁ is k ₁ It may represent a 3D points corresponding to x _1. X ₂ ( k ₂ ) may represent a 3D point corresponding to x ₂ when the value of n ₂ is k ₂ . That is, X ₂ ( k ₂ ) is determined by the position of the second camera 120, the direction of the second camera 120, d ₂ and d _max ( f ₂ ) when No ₂ of x ₂ is k ₂ x ₂ < / RTI >

The symbol '≡' can be used to indicate the distance of the 3D point. The formula ' X ₁ ( k ₁ ) ≡ d ₁ ' may indicate that the distance between the first camera 110 and X ₁ ( k ₁ ) is d ₁ . The formula ' d ₂ ≡ X ₂ ( k ₂ )' may indicate that the distance between the second camera 120 and X ₂ ( k ₂ ) is d ₂ .

In Figure _{4, X 1 (0),} X 1 (1) and X ₁ (2) was shown that each _{position, X 2 (0), X} 2 (1) and X _{2 (2)} showing the respective positions .

Processing unit 140 may restore the 3D points corresponding to X ₁ x ₁ on the basis of x _1. Here, restoration may mean to determine No ₁ of x ₁ . That is, the restoration may mean to determine one of X ₁ (0) to X ₁ ( m ₁ ) as the 3D point corresponding to x ₁ . Here, m ₁ may be the maximum value of k ₁ .

If the x ₁ and x ₂ correspond to each other, the 3D point reconstructed from x ₁ and the 3D point reconstructed from x ₂ may be points that are in the same position or within a certain distance.

In Fig. 4, the trajectory of X ₁ corresponding to x ₁ as the value of n ₁ is changed, and the trajectory of X ₂ corresponding to x ₂ as the value of n ₂ is changed is shown as a straight line. In Figure 4, when the NoM and NoM of X ₂ of X ₁ 2, respectively, X ₁ and X ₂ may intersect at one point. Therefore, according to Fig. 4, when n ₁ and n ₂ are 2, respectively, it can be seen that X ₁ and X ₂ are restored correctly. That is, the 3D point reconstructed from x ₁ may be X ₁ (2), and the 3D point reconstructed from x ₂ may be X ₂ (2).

Processor 140 on the basis of Equation (7) below can be calculated at the best NoM of X ₁ and X _2, respectively.

Here, R may be a rotation matrix indicating a rotation relationship between the first camera 110 and the second camera 120. R may be a rotation matrix representing the rotation from the direction of the first camera 110 to the direction of the second camera 120. R may be a rotation matrix of 3x3. R ^T may be a transpose matrix of R.

T may be a 3D translation vector representing the translation relationship between the first camera 110 and the second camera 120. [ T may be a 3D transform vector representing the translation of the position of the first camera 110 from the position of the second camera 120 to the position of the second camera 120.

R and T may be predetermined values given by extrinsic calibration between the first camera 110 and the second camera 120, respectively.

In Equation (7), m may be NoM of the 3D point X ₁ corresponding to one pixel x ₁ in the first depth image photographed by the first camera 110. n may be NoM of the 3D point X ₂ corresponding to one pixel x ₂ in the second depth image photographed by the second camera 120.

X ₂ ( n ) may represent a 3D point when X ₁ ( m ) is projected onto the second depth image, and NoM of the pixel projected on the second depth image is n . Here, the pixel where X ₁ ( m ) is projected on the second depth image may be X ₂ . X ₁ ( m ) may represent a 3D point when X ₂ ( n ) is projected on the first depth image, and NoM of the pixel projected on the first depth image is m . Here, the pixel projected on the second depth image by X ₂ ( n ) may be X ₁ .

m ^* and n ^* may be, NoM optimum value to minimize the distance between the 3D point generated by reconstructing a 3D point, and X ₂ generated by reconstructing the X _1, respectively.

The functions of the above-mentioned Equation (7) are referred to as a data cost function.

FIG. 5 illustrates reconstructed depth images using a data cost function according to an example.

In step 340 described above with reference to FIG. 3, the processing unit 140 applies the above-described data cost function to the first depth image and the second depth image to generate a first reconstructed depth image 510 and a second reconstructed depth image 510, The depth image 520 can be generated.

If the viewpoint of the first camera 110 and the viewpoint of the second camera 120 are the same, the first depth image photographed by the first camera 110 and the first depth image photographed by the second camera 120 There is no occlusion between the second depth images. Thus, for each of all pixels in a depth image, the corresponding pixel may be in another depth image.

However, since the first camera 110 and the second camera 120 have different positions and / or directions with respect to the scene, for all the pixels in one depth image, the corresponding pixels are present in the other depth image I can not. That is, there may be occlusion between the first depth image and the second depth image. Also, due to the noise existing in each of the first depth image and the second depth image, the value of NoM calculated by considering the data cost function may not always be accurate.

The illustrated first and second reconstructed depth images 510 and 520 may illustrate that the reconstructed depth image may not always be accurate due to the occlusion and noise described above.

MRF energy can be defined for each of the first depth image and the second depth image, as shown in Equation (8) below. The processing unit 140 may use the MRF energy function defined according to the following Equation 8 to determine the NoM of each of the pixels in the first depth image and the NoM of each of the pixels in the second depth image.

Where E ₁ may be the MRF energy for the first depth image. E ₂ may be the MRF energy for the second depth image.

i and j may be indexes of pixels in the first depth image or the second depth image, respectively.

N may be a set of all neighboring pixel-pairs within the first depth image or the second depth image. That is, ( i , j ) may represent an ordered pair of indices of two neighboring pixels within the first depth image or the second depth image. The neighboring pixels of the index i may be four or eight.

m _i may be NoM of a pixel whose index of the first depth image is i . and m _j may be NoM of the pixel whose index of the first depth image is j . and n _i may be NoM of a pixel whose index of the second depth image is i . and n _j may be NoM of the pixel whose index of the second depth image is j .

D ₁ may be the data cost for the first depth image. D ₂ may be the data cost for the second depth image. D ₁ ( m _i ) may be the data cost for the NoM of the pixel whose index of the first depth image is i . D ₂ ( n _i ) may be the data cost for the NoM of the pixel whose index of the second depth image is i .

V ₁ may be the discontinuity cost of the first depth image. V ₂ may be the discontinuity cost of the second depth image. _{V 1 (m i, m j} ) may be a discontinuity in the cost NoM of the pixel, the index of the first pixel of a depth image and NoM first depth image of the index i of j. _{V 2 (n i, n j} ) may be a discontinuity in the cost NoM of the pixel, the index of the second depth of the image pixel of the index i NoM and a second depth image-j.

The discontinuity cost may be a cost reflecting the constraint that the NoMs of the two pixels are equal to each other if the three-dimensional distances of each of the spatially neighboring two pixels within one depth image are similar to each other. Here, the three-dimensional distance of the pixel may be the distance between the three-dimensional point corresponding to the camera and the pixel that generated the depth image including the pixel.

lambda can be a discontinuity cost and a balancing coefficient that is a value that matches the unit or discontinuity cost between the data costs and the unit of MRF energy. lambda can have a constant value.

That is, the processing unit 140 can determine the MRF energy for the first depth image based on the data cost of each of the pixels in the first depth image and the discontinuity cost of adjacent pixels in the first depth image, The MRF energy for the second depth image may be determined based on the data cost of each of the pixels and the discontinuity cost of adjacent pixels in the second depth image.

The processing unit 140 may calculate NoM of each pixel of the first depth image that minimizes the MRF energy for the first depth image and may calculate all pixels of the second depth image that minimize the MRF energy for the second depth image, Can be calculated.

D ₁ ( m _i ) and D ₂ ( n _i ) can be defined according to Equation (9) below. The processing unit 140 may use a function defined according to the following equation (9) to calculate the data cost for each of M _i and N _i .

x _{1 i} may represent a pixel whose index of the first depth image is i . x _{2 i} may represent a pixel whose index of the second depth image is i . x _{1 k} may represent a pixel whose index of the first depth image is k . x _{2 k} may represent a pixel whose index of the second depth image is k . x _{1 i} and x _{2 k} may be the first pixel and the second pixel described above with reference to FIG. 4, respectively. That is, x _{1 i} and x _{2 k} may be pixels corresponding to each other. Further, x _{1 k} and x _{2 i} may be the first pixel and the second pixel described above with reference to FIG. 4, respectively. That is, x _{1 k} and x _{2 i} may be pixels corresponding to each other.

X _{1 i} (m _i) when one NoM yi m _i of x _{i 1,} may represent a three-dimensional point corresponding to a, X _{1 i} calculated on the basis of m _i. X _{1 k} (m _k) is when NoM yi m _k of the _k x _1, can represent the three-dimensional point corresponding to a, x _{1 k} calculated on the basis of m _k. X _{2 i} (n _i) may represent a three-dimensional point corresponding to the time of NoM x _{2 i} n _i day, calculated on the basis of n _i, x _{2 i.} X _{2 k} (n _k) may represent the three-dimensional point corresponding to a, x _{2 k} calculated by the NoM of ₂ x _k based on, when the n _k n _k.

[delta] may be a function that gives a limit of the final distance error. δ may be a robust potential function. In Equation (9), the data costs D ₁ and D ₂ can each represent a form in which the robust potential function δ is applied. delta may serve to substitute a larger calculated value for the threshold value when the distance between the two 3D points is calculated larger than a predetermined threshold value by occlusion or noise.

In Equation (9), when the value of the calculated data cost is greater than the threshold value, the value of the data cost can be changed to the threshold value by the robust latent function. The processing unit 140 may change the value of the data cost to a threshold value when the value of the data cost calculated by using the robust potential sum is larger than the threshold value.

As shown in Equation (9), the processing unit 140 includes a first 3D point obtained by applying NoM to the first pixel, and a first 3D point obtained by applying NoM to the second pixel by minimizing the distance between the restored second 3D point and the first 3D point The data cost can be calculated by calculating the determined NoM of the pixel and the determined NoM of the second pixel.

V ₁ ( m _i , m _j ) and V ₂ ( n _i , n _j ) can be defined according to Equation (10) below. The processing unit 140 may use a function defined according to the following Equation 10 to calculate the discontinuity cost of each ordered pair m _i , m _j and ( n _i , n _j ).

Wherein, τ is a threshold may be a value. For example, if the distance between the two 3D points X _{1 i} and X _{1 j} is equal to or greater than the threshold value τ, the value of the discontinuity cost V ₁ ( m _i , m _j ) can be zero. τ can have a constant value.

v may be a function or a constant for determining V ₁ ( m _i , m _j ) based on the distance between X _{1 i} and X _{i j} . v may be a function or a constant for determining V ₂ ( m _i , m _j ) based on the distance between X _{2 i} and X _{2 j} . v may be a function that produces an output value that is inversely proportional to the distance between the input values X _{1 i} and X _{i j} or the distance between X _{2 i} and X _{2 j} . In other words, ν is If the distance between X _{1 i} and X _{i j} or the distance between X _{2 i} and X _{2 j} is short and the values of m _i and m _j are different from each other, then V ₁ ( m _i , m _j ) or V ₂ ( n _i , n _j ) may have a large value.

Figure 6 illustrates the role of the discontinuity cost according to one example.

When the discontinuity cost V is two spatially neighboring pixels in one depth image and the three-dimensional distances of two neighboring pixels are similar to each other, the values of NoM of the two neighboring pixels Can be the same.

Pixels x _{1 a} and x _{1 b} may be neighboring pixels within the first depth image. Pixels x _{2 a} and x _{2 b} may be neighboring pixels within the second depth image.

x _{1 a} and x _{2 a} may correspond to each other. x _{2 a} and x _{2 b} may correspond to each other.

In Figure 6, the 3D point of 3D points corresponding to the trajectory (620), x _{2 a} of the X _{1 b} 3D points corresponding to the trajectory (610), x _{1 b} of X _{1 a} which corresponds to x _{1 a} X The trajectory 630 of _{2 a} and the trajectory 640 of X _{2 b} which is the 3D point corresponding to x _{2 b} are shown.

X _{1 a} , X _{1 b} , X _{2 a} , and X _{2 b} when NoM is 0, 1, and 2 in the direction from the first camera 110 or the second camera 120, Are indicated by circles or triangles, respectively.

Referring to Figure 6, the value of _{X 1 a, X 1 b,} X 2 a, _b and X ₂ are both NoM 2. Thus, the reconstructed depths of neighboring pixels in one depth image may also have consistency.

The reconstruction of the depth value of the depth image using the MRF energy described above with reference to FIG. 6 may proceed separately for each of the depth images. Accordingly, the first depth image in which the depth value is restored and the second depth image in which the depth value is restored in step 360 described above with reference to FIG. 3 may be inconsistent with each other.

In order to solve the above inconsistency, in step 360, the processing unit 160 updates the restored depth value of the other depth image using one depth image of the restored depth images . That is, the processing unit 160 may update the depth values of the first depth image and the second depth image in which the depth values are restored by using the depth image in which the different depth values are restored, in step 350.

The processing unit 140 can obtain the consistent depth images by updating the depth values of both the first depth image in which the depth value is restored and the second depth image in which the depth value is restored.

When the depth values of the first depth image and the second depth image are reconstructed, the processing unit 140 may calculate the first depth image and the second depth image based on Equation (11) below in order to make the first depth image consistent with the second depth image. The reconstructed depth value of the 1 depth image can be updated.

Here, F ₁ ( m _i ) may be a data cost for updating the depth value of the first depth image.

m _i may be NoM of the pixel x _{1 i} whose index of the first depth image is i . n _k * may be a NoM of a pixel x _{2 k} whose index of the second depth image whose depth value is reconstructed is k . n _i * may be NoM of the pixel whose index of the second depth image determined in step 340 is k . The pixel x _{1 i} and the pixel x _{2 k} may be pixels corresponding to each other that represent the same 3D point.

X _{2 k} ( n _k *) may represent a remotely reconstructed 3D point corresponding to X _{1 i} ( m _i ).

The processing unit 140 may update the depth value of the pixel by determining the NoM m _i of the pixel to minimize F ₁ ( m _i ) for the pixel whose index of the first depth image is i . F ₁ ( m _i ) may be a data cost for a pixel of the first depth image calculated based on the depth value of the restored second depth image.

If the reconstructed depth value of the second depth image is correct, F ₁ ( m _i ) in Equation (11) can replace D ₁ ( m _i ). However, the reconstructed depth value of the second depth image may not necessarily be guaranteed to be correct. In step 362, the processing unit 140 restores the depth value based on the restored depth value of the second depth image so that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other The MRF energy for the first depth image can be repeatedly minimized. The processing unit 140 may update the reconstructed depth value of the first depth image by iteratively minimizing the MRF energy.

The processing unit 140 may repeatedly minimize the MRF energy based on Equation (12) below.

Where T may be the maximum number of iterations. T may be an integer equal to or greater than zero. t may be the current number of iterations. t may be an integer of 0 or more and T or less. Equation (11) can represent the blending of D ₁ ( m _i ) and F ₁ ( m _i ).

D ₁ ^t ( m _i ) may be the data cost for the NoM of the pixel x _{1 i} where the index of the first depth image in the t- th iteration is i . The processing unit 140 can update the depth value of x _{1 i} by determining NoM m _i of x _{1 i} to minimize D ₁ ^t ( m _i ).

If the value of t is 0, the processing unit 140 can use D ₁ ( m _i ) described above with reference to Equation (9) to restore the depth value of the first depth image. If the value of t is equal to or greater than 1, the processing unit 140 may use F ₁ ( m _i ) described above with reference to Equation (11) to restore the depth value of the first depth image together with D ₁ ( m _i ) .

The description of updating the depth value described above can also be applied to the second depth image in which the depth value is restored.

The processing unit 140 may update the reconstructed depth value of the second depth image based on Equation (13) to make the second depth image consistent with the first depth image.

Here, F ₂ ( n _i ) may be a data cost for updating the depth value of the second depth image.

n _i may be NoM of the pixel x _{2 i} whose index of the second depth image is i . m _k ^* may be NoM of a pixel x _{1 k} whose index of the first depth image whose depth value is restored is k . m _k ^* may be NoM of a pixel whose index of the first depth image determined in step 340 is k . The pixel x _{2 i} and the pixel x _{1 k} may be pixels corresponding to each other that represent the same 3D point.

X _{1 k} ( m _k ^* ) may represent a remotely reconstructed 3D point corresponding to X _{2 i} ( n _i ).

The processing unit 140 may update the depth value of the pixel by determining the NoM n _i of the pixel to minimize F ₂ ( n _i ) for the pixel whose index of the second depth image is i . F ₂ ( n _i ) may be a data cost for a pixel of the second depth image calculated based on the depth value of the restored first depth image.

In step 364, the processing unit 140 restores the depth value based on the restored depth value of the first depth image so that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other The MRF energy for the second depth image can be repeatedly minimized.

The processing unit 140 may repeatedly minimize the MRF energy based on the following equation (14).

D ₂ ^t ( m _i ) may be the data cost for the NoM of the pixel x _{2 i} where the index of the second depth image in the t- th iteration is i . The processing unit 140 can update the depth value of x _{2 i} by determining the NoM m _i of x _{2 i} to minimize D ₂ ^t ( m _i ).

If the value of t is 0, the processing unit 140 can use D ₂ ( m _i ) described above with reference to Equation 9 to restore the depth value of the second depth image. If the value of t is equal to or greater than 1, the processing unit 140 can use F ₂ ( m _i ) described above with D ₂ ( m _i ) with reference to Equation 13 to restore the depth value of the second depth image .

In step 360, the processing unit 140 repeatedly minimizes the MRF energy function based on Equation (15) below so that the depth value of the first depth image and the depth value of the second depth image The depth value of the depth image can be updated.

Where E ₁ ^t and E ₂ ^t may represent the MRF energy for the first depth image and the MRF energy for the second depth image, respectively, at the t th iteration.

FIG. 7 shows depth images to which a depth value update according to an example is applied.

The first updated depth image 710 is generated by updating the depth value of the first reconstructed depth image 510 using the depth value of the second reconstructed depth image 520 described above with reference to FIG. It is a video.

The second updated depth image 720 is generated by updating the depth value of the second reconstructed depth image 520 using the depth value of the first reconstructed depth image 510 described above with reference to FIG. It is a video.

7, compared to the consistency between the first reconstructed depth image 510 and the first reconstructed depth image 520, the first updated depth image 710 and the second updated depth image 520, as compared to the consistency between the first reconstructed depth image 510 and the first reconstructed depth image 520, The consistency between the first and second regions 720 can be further improved.

The embodiments described above can recover depth values of the depth images by using depth images taken by a plurality of cameras at the same time. Also, in the above-described embodiments, a depth value of depth images can be restored even in a situation where a plurality of cameras are moved and an object moving independently of movement of the plurality of cameras exists. At this time, the restored depth value may be a depth value at a distance greater than the maximum acquisition distance.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

110: First camera
120: Second camera
130:
140:

Claims (17)

Receiving a first depth image representing a first distance measured between the first depth camera and an object and a second depth image representing a second distance measured between the second depth camera and the object;
Determining a number of modes (Number of Mods: NoM) of each of the corresponding pixels in the first depth image and the second depth image; And
Restoring at least one of the measured first distance and the measured second distance based on the determined NoM
Lt; / RTI >
The corresponding pixels representing the same three-dimensional point,
The NoM is an integer that is multiplied by the maximum acquisition distance corresponding to the frequency of the irradiation light in order to indicate the change in distance of the three-dimensional point
Depth image processing method. The method according to claim 1,
Wherein the determining comprises:
Determining NoM of each of the pixels in the first depth image that minimizes Markov Random Field (MRF) energy for the first depth image; And
Determining NoM of each of the pixels in the second depth image that minimizes MRF energy for the second depth image
/ RTI > 3. The method of claim 2,
Wherein the MRF energy for the first depth image is determined based on the data cost of each of the pixels in the first depth image and the discontinuity cost of two adjacent pixels in the first depth image,
The MRF energy for the second depth image is determined based on the data cost of each of the pixels in the second depth image and the discontinuity cost of two adjacent pixels in the second depth image,
The discontinuity cost is a cost reflecting the constraint that if the three-dimensional distances of two spatially neighboring pixels within one depth image are similar to each other, the NoMs of the two pixels are equal to each other,
Wherein the three-dimensional distance of the pixel is the distance between the three-dimensional point corresponding to the camera and the pixel that generated the depth image including the pixel. The method of claim 3,
Wherein the first pixel in the first depth image and the second pixel in the second depth image are pixels corresponding to each other,
Wherein the data cost is determined by minimizing the distance between the restored first three-dimensional point by applying NoM to the first pixel and the restored second three-dimensional point by applying NoM to the second pixel, And calculating the NoM and the determined NoM of the second pixel. 5. The method of claim 4,
Wherein the data cost value is changed to the threshold value by a robust potential function when the calculated data cost value is greater than a threshold value. The method according to claim 1,
Updating the restored depth value of the first depth image using the second depth image having the restored depth value
The depth image processing method further comprising: The method according to claim 6,
Wherein the updating comprises:
And a second depth image reconstructing unit for reconstructing the reconstructed depth value of the second depth image based on the restored depth value of the second depth image so that the restored depth value of the first depth image and the restored depth value of the second depth image are consistent with each other, Wherein the reconstructed depth value of the first depth image is updated by repeatedly minimizing MRF energy for the image. The method according to claim 1,
Generating the first depth image; And
Generating the second depth image
Further comprising:
Wherein the first depth image and the second depth image are generated by photographing the scene at least one of different positions and directions different from each other. 9. A computer-readable recording medium recording a program for performing the method of any one of claims 1 to 8. A receiver for receiving a first depth image representing a first distance measured between the first depth camera and an object, and a second depth image representing a second distance measured between the second depth camera and the object; And
Determining a value of a number of modes (Number of Mods: NoM) of each of the corresponding pixels in the first depth image and the second depth image, and determining the first distance and the second depth based on the determined NoM value. A processor for restoring at least one of the measured second distances;
Lt; / RTI >
The corresponding pixels representing the same three-dimensional point,
The NoM is an integer that is multiplied by the maximum acquisition distance corresponding to the frequency of the irradiation light in order to indicate the change in distance of the three-dimensional point
Depth image processing device. 11. The method of claim 10,
The processor determines the NoM of each of the pixels in the first depth image that minimizes the Markov Random Field (MRF) energy for the first depth image, and minimizes the MRF energy for the second depth image And determines the NoM of each of the pixels in the second depth image. 12. The method of claim 11,
Wherein the processing unit determines the MRF energy for the first depth image based on the data cost of each of the pixels in the first depth image and the discontinuity cost of two adjacent pixels in the first depth image, Determines the MRF energy for the second depth image based on the data cost of each of the pixels in the first depth image and the discontinuity cost of two adjacent pixels in the second depth image,
The discontinuity cost is a cost reflecting the constraint that if the three-dimensional distances of two spatially neighboring pixels within one depth image are similar to each other, the NoMs of the two pixels are equal to each other,
Wherein the three-dimensional distance of the pixel is the distance between the three-dimensional point corresponding to the camera and the pixel that generated the depth image including the pixel. 13. The method of claim 12,
Wherein the first pixel in the first depth image and the second pixel in the second depth image are pixels corresponding to each other,
Wherein,
Wherein the determined NoM of the first pixel to minimize the distance between the restored first three-dimensional point by applying NoM to the first pixel and the restored second three-dimensional point by applying NoM to the second pixel, And calculates the data cost by calculating the determined NoM of 2 pixels. 14. The method of claim 13,
Wherein the processor uses a robust potential function to change the value of the data cost to the threshold value if the calculated data cost is greater than a threshold value. 11. The method of claim 10,
Wherein the processing unit updates the restored depth value of the first depth image using the second depth image whose depth value is restored. 16. The method of claim 15,
Wherein the processing unit is configured to reconstruct the first depth image based on the reconstructed depth value of the second depth image so that the reconstructed depth value of the first depth image and the reconstructed depth value of the second depth image are consistent with each other, Wherein the reconstructed depth value of the first depth image is updated by repeatedly minimizing MRF energy for the image. 11. The method of claim 10,
A first camera for generating the first depth image by photographing a scene; And
A second camera for generating the second depth image by photographing the scene,
Further comprising:
Wherein the first camera and the second camera photograph the scene in different positions and / or directions from each other.

Images