KR101882421B1 - Apparatus and method for estimation disparity using visibility energy model - Google Patents

Abstract

Disclosed is a variation estimation apparatus and method based on a visibility energy model. The variation estimation apparatus includes an energy calculation unit for calculating energy related to stereo matching of a left image and a right image constituting a stereo image; A map generator for generating a visibility map for determining a disparity error between the left image and the right image using the energy; An energy recomputing unit for recomputing energy for an area where a visibility error exists in the visibility map; And a variation determination unit for determining a variation from the stereo image using the final energy of the left image and the right image.

Description

[0001] APPARATUS AND METHOD FOR ESTIMATION DISPARITY USING VISIBILITY ENERGY MODEL [0002]

One embodiment of the present invention relates to a variation estimation apparatus and method, and more particularly, to an apparatus and method for estimating a variation in a stereo image based on an energy model using a dynamic program.

In order to generate a three-dimensional image, it is necessary to restore depth information in a color image. In particular, techniques for restoring depth information in two images (a left image and a right image), such as a stereo image, are being studied. In order to restore the depth information, it is necessary to extract the variation between the left image and the right image.

In the conventional case, it is difficult to extract the variation in the occlusion region by applying the same energy model to all regions of the image. This is because the occlusion region does not have a matching region between the left image and the right image.

Therefore, it is necessary to extract a more accurate variation even in the occlusion region where there is no matching point between the left image and the right image.

An apparatus for estimating a disparity according to an embodiment of the present invention includes an energy calculation unit for calculating energy related to stereo matching of a left image and a right image constituting a stereo image; A map generator for generating a visibility map for determining a disparity error between the left image and the right image using the energy; An energy recomputing unit for recomputing energy for an area where a visibility error exists due to an error of the variation in the visibility map; And a variation determination unit for determining a variation from the stereo image using the final energy of the left image and the right image.

According to another aspect of the present invention, there is provided a variation estimation apparatus comprising: an energy calculation unit for calculating in parallel energy associated with stereo matching of a left image and a right image constituting a stereo image; A map generator for generating a visibility map for determining a disparity error between the left image and the right image using the energy; An energy recalculation unit for recalculating energy for an area where a visibility error exists due to an error of a variation in the visibility map of each of the left image and the right image; And a variation determining unit that performs local matching using the final energy of the left image and the right image, and determines a variation from a locally matched stereo image.

According to an aspect of the present invention, there is provided a variation estimation method, comprising: calculating energy related to stereo matching of a left image and a right image constituting a stereo image; Generating a visibility map for determining a disparity error between the left image and the right image using the energy; Recalculating energy for an area in which there is a visibility error due to an error in the variation in the visibility map; And determining the variation from the stereo image using the final energy of the left image and the right image.

According to another aspect of the present invention, there is provided a variation estimation method, comprising: parallelly calculating energy related to stereo matching of a left image and a right image constituting a stereo image; Generating a visibility map for determining a disparity error between the left image and the right image using the energy; Recalculating energy for an area in which there is a visibility error due to an error in the variation in the visibility map of each of the left image and the right image; Performing local matching using the final energy of the left and right images, and determining disparity from a locally matched stereo image.

According to an embodiment of the present invention, a new energy model is constructed for a region generating a visibility error, and a smoothing cost is preferentially applied to a smoothing cost rather than a matching cost of an energy model in a occlusion region where a visibility error occurs, The variation can be extracted accurately.

1 is a block diagram illustrating a variation estimation apparatus according to an embodiment of the present invention.
2 is a diagram for explaining stereo matching according to an embodiment of the present invention.
3 is a diagram illustrating a variation map estimated using dynamic programming according to an embodiment of the present invention.
FIG. 4 is a block diagram illustrating an energy calculation unit of the variation estimation apparatus of FIG. 1 according to an embodiment of the present invention.
5 is a view for explaining a process of calculating energy in a forward direction according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining a process of calculating energy by simultaneously considering a forward direction and a backward direction according to an embodiment of the present invention.
FIG. 7 is a block diagram illustrating the map generator of FIG. 1 according to an embodiment of the present invention.
FIG. 8 is a block diagram illustrating the map determination unit of FIG. 7 according to an embodiment of the present invention.
FIG. 9 is a block diagram illustrating an energy recalculation unit of FIG. 1 according to an embodiment of the present invention.
10 is a diagram showing a variation map derived using energy in a forward direction according to an embodiment of the present invention.
11 is a diagram showing a variation map derived using energy in the backward direction according to an embodiment of the present invention.
FIG. 12 is a view illustrating a transition map using a forward direction and a backward direction simultaneously according to an embodiment of the present invention and a transition map using an optimization method of dynamic programming.
13 is a diagram for explaining a valid region and an invalid region of a visibility map according to an embodiment of the present invention.
14 is a view for explaining local matching using energy of dynamic programming according to an embodiment of the present invention.
15 is a flowchart illustrating a variation estimation method according to an embodiment of the present invention.
16 is a flowchart illustrating a variation estimation method using local matching according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a variation estimation apparatus according to an embodiment of the present invention.

1, the variation estimation apparatus 100 may include an energy calculation unit 101, a map generation unit 102, an energy recalculation unit 103, and a variation determination unit 104. [ The variation estimation apparatus 100 according to an embodiment of the present invention can restore a more accurate depth by using visibility for determining an error of a disparity between a left image and a right image. In particular, the variation estimator 100 preferentially applies a smoothing cost over the matching cost to the area where the visibility error occurs, such as occlusion and variation estimation error areas, It is possible to estimate the variation more precisely on the basis of the difference. At this time, the variation estimation apparatus 100 can utilize an energy modeling method based on dynamic programming.

The energy calculation unit 101 may calculate the energy of each of the left image and the right image constituting the stereo image. The energy calculation unit 101 will be described in more detail with reference to FIG. Here, the left image and the right image are color images, and energy is related to stereo matching.

The map generating unit 102 may generate a visibility map for determining an error of disparity between the left image and the right image using energy. The map generating unit 102 will be described in more detail with reference to FIGS. 7 to 8. FIG.

The energy recalculation unit 103 can recalculate the energy for the region where the visibility error exists due to the error of the variation in the visibility map. The energy recalculation unit 103 will be described in detail with reference to FIG. At this time, the energy is not recomputed for the area where no visibility error exists. According to an embodiment of the present invention, a new energy model is presented in the occlusion region where the visibility error exists, and the smoothing cost is more preferentially applied than the matching cost, so that the variation can be more accurately estimated for the occlusion region.

The disparity determining unit 104 can determine the disparity from the stereo image using the final energy of the left image and the right image. Here, the final energy means the energy derived through the energy calculation unit 101 or the energy derived through the energy recollection unit 103.

2 is a diagram for explaining stereo matching according to an embodiment of the present invention.

Stereo matching means reconstructing the depth image with the color image of the multi-viewpoint. Stereo matching restores the depth image from the left and right images and extends to restoring the depth image from the multi-view image.

A method of reconstructing a depth image through stereo matching is as follows. Specifically, the mutation can be determined by finding a matching point in which the same three-dimensional point is separated from the multi-point image using the left image and the right image, which are images at different points in time. Then, the depth of the three-dimensional point can be estimated using the determined variation.

Referring to FIG. 2, stereo matching is performed by finding the same three-dimensional point in an image at another point of time, such as a left image and a right image, and estimating a variation, which is a distance between a three-dimensional point located in the left image and a three- .

There are global and local methods of acquiring three-dimensional information through stereo matching. The global method sets all the pixels of the image as nodes, and assumes that there are four neighboring pixels in each node. Then, the global scheme finds the optimal variation in the region where there are four neighboring pixels. The local method extracts the best-matching variation in the local region of the image without considering the entire image.

<Energy Calculation>

In order to perform stereo matching using the global method, energy needs to be calculated. The energy is described by the cost when a possible variation is assigned to each pixel. Specifically, when the number of horizontal pixels w, the number of vertical pixels h, and the number of possible variations d are input, a cost corresponding to w * h * d is obtained in the case of a solution capable of being derived. According to the global approach, a variance corresponding to the smallest value among these solutions can be estimated. That is, the technique of selecting the smallest value without calculating the number of all cases corresponding to w * h * d is an optimization method based on the global method.

In order to optimize the entire image at the same time, not only adjacent pixels but also distant pixels must be considered. However, as described above, when the energy of the region in which the four neighboring pixels exist is defined, the energy can be calculated as shown in Equation 1 below.

와 특정 픽셀의 주변에 위치한 이웃 픽셀과 관련된 에너지

를 이용하여 결정된다. 구체적으로,

는 왼쪽 이미지에서의 특정 픽셀과 오른쪽 이미지에서의 특정 픽셀 간의 컬러 차이 또는 변이 차이로 결정된다. 그리고,

는 왼쪽 이미지에서 특정 픽셀의 주변에 위치한 이웃 픽셀들과 오른쪽 이미지에서 특정 픽셀의 주변에 위치한 이웃 픽셀들 간의 컬러 또는 변이 차이로 결정된다. 여기서, d는 왼쪽 이미지와 오른쪽 이미지 간의 변이를 의미한다. 이러한 글로벌 에너지는 절대값 차이(absolute difference), 합계 절대값 차이(sum absolute difference)로 결정될 수 있다. 수학식 1에서,

는 특정 픽셀에 인접한 이웃 픽셀들은 유사한 변이를 가져야 하는 것을 의미한다.Equation (1) means a global energy model for stereo matching. In Equation (1), E (d) is global energy, the energy associated with a particular pixel

And the energy associated with neighboring pixels located around a particular pixel

. Specifically,

Is determined by the color difference or the difference between the specific pixel in the left image and the specific pixel in the right image. And,

Is determined by the difference in color or difference between neighboring pixels located around the specific pixel in the left image and neighboring pixels located around the specific pixel in the right image. Here, d represents the variation between the left image and the right image. This global energy can be determined by an absolute difference, a sum absolute difference. In Equation (1)

Means that neighboring pixels adjacent to a particular pixel should have similar variations.

<Dynamic Programming>

The optimization method of dynamic programming is to find an optimal value that minimizes the sub-problem by creating a sequential sub-problem without processing all the variables simultaneously.

들의 분포를 나타내는 에너지

의 최소값을 찾기 위해서는 아무런 풀이(solver)가 없는 경우, 모든 변수들

가 만들 수 있는 모든 에너지를 계산한 후에 가장 작은 값이 선택되어야 한다.Equation 2- (1) shows a method of minimizing energy using a dynamic program. variable

Energy representing the distribution of

If there is no solver to find the minimum of all variables,

The smallest value should be selected after all the energy that can be produced is calculated.

This method exponentially increases the amount of computation depending on the range of variables and variables. To solve this problem, dynamic programming in a global way does not find a value that minimizes energy by substituting all values. Specifically, the dynamic programming solves the sub-problem as shown in Equation 2- (2) since the energy of all the variables is treated as each energy as in Equation 2- (1).

의 에너지값을 저장하는 단계를 나타낸다. 마지막으로 수학식 2-(3)과 같이 각각의 단계를 최소화하는 변수로 수학식 2-(2)의 서브 문제의 최소값을 추적해 나가면, 수학식 2-(1)의

를 최소화 시킬 수 있는 글로벌 최적화 결과가 도출될 수 있다.Equation 2- (2) shows the variables before making the minimum value and the minimum value of the sub-

And the energy value of the energy stored in the memory is stored. Finally, if the minimum value of the sub-problem of Equation 2- (2) is traced as a parameter minimizing each step as shown in Equation 2- (3), then Equation 2- (1)

Can be obtained.

<Stereo matching using dynamic programming>

Equation (2) is applied to Equation (1).

In Equation (3), P (x, y) denotes a specific pixel, and P (x-1, y) denotes a neighboring pixel which is diagonally di away from a specific pixel. di is a variation between a specific pixel and a neighboring pixel, and the maximum value is d. Each pixel included in the image is a variable of Equation (2), and a variable can have any value within a Disparity Range. In the sub-problem of stereo matching, the variable of the pixel p (x) and the neighboring pixel p (x-1) located before is used as the energy function of the pixel p (x-1) Variations and energy can be saved. In this way, by using Equation 2- (3), the variation of each pixel that minimizes the entire image can be derived by determining each transition and energy for all the pixels from the first pixel to the last pixel of the image .

3 is a diagram illustrating a variation map estimated using dynamic programming according to an embodiment of the present invention.

The dynamic programming described above can simplify the global approach, but dynamic programming requires that only one neighbor pixel exists when dividing the entire image into sequential subproblems.

Then, in order to apply dynamic programming to stereo matching, it is necessary to perform matching on a scan line basis. According to the stereo matching in units of scan lines, a variation map in which an artifact occurs as shown in Fig. 3 can be derived.

To this end, according to an embodiment of the present invention, there is provided a method for removing artifacts while applying dynamic programming.

FIG. 4 is a block diagram illustrating an energy calculation unit of the variation estimation apparatus of FIG. 1 according to an embodiment of the present invention.

Referring to FIG. 4, the energy calculation unit 101 may include a second energy calculation unit 402 and an energy merging unit 403.

The first energy calculation unit 401 can calculate the energy in the forward direction for each of the left image and the right image. The second energy calculation unit 402 may calculate energy in the backward direction for each of the left image and the right image. In this case, the first energy calculation unit 401 and the second energy calculation unit 402 may calculate the energy using Equation (3). In this case, the first calculation unit 401 may calculate the energy in the forward direction as shown in FIG. 5, and the second calculation unit 402 may calculate the energy in the backward direction in the same manner as FIG. As described above, energy is related to stereo matching and is determined by the difference or color difference between the pixels present in the left image and the right image.

The energy merging unit 403 can combine the energy in the forward direction and the energy in the backward direction. The energy merging unit 403 can obtain energy by dynamic programming of the horizontal line as shown in FIG. At this time, the finally merged energy can be calculated by the following equation (4).

In Equation (4), E (p (x, y), d) means energy based on a variation d of a specific pixel p (x, y) and a specific pixel p And, E _f (p (x, y), d) the particular pixel p (x, y) given pixel p (x, y) and neighboring pixels p (x-1, y) the energy of a forward direction based on the meaning, E _b of (p (x, y), d) is based on a certain pixel p (x, y) given pixel p (x, y) and neighboring pixels p (x + 1, y) for the backward Direction energy. d _i is a variation between a specific pixel and a neighboring pixel, and the maximum value is d.

를 만족시키는 d값이

변수의 변이가 된다.Each pixel p (x, y) included in the image includes a variable di that can be varied by the variation range d, and the variable d _i can be assigned the minimum energy derived according to Equation (3). Here, if Equation 2- (2) is not used,

&Lt; / RTI &gt;

Variation of the variable.

In Equation (3), only the energy cost in one direction is calculated. In this case, an error due to the smoothness term may occur in a region where the variation suddenly changes. Here, the error means a circular area in Figs. 10 and 11. For this, according to an embodiment of the present invention, as a result of merging the energy in the forward direction and the energy in the backward direction for a specific pixel as shown in Equation (4), the result of the energy cost of the dynamic programming Can be utilized. Here, FIG. 10 shows the variation derived in the forward direction, and FIG. 11 shows the variation derived according to the backward direction. 12 shows a variation derived from the Disparity Map obtained by the optimization of dynamic programming, and the energy in the forward direction and the energy sum in the backward direction, which are derived from Equation (4). That is, referring to FIG. 12, it can be seen that artifacts, which are errors shown in FIGS. 10 and 11, do not occur.

FIG. 7 is a block diagram illustrating the map generator of FIG. 1 according to an embodiment of the present invention.

After obtaining the global energy in stereo matching, it is necessary to specify the area where the error occurs in order to recalculate the energy of the area where the visibility error occurs. The energy of the area in which no error occurs is used as it is because it is optimized. In order to derive an error in the visibility map, it is necessary to calculate the initial variation of the left and right images. After that, visibility check can identify the area where the error occurs. The visibility map is used to determine the error of the variation. If an error in the variation occurs due to the occlusion region, it is determined that there is a visibility error. At this time, the error of the variation can be divided into ON / OFF.

Referring to FIG. 7, the map generator 102 may include an initial variation calculator 701 and a map determiner 702.

을 이용하여 왼쪽 이미지와 오른쪽 이미지 각각의 초기 변이를 계산할 수 있다.The initial variation calculation unit 701 can calculate the initial variation of the left image and the right image using the energy of the left image and the energy of the right image. Specifically, the initial variation calculator 701 calculates

Can be used to calculate the initial variation of each of the left and right images.

The map determining unit 702 can determine the visibility map of each of the left image and the right image using the initial variation of the left image and the right image, respectively. The map determining unit 702 will be described in more detail with reference to FIG.

FIG. 8 is a block diagram illustrating the map determination unit of FIG. 7 according to an embodiment of the present invention.

Referring to FIG. 8, the map determination unit 702 may include a first map determination unit 801 and a second map determination unit 802.

The first map determination unit 801 can determine the visibility map for the left image using the initial variation of the right image. Conversely, the second map determination unit 802 may determine the visibility map for the right image using the initial variation of the left image. For example, the first map determining unit 801 and the second map determining unit 802 can determine the visibility map according to Equation (6).

은 왼쪽 이미지에서의 변이를 나타내고,

은 오른쪽 이미지에서의 변이를 나타낸다. 수학식 6에서, (1)은 왼쪽 이미지와 오른쪽 이미지가 서로 매칭되는 영역을 나타내고, (2)는 왼쪽 이미지와 오른쪽 이미지가 서로 매칭되지 않는 영역인 폐색 영역(occlusion area)을 의미한다. In Equation (6)

Represents the variation in the left image,

Represents the variation in the right image. In Equation (6), (1) represents an area where the left image and the right image match each other, and (2) represents an occlusion area, which is an area where the left image and the right image do not match with each other.

That is, when the left image and the right image are matched, the variation of the first pixel in the left image and the variation of the second pixel in the right image corresponding to the position shifted by the variation should be the same as in (1) of Equation 6 . And the Disocclusion Area of the left image is the area obscured by the foreground in the right image. Therefore, when a pixel corresponding to a position shifted by a variation of an unobstructed region of the left image is sought in the right image, the variation in the right image should be larger than the variation in the left image as shown in Equation 6 (2) .

However, if the variation in the left image is larger than that in Equation 6 (2), it is a portion to be seen in the right image, but is not actually seen in the right image. At this time, FIG. 13 shows a valid area (area where no error occurs) and an invalid area (area where an error occurs) of the visibility map according to an embodiment of the present invention. As described above, the visibility map is utilized to determine the error of the variation with respect to the left image and the right image.

FIG. 9 is a block diagram illustrating an energy recalculation unit of FIG. 1 according to an embodiment of the present invention.

Referring to FIG. 9, the energy recalculation unit 103 may include a diffusion direction determination unit 901 and an energy diffusion unit 902. The energy recalculation unit 103 can recalculate the energy of the area (VER: Visibility Error Region (VER)) where the visibility error occurs in the visibility map.

The energy recursive unit 103 may recalculate the energy according to the following equation (7).

가 추가된다. 스테레오 매칭에서는 왼쪽 이미지와 오른쪽 이미지 사이에 매칭이 되지 않는 폐색 영역이 존재할 수 있다. 수학식 1에 따르면, smoothness term인

로 인해서 픽셀 간의 컬러가 비슷하다면 특정 픽셀의 변이는 인접한 이웃 픽셀의 변이와 유사한 값을 가진다. 픽셀간 컬러가 크게 다르다면 특정 픽셀은 인접한 이웃 픽셀의 변이와 다른 값으로 할당하는 에너지 분포를 가진다. 그러나, smoothness term인

만으로 폐색 영역의 변이를 추정하는 것은 한계가 있으므로, 수학식 8에 따라 에너지를 재계산할 수 있다.Equation (7) is the visibility term &lt; RTI ID = 0.0 &gt;

Is added. In stereo matching, there may be occlusion areas that do not match between the left and right images. According to Equation (1), the smoothness term

The variation of a particular pixel has a value similar to the variation of adjacent neighboring pixels if the color between pixels is similar. If the inter-pixel color is largely different, a certain pixel has an energy distribution that assigns it to a value different from the variation of neighboring neighboring pixels. However, the smoothness term

The energy can be recalculated according to Equation (8) because there is a limit in estimating the variation of the occlusion region.

Equation (8) represents a stereo matching function based on dynamic programming in consideration of visibility. The visibility is independent of the neighboring pixel p (x-1, y) or p (x + 1, y) and is therefore considered only at a particular pixel p (x, y).

The diffusion direction determination unit 901 can determine the diffusion direction of the energy in the region where the visibility error exists using the initial variation of each of the left image and the right image. That is, the diffusion direction determination unit 901 determines the propagation direction of the energy to be allocated to the area where the visibility error occurs.

The areas where the visibility error occurs when performing stereo matching are occlusion and error regions. Since the occlusion region has no matching point between the left image and the right image, the penalty for the smoothness term, which is the data term, must be higher in the energy model.

의 패널티를 가시성 맵에서 에러가 발생하는 무효한 영역(invalid region)에서 조정하는 식을 나타낸다. d는 특정 픽셀 p에서 최대 변이를 의미하고, di는 변수에 해당하는 변이를 의미한다. V(p)는 가시성 맵에서 특정 픽셀 p의 유효성 여부를 나타낸다. 만약, V(p)가 1이면 가시성 맵에서 특정 픽실에 무효한 영역에 위치하는 것을 의미하고, V(p)가 0이면 가시성 맵에서 특정 픽셀이 유효한 영역에 위치하는 것을 의미한다. 특정 픽셀이 무효한 영역에 존재하는 경우 수학식 9에 따라 Smoothness term인

의 팩터(factor)를 더 높여서, 더 큰 패널티를 줄 수 있다.Equation 9 is the Smoothness term

Is adjusted in an invalid region where an error occurs in the visibility map. d means the maximum variation at a specific pixel p, and di means the variation corresponding to the variable. V (p) indicates whether or not the specific pixel p is valid in the visibility map. If V (p) is 1, it means that the visibility map is located in the invalid area in the specific pixilated area. If V (p) is 0, it means that the specific pixel is located in the valid area in the visibility map. When a specific pixel exists in an invalid area, the Smoothness term &quot;

A higher penalty can be given.

에 패널티를 주는 것만으로 매칭 포인트를 찾을 수 없다. 이 때, 특정 픽셀에 인접한 이웃 픽셀의 컬러가 특정 픽셀의 컬러와 비슷하다면 인접한 이웃 픽셀에 특정 픽셀의 변이와 비슷한 변이를 할당하고(Type I), 특정 픽셀에 인접한 이웃 픽셀의 컬러가 특정 픽셀의 컬러와 다르다면 인접한 이웃 픽셀에 특정 픽셀의 변이와 다른 변이를 할당할 수 있다(Type II).However, depending on the color difference between the left image and the right image,

You can not find a matching point just by giving it a penalty. In this case, if the color of the neighboring pixel adjacent to the specific pixel is similar to the color of the specific pixel, a variation similar to the variation of the specific pixel is assigned to the neighboring neighboring pixel (Type I), and the color of the neighboring pixel adjacent to the specific pixel is If it is different from the color, it is possible to assign a different pixel to the adjacent pixel (Type II).

은 가시성 에러가 발생한 무효한 영역에 대해 가로 방향의 시작점에 대응하는 왼쪽 픽셀의 변이이다.

은 가시성 에러가 발생한 무효한 영역에 대해 가로 방향의 끝 지점에 대응하는 오른쪽 픽셀의 변이이다. 수학식 10을 만족한다면, 무효한 영역은 Type I 이거나 Type II가 된다. 이 경우, 수학식 10을 만족하는 영역은 순서 제한(ordering constraint)이 만족되는 영역으로 배경 영역에서 에너지 확산을 수행할 수 있다. 만약, 수학식 10을 만족하지 않는 영역은 Type III로 오른쪽 방향과 왼쪽 방향으로 모두 에너지 확산을 수행할 수 있다.In Equation (10), VER_width denotes the width of a visibility error region where a visibility error occurs.

Is the variation of the left pixel corresponding to the starting point in the horizontal direction for the invalid area where the visibility error occurred.

Is a variation of the right pixel corresponding to the end point in the horizontal direction with respect to the invalid area where the visibility error occurs. If the expression (10) is satisfied, the invalid area is Type I or Type II. In this case, the region satisfying Equation (10) can perform the energy spreading in the background region to the region where the ordering constraint is satisfied. If the region that does not satisfy Equation (10) is Type III, both the right direction and the left direction can perform energy spreading.

That is, according to Equation (10), the diffusion direction determination unit 901 determines whether or not the difference between the pixel located at the left boundary of the invalid region where the visibility error exists and the pixel located at the right boundary exists in the region where the visibility error exists The direction of diffusion of energy can be determined depending on whether or not it is equal to the width.

The energy spreader 902 can determine the representative energy in the region where the visibility error exists, and perform the energy spread based on the representative energy. That is, the energy spreader 902 determines the energy to be propagated and determines a method of propagating the energy.

In one example, the energy spreading unit 902 may include at least one of (i) the left pixel of the pixel located at the left boundary of the region where the visibility error exists, or (ii) the right pixel of the pixel located at the right boundary of the region where the visibility error exists One can be set as the representative energy, and the energy diffusion can be performed in either the left direction, the right direction, or both directions.

Specifically, in the case of Type I, the energy spreader 902 can set the energy of a pixel located to the left of one pixel in the left start pixel of the region where the visibility error occurs as the representative energy. Then, the energy spreader 902 can recalculate the energy according to Equations (8) and (9).

In the case of Type II, the energy spreader 902 can set the energy of a pixel located to the right of one pixel to the right end pixel of the region in which the visibility error occurs as the representative energy. Then, the energy spreader 902 can recalculate the energy according to Equations (8) and (9).

In the case of Type III, the energy spreading unit 902 calculates (i) the energy of the pixel located to the left of one pixel in the left starting pixel of the region in which the visibility error occurs, and (ii) As the representative energy. Then, the energy spreader 902 can perform energy spreading in both the left and right directions, and then calculate the energy again as a sum of the two energies.

10 is a diagram showing a variation map derived using energy in a forward direction according to an embodiment of the present invention. 11 is a diagram illustrating a variation map derived using energy in the backward direction according to an embodiment of the present invention.

In Figs. 10 and 11, the circle indicates an area where a visibility error occurs. In FIGS. 10 and 11, a visibility error occurs in the path that moves from the foreground to the background. At this time, since the forward direction and the backward direction are different from each other as shown in Equation (4), the error occurrence region also differs, as shown in FIGS. 10 and 11.

FIG. 12 is a view illustrating a transition map using a forward direction and a backward direction simultaneously according to an embodiment of the present invention and a transition map using an optimization method of dynamic programming.

Referring to FIG. 12, it can be seen that the mutation map considering the forward direction and the backward direction at the same time is similar to the mutation map using the optimization method of dynamic programming. Also, it can be seen that the artifacts seen in Fig. 3 do not occur in Fig.

14 is a view for explaining local matching using energy of dynamic programming according to an embodiment of the present invention.

The local matching in FIG. 14 is derived by using the energy of the left image and the right image in parallel, and then the initial variation generated in the visibility map is used mutually between the left image and the right image. Specifically, the left visibility disparity map is generated based on the right initial variation, and the right visibility disparity map is generated based on the left initial variation. Thereafter, when the energy is recomputed in the invalid region where the visibility error occurs, the left image and the right image are processed in parallel again.

The local matching shown in FIG. 14 can be performed according to the following equation (11).

는 수학식 7에 따라 도출될 수 있다.D (p) represents a variation when local matching occurs at a specific pixel p. And q denotes a neighboring pixel belonging to the 3 * 3 region corresponding to the variation d about a specific pixel p. And,

Can be derived according to Equation (7).

15 is a flowchart illustrating a variation estimation method according to an embodiment of the present invention.

In step 1501, the variation estimation apparatus can calculate the energy of each of the left image and the right image constituting the stereo image. For example, the variation estimator can calculate the energy in the forward and backward directions for the left image and the right image, respectively. Then, the variation estimator can sum the energy in the forward direction and the backward direction.

In step 1502, the variation estimation apparatus may generate energy using a visibility map. For example, the variation estimator can calculate the initial variation of each of the left and right images using the energy of the left image and the energy of the right image. In addition, the variation estimation apparatus can determine the visibility map of each of the left image and the right image using the initial variation of the left image and the right image, respectively.

At this time, the variation estimation apparatus can determine the visibility map of the left image using the initial variation of the right image. Then, the variation estimation apparatus can determine the visibility map of the right image using the initial variation of the left image.

In step 1503, the variation estimation apparatus can extract an area where a visibility error occurs in the visibility map.

If a visibility error has occurred, at step 1504, the variation estimation device may recalculate the energy for the region in which the visibility error is present. For example, the variation estimator may use the initial variation of each of the left and right images to determine the direction of energy diffusion in the region where the visibility error is present. Then, the variation estimator can determine the representative energy in the region where the visibility error exists, and perform the energy spread based on the representative energy.

At this time, the variation estimating apparatus estimates the diffusion direction of the energy according to whether the difference between the pixel located at the left boundary of the region where the visibility error exists and the variation of the pixel located at the right boundary is equal to the width of the region where the visibility error exists Can be determined.

Then, the variation estimation apparatus calculates at least one of the following: (i) the left pixel of the pixel located at the left boundary of the region where the visibility error exists, or (ii) the right pixel of the pixel located at the right boundary of the region where the visibility error exists, The energy spreading can be performed in either the left direction, the right direction, or both directions.

Through this process, in step 1505, the variation estimation apparatus performs optimization with the finally determined energy, and in step 1506, the variation estimation apparatus can determine the variation with respect to the stereo image.

16 is a flowchart illustrating a variation estimation method using local matching according to an embodiment of the present invention.

FIG. 16 shows that the left image and the right image are performed in parallel in the process of FIG. Accordingly, steps 1601 to 1606 and steps 1607 to 1612 operate substantially the same as steps 1501 to 1506 in Fig.

However, when performing the steps 1602 and 1608 of generating the visibility map, it is necessary to interact with the left image and the right image. That is, when the mutation estimating device generates the visibility map for the left image, it can refer to the right image and refer to the left image when generating the visibility map for the right image.

Also, when performing the local matching step 1605 and the step 1611, it is necessary to perform an interaction between the left image and the right image. That is, the variation estimation apparatus refers to the right image when locally matching the left image, and refers to the left image when locating the right image locally.

The methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

101: Energy calculation unit 102: Map generation unit
103: energy recalculation unit 104:

Claims (17)

An energy calculation unit for calculating energy related to stereo matching of the left image and the right image constituting the stereo image;
A map generator for generating a visibility map for determining a disparity error between the left image and the right image using the energy;
Recalculating energy for an area in which there is a visibility error due to an error of the variation in the visibility map, and maintaining the energy in the visibility map without recalculating energy for the area in which there is no visibility error; And
A variation determining unit for determining a variation from the stereo image using the final energy of the left image and the right image,
. The method according to claim 1,
Wherein the energy calculator comprises:
A first energy calculation unit for calculating an energy in a forward direction for each of the left image and the right image;
A second energy calculation unit for calculating energy in the backward direction for each of the left image and the right image; And
An energy merging unit for merging the energy in the forward direction and the energy in the backward direction,
. The method according to claim 1,
The map generation unit generates,
An initial disparity calculating unit for calculating an initial disparity of each of the left image and the right image using the energy of the left image and the right image; And
A map determination unit for determining a visibility map of each of the left image and the right image using the initial variation of the left image and the right image,
. The method of claim 3,
The map determining unit may determine,
A first map determining unit determining a visibility map of the left image using the initial variation of the right image; And
A second map determining unit for determining a visibility map of the right image using the initial variation of the left image,
. The method according to claim 1,
Wherein the energy recycling unit comprises:
A diffusion direction determination unit that determines a diffusion direction of energy in a region where a visibility error exists using an initial variation of each of the left image and the right image; And
An energy diffusion unit for determining representative energy in a region where the visibility error exists and performing energy diffusion based on representative energy,
. 6. The method of claim 5,
Wherein the diffusion direction determination unit determines,
The diffusion direction of the energy is determined according to whether or not the difference between the pixel located at the left boundary of the region where the visibility error exists and the variation of the pixel located at the right boundary is the same as the width of the region where the visibility error exists. Lt; / RTI &gt; 6. The method of claim 5,
Wherein the energy spreading unit comprises:
(i) the left pixel of the pixel located at the left boundary of the region where the visibility error exists, or (ii) the right pixel of the pixel located at the right boundary of the region where the visibility error exists, And performs energy diffusion in either the rightward direction or the bi-directional direction. An energy calculation unit for calculating in parallel the energy related to the stereo matching of the left image and the right image constituting the stereo image;
A map generator for generating a visibility map for determining a disparity error between the left image and the right image using the energy;
Calculating an energy for an area in which there is a visibility error due to an error of the variation in the visibility map of each of the left image and the right image, and for the area where the visibility error does not exist in the visibility map, An energy recalculation unit for maintaining energy; And
A local determination unit that performs local matching using the final energy of the left image and the right image and determines a variation from a locally matched stereo image,
. Calculating energy associated with stereo matching of a left image and a right image, each of which constitutes a stereo image;
Generating a visibility map for determining a disparity error between the left image and the right image using the energy;
Recalculating energy for an area in which there is a visibility error due to an error of the variation in the visibility map and maintaining the energy in the visibility map without recalculating the energy for the area in which there is no visibility error; And
Determining a variation from the stereo image using the final energy of the left image and the right image
/ RTI &gt; 10. The method of claim 9,
Wherein the step of calculating the energy of each of the left image and the right image comprises:
Calculating energy in a forward direction for each of the left image and the right image;
Calculating energy in the backward direction for each of the left image and the right image; And
Merging the energy in the forward direction and the energy in the backward direction
/ RTI &gt; 10. The method of claim 9,
Wherein generating the visibility map comprises:
Calculating an initial variation of each of the left image and the right image using energy of the left image and the right image; And
Determining a visibility map of each of the left image and the right image using an initial variation of each of the left image and the right image,
/ RTI &gt; 12. The method of claim 11,
Wherein determining the visibility map of each of the left image and the right image comprises:
Determining a visibility map of the left image using the initial variation of the right image; And
Determining a visibility map of the right image using the initial variation of the left image
/ RTI &gt; 10. The method of claim 9,
Wherein the step of recalculating energy for an area in which the visibility error is present comprises:
Determining an energy diffusion direction in a region where a visibility error exists using an initial variation of each of the left image and the right image; And
Determining representative energy in a region where the visibility error is present, and performing energy diffusion based on representative energy
/ RTI &gt; 14. The method of claim 13,
Wherein the step of determining the diffusion direction of the energy comprises:
The diffusion direction of the energy is determined according to whether or not the difference between the pixel located at the left boundary of the region where the visibility error exists and the variation of the pixel located at the right boundary is the same as the width of the region where the visibility error exists. . 14. The method of claim 13,
Wherein the performing the energy spreading comprises:
(i) the left pixel of the pixel located at the left boundary of the region where the visibility error exists, or (ii) the right pixel of the pixel located at the right boundary of the region where the visibility error exists, Wherein the energy spreading is performed in either the rightward direction or the bi-directional direction. Calculating in parallel the energy associated with the stereo matching of the left and right images of the stereo image;
Generating a visibility map for determining a disparity error between the left image and the right image using the energy;
Calculating an energy for an area in which there is a visibility error due to an error of the variation in the visibility map of each of the left image and the right image, and for the area where the visibility error does not exist in the visibility map, Maintaining;
Performing local matching using the final energy of the left and right images, and determining a variation from a locally matched stereo image
/ RTI &gt; A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 9 to 16.

Images