KR20130067474A - Hole filling method and apparatus - Google Patents

Abstract

PURPOSE: A hole filling method and an apparatus thereof are provided to fill a hole of an image generated in an image warping process through a depth image based rendering with high quality, by using a sparsity-based in-painting process adaptively through a depth map. CONSTITUTION: A hole filling apparatus generates a depth map, by predicting the depth of a three dimensional image(120). The apparatus calculates the sparsity of an edge pixel of a hole area by using the degree of depth of the hole area on the basis of the depth map, and determines the priority of a hole filling sequence on the basis of the sparsity(150). The apparatus fills the hole area of the three dimensional image according to the priority sequence(160). [Reference numerals] (110) 3D image warping; (120) Generate a depth map through depth prediction of a hole area from the generated image; (140) Hole region completely filled?; (150) Calculate the sparsity of an edge pixel of a hole area by using the degree of depth of the hole area on the basis of the depth map and determine the priority of a hole filling sequence on the basis of the sparsity; (160) Apply inpainting in the selected patch considering texture of which the depth is less than Xhole_depth?; (170) All holes of which the depth is Xhole_depth filled?; (180) Hole filling completed; (AA) Start; (BB,DD) Yes; (CC,EE) No

Description

Hole Filling Method and Apparatus {HOLE FILLING METHOD AND APPARATUS}

The present invention relates to a hole filling method and apparatus, and more particularly, to a method and apparatus for filling high-quality holes of an image generated in the process of generating a stereoscopic 3D image through a color image and a depth map.

One of the methods for generating a stereoscopic 3D image is a computer vision based method of generating a 3D image through depth image based rendering after obtaining a pair of color images and a depth map. This method is not limited by the vertical parallax of the image, and also generates the virtual viewpoint by analyzing the current screen, so there is no problem of color distortion due to camera exposure and light change. As such, when a 3D image is generated through a depth map based depth image based rendering, an accurate and high quality 3D stereoscopic image may be obtained, but a part of the generated image may have empty information due to parallax. Occurs. Such empty portions are called holes, which are a major cause of deterioration of image quality of 3D images generated through the depth image based rendering technique.

In order to solve the above problem, various methods for filling the empty hole area have been proposed.

Layered depth-images (LDIs) solves the problem of holes, including background color and depth information in addition to the original color image and depth map. There is a limit of data increase due to difficulty and additional information (Non-Patent Document 1).

Research into filling holes through smoothing preprocessing of depth maps has also been conducted until recently, and these techniques reduce hole discontinuities in depth maps through preprocessing, thereby reducing hole filling in the generated image. Make it more convenient. However, there is a fatal problem that geometric distortion occurs in the generated image due to the smoothing of the depth map (Non-Patent Documents 2 and 3).

Fills holes that occur in depth image-based rendering without modifying the depth map.Constant color filling using horizontal pixels, horizontal interpolation, or horizontal extrapolation using depth information. ), A laplacian filling by smoothing pixel values around the hole, and a hole filling method by repeating downsampling and upsampling of an image (Non-Patent Documents 4, 5, and 6). However, this hole fill technique is difficult to expect high-quality hole fill results due to texture distortion in the hole area caused by the limitation of the utilization of pixel information in the periphery and the screen, and also causes a problem of mixing the foreground and background textures around the hole. .

 J. Shade, S. Gortler, L. He, and R. Szeliski, "Layered Depth Images", in Proc. of ACM SIGGRAPH '98, pp. 231-242, (Orlando, FL, USA), July 1998.  S. Lee, and Y. Ho, "Discontinuity-adaptive Depth Map Filtering for 3D View Generation", Proceedings of the 2nd International Conference on Immersive Telecommunications, 2009.  Jae-Ho Lee, Chang-Ik Kim, “Optimum Automatic Stereoscopic Image Generation Method through Preprocessing of Depth Map and Geometric Analysis of Depth Information,” Journal of Broadcast Engineering, 2009, Vol. 14, No 2, pp. 164-177, 2009.  K. Jung, Y. Park, and J. Kim, "2D / 3D Mixed Service in T-DMB System Using Depth Image Based Rendering", Proceedings of the Conference on 2008 ICACT.  C. Vazquez, W. J. Tam and F. Speranza, "Stereoscopic Imaging: Filling Disoccluded Areas in Depth Image-Based Rendering", Proceedings of the SPIE, Vol. 6392, pp. 63920D, 2006.  M. Solh and G. AlRegib, "Hierarchical Hole-Filling (HHF): Depth Image Based Rendering without Depth Map Filtering for 3D-TV", Proceedings of the Conference on 2010 Multimedia Signal Processing (MMSP), 2010.

An object of the present invention for solving the above problems is to fill a high-quality hole in the image generated during image warping through depth image-based rendering in the process of generating a stereoscopic three-dimensional image through a pair of color image and depth map To this end, it is to provide a method and apparatus for filling a hole by adaptively using sparsity-based inpainting through a depth map.

Another object of the present invention is to prevent the problem that the foreground and the background is mixed by using only the background texture information around the hole, and the high-quality hole in which the edge information remains in the filled hole by first processing the area with the edge of the hole area It is to provide a hole filling method and apparatus that can be filled.

A hole filling method for achieving the above object includes a method of filling a hole area of a stereoscopic 3D image, the method comprising: generating a depth map by predicting a depth degree of the 3D image; A priority determining step of calculating a sparsity of edge pixels of the hole area based on the depth map and determining a priority to execute hole filling based on the scarcity; And a hole filling step of performing hole filling on the hole area of the 3D image according to the priority.

In the depth map generation step, when the 3D image is an image generated through 3D image warping, the depth map of the 3D image may be obtained by applying a horizontal shift mapping of original color pixels and a horizontal shift mapping of original depth map information. It may comprise the step of generating.

The step of generating the depth map may include comparing depths of the left side and the right side of the hole area, and then determining a low depth degree of the hole area.

The priority determining step may include: a depth priority determining step of determining a depth priority based on a depth degree of the hole area so that an area having a low depth level has a high priority; A sparsity calculation step of calculating sparsity of hole areas of the same depth degree based on the depth priority; And determining a final priority based on the depth priority and the calculated scarcity.

The sparsity calculation step may include a first step of defining a first patch centered at an arbitrary pixel p point of an edge of the hole area; A second step of defining a second patch in which pixel information exists about an arbitrary pixel p _j in a window area centered on the p point; Calculating a similarity degree between the first patch and the second patch with respect to all the p _j points in the region where the pixel information exists in the first patch and the window region; A fourth step of calculating the sparsity of the p point based on the similarity; And a fifth step of calculating sparsity for all pixels of the edge of the hole region by repeatedly applying the sparsity calculation process of the p point to all the pixels of the edge of the hole region.

The final prioritizing step may include calculating patch priorities of all pixels on the edge of the hole based on the scarcity of all the pixels on the edge of the hole area; And determining the final priority based on the patch priority.

(여기서, ψ_p는 p점을 중심으로 하는 패치, ψ_pj는 상기 윈도우 영역 내의 p_j점을 중심으로 하는 패치,

는 ψ에서의 화소 정보가 존재하는 영역 중 깊이도가 특정 깊이도보다 낮은 영역만을 추출하는 매트릭스, d(A,B)는 A와 B 사이의 평균 제곱 오차, Z(p)는 정규화 상수를 나타내고, σ는 상수로서 0.5의 값을 갖음)에 의해 상기 유사도를 계산하는 단계를 포함할 수 있다.In the third step,

(Wherein ψ _p is a patch centering on p point, ψ _pj is a patch centering on p _j point in the window region,

Is a matrix for extracting only the regions where the pixel information in ψ is lower than a specific depth diagram, d (A, B) is the mean square error between A and B, and Z (p) is a normalization constant. , sigma has a value of 0.5 as a constant).

(여기서,

는 A의 화소 수를 의미함)에 의해 상기 P점의 희소성을 계산하는 단계를 포함할 수 있다.The fourth step is

(here,

May mean the number of pixels of A).

The filling of the hole may include filling holes from a lower depth hole based on the priority, but filling all the hole areas having the same depth, and then filling the hole area of the next depth.

The hole filling step selects a patch having the highest patch priority and obtains an approximate patch through a linear combination of patch candidates, wherein patch candidates represent N patches most similar to the selected patches, based on the selected patch. step; And filling the unknown region of the selected patch based on the approximate patch using the approximate patch.

(여기서,

는 근사 패치, α는 각 패치 후보의 가중치 값을 의미함)를 통해 구할 수 있다.The approximate patch

(here,

Is an approximate patch, and α denotes a weight value of each patch candidate.

According to an aspect of the present invention, a hole filling apparatus includes: a depth map generator configured to fill a hole area of a stereoscopic 3D image to generate a depth map by predicting a depth degree of the 3D image; A priority determining unit configured to calculate a sparsity of edge pixels of the hole area based on the depth map and determine a priority to execute hole filling based on the scarcity; And a hole filling unit for performing hole filling on the hole area of the 3D image according to the priority.

When the 3D image is an image generated through 3D image warping, the depth map generator generates a depth map of the 3D image by applying a horizontal shift mapping of original color pixels and a horizontal shift mapping of original depth map information. can do.

The depth map generator may compare the depths of the left side and the right side of the hole area, and then determine a low depth degree as the depth degree of the hole area.

The priority determiner may include a depth priority determiner configured to determine a depth priority based on a depth degree of the hole area so that an area having a low depth degree has a high priority; A sparsity calculator for calculating sparsity of the hole areas of the same depth degree based on the depth priority; And a final priority determiner configured to determine a final priority based on the depth priority and the calculated scarcity.

The sparseness calculator may include: a first patch definition unit defining a first patch centered on an arbitrary pixel p point of an edge of the hole area; A second patch definition unit defining a second patch having pixel information centered on an arbitrary pixel p _j point in a window area centered on the point P; A similarity calculator configured to calculate a similarity degree between the first patch and the second patch with respect to all of the p _j points in the region where the pixel information exists among the first patch and the window area; A p point sparseness calculator for calculating the sparsity of the p point based on the similarity; And a total pixel sparsity calculation unit for calculating the sparsity of all the pixels at the edge of the hole region by repeatedly applying the spacing of the p point to all the pixels at the edge of the hole region.

The final priority determiner may include a patch priority calculator configured to calculate patch priorities of all pixels on the edge of the hole based on the scarcity of all the pixels on the edge of the hole area; And a ranking determiner configured to determine the final priority based on the patch priority.

(여기서, ψ_p는 p점을 중심으로 하는 패치, ψ_pj는 제 1 영역 내의 p_j점을 중심으로 하는 패치,

는 ψ에서의 화소 정보가 존재하는 영역 중 깊이도가 특정 깊이도보다 낮은 영역만을 추출하는 매트릭스, d(A,B)는 A와 B 사이의 평균 제곱 오차, Z(p)는 정규화 상수를 나타내고, σ는 상수로서 0.5의 값을 갖음)에 의해 상기 유사도를 계산할 수 있다.The similarity calculation unit

(Wherein ψ _p is a patch centering on p point, ψ _pj is a patch centering on p _j point in the first region,

Is a matrix for extracting only the regions where the pixel information in ψ is lower than a specific depth diagram, d (A, B) is the mean square error between A and B, and Z (p) is a normalization constant. , sigma has a value of 0.5 as a constant).

(여기서,

는 A의 화소 수를 의미함)에 의해 상기 p점의 희소성을 계산할 수 있다.The p point sparsity calculation unit

(here,

Denotes the number of pixels of A).

The hole filling unit may fill a hole from a hole area having a low depth level based on the priority, but may fill the hole area having a next depth level after all hole areas having the same depth degree are filled.

Do patch candidate are meant the N patch most similar to the selected patch ψ _p - - the hole filling unit wherein the patch priority is the selected high point to patch the selected patch based on the candidate of the approximate patch through a linear combination of An approximate patch calculator; And a fill portion filling the unknown region of the selected patch based on the approximate patch using the approximate patch.

(여기서,

는 근사패치, α는 각 패치 후보의 가중치 값을 의미함)를 통해 구할 수 있다.
The approximate patch

(here,

Is an approximate patch, and α denotes a weight value of each patch candidate).

According to the hole filling method and apparatus of the present invention, a method for estimating the hole area depth of an image generated for the hole filling included in the hole filling technique using the sparsity-based inpainting, and using the predicted depth of the hole area By calculating the sparsity of the pixels around the hole and using the background based on the priority based on the calculated sparsity, it is possible to fill the holes of the image generated during 3D image warping with high quality in depth image-based rendering. .

In addition, according to the hole filling method and apparatus of the present invention, by overcoming the limitation of the existing inpainting method in which the foreground and the background are mixed by using only the background texture information around the hole, the area with the outline of the hole area is preferentially The edge information has a natural effect even in the filled holes by processing.

1 is an overall flowchart of a hole filling method according to an embodiment of the present invention;
2 is a flow diagram schematically showing a hole filling method according to an embodiment of the present invention,
3 is a conceptual diagram illustrating a method of predicting a depth map of an image through a depth map generation step of the hole filling method of the present invention;
4 is a detailed flowchart illustrating the prioritization step of the hole filling method according to an embodiment of the present invention;
5 is a detailed flow chart showing in detail the scarcity calculation step of the hole filling method according to an embodiment of the present invention,
FIG. 6 is a conceptual diagram illustrating a sparsity based inpainting algorithm in a sparsity calculation step according to an embodiment of the present invention; FIG.
7 is a detailed flowchart illustrating the final prioritization step of the hole filling method according to an embodiment of the present invention;
8 is a detailed flowchart showing the hole filling step of the hole filling method according to an embodiment of the present invention;
9 is a conceptual diagram illustrating a patch inpainting algorithm in the hole filling step according to an embodiment of the present invention;
10 is a block diagram schematically showing a hole filling device according to an embodiment of the present invention;
11 is a detailed block diagram specifically showing a priority determining unit of a hole filling apparatus according to an embodiment of the present invention;
12 is a detailed block diagram showing in detail the scarcity calculation unit of the hole filling apparatus according to the embodiment of the present invention;
FIG. 13 is a detailed block diagram specifically illustrating a final priority determining unit of a hole filling apparatus according to an embodiment of the present invention; FIG.
14 is a detailed block diagram illustrating in detail a hole filling unit of the hole filling apparatus according to an exemplary embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

How to fill the hole

In the hole filling method according to an embodiment of the present invention, when filling a hole, the above-described problems, `` texture distortion of the hole area caused by the limitation of utilization of pixel information in the periphery of the hole and the screen, '' and `` the foreground and the background of the hole filling. '' Texture mixing problem. To this end, we propose a method of applying a sparsity-based inpainting technique to the hole filling, considering only the background texture, to make the hole filling more efficient by utilizing the pixel information around the hole and the inside of the screen, and the foreground texture and the background texture. By separating and using only background texture information, it is possible to provide high-quality hole fills.

The hole filling method according to the present invention applies a sparsity-based inpainting technique to the hole filling considering only the background texture. Sparity-based image inpainting allows for high-quality inpainting, whereby the contours or structures of objects in the image can be preserved by utilizing the similarity of surrounding pixels or textures in the screen. In addition, for inpainting considering only the background texture, the depth of the hole region is predicted in the generated image, and then the hole region is inpainted with reference to only the background texture having the same or lower depth.

1 is a general flow diagram of a hole filling method according to an embodiment of the present invention. As shown in FIG. 1, the hole filling method according to an exemplary embodiment of the present invention includes a 3D image warping step 110, a depth map generation step 120 by predicting a depth of a hole area in a generated image, and a first step. Hole depth setting step 130, determining whether the hole area is filled 140, inpainting priority patch selection step 150 by calculating sparsity of patches having a depth of hole area first depth, Applying inpainting to the selected patch in consideration of only a texture whose depth of the hole area is less than or equal to the first hole depth (160), determining whether all of the holes having the first depth of depth are filled (170) and the holes The filling completion step 180 may be included.

Referring to Figure 1, the hole filling method according to an embodiment of the present invention can be divided into two steps [1] and [2]. First, in step [1], since the depth map of the generated image is required to fill the hole in consideration of only the background texture in the image generated by the 3D image warping 110, the depth map of the generated image is predicted and generated. (120). In operation [2], first, a first depth degree X _{hole_depth} (initial value 0) is set (130). Then, it is determined whether all of the hole areas are filled (140). At this time, if all hole areas are filled, the hole filling may be completed (180). If the hole area is not filled, low depth is also a priority after determining the patch 150, the depth through the calculation scarcity considering only patch (patch) of the hole region having a (the first depth degrees) the X _{hole _} Hole filling is performed by applying inpainting considering only a background texture having a _depth or less (160). When all holes of the same depth are filled, the X _{hole _ depth} is increased to fill the hole area of the next depth with sparsity-based inpainting. If all of the hole regions are filled by repeating the above process, the repeating process is completed (180).

2 is a flowchart schematically illustrating a hole filling method according to an exemplary embodiment of the present invention. As shown in FIG. 2, in the hole filling method according to an embodiment of the present invention, a depth map is generated by predicting a depth degree of a dimensional image (210) by using a depth diagram of a hole area based on a depth map. Calculating the sparsity of the edge pixels of the hole area and determining a priority to execute hole filling based on the scarcity (220); and performing hole filling on the hole area of the dimensional image according to the priority (230). It may include.

Referring to FIG. 2, first, in a depth map generation step 210, a hole filling apparatus (not shown) according to an exemplary embodiment of the present invention predicts a depth map of an image generated through warping of a 3D image, and determines a depth map. Create The depth map of the generated image is required to fill the hole by considering only the background texture in the image generated through the 3D image warping. To this end, when the horizontal shift mapping of the original color pixels is applied to the horizontal depth mapping of the original color pixels, the depth map of the generated image may be obtained. However, the depth map obtained through this process will have a hole area like the generated color image. Therefore, it is necessary to predict and generate the depth diagram of the hole area. The depth map generation step 210 of the present invention focuses on the fact that the hole region is a background texture region having a low depth, and uses a method of filling the depth of the hole region with a low depth of the surrounding. That is, after comparing the depths of the left side and the right side of the hole, the low depth degree is determined as the depth degree of the hole area. It is possible to fill the hole area of the generated depth map by considering the depth of the surrounding background texture.

3 is a conceptual view illustrating a method of predicting a depth map of an image through the depth map generation step 210 of the hole filling method of the present invention. The image 310 is a color image generated through 3D warping, the image 320 is a depth image in which the hole 315 exists, and the image 330 applies depth prediction to the hole regions 335 and 336. The depth image is generated. As illustrated in FIG. 3, a hole area 315 may be generated between the object 311 and the backgrounds 312 and 313 in the color image 310. In the depth image 320, the hole area 315 is present between the object 321 and the backgrounds 322 and 323. In this case, the backgrounds 312, 313, 322, and 323 may be divided into the backgrounds 312 and 322 having a low depth and the backgrounds 313 and 323 having a high depth. In the depth map generation step 210 of the present invention, the depth diagram of the hole region may be predicted based on the background divided as described above. The image 330 predicts the depth diagram through a background texture having a low depth of periphery and uses a background 332 texture having a low depth diagram to show a low depth of the hole area as shown in FIG. 3. A depth diagram of the portion 335 having a degree is predicted, and a background 333 texture having a high depth degree is used to predict a depth diagram of the portion 336 having a high depth degree of the hole area. As a result, the depth diagrams of the hole regions 335 and 336 can be more accurately predicted, and thus a more accurate depth map can be generated.

2, in the prioritization step 220, the hole filling apparatus according to the exemplary embodiment of the present invention may determine the depth of the hole area based on the depth map generated through the depth map generation step 210. Using the diagram, the sparsity of the edge pixels of the hole area is calculated, and the priority to execute the hole filling is determined based on the scarcity.

4 is a detailed flowchart illustrating the prioritization step 220 of the hole filling method according to an embodiment of the present invention. As shown in FIG. 4, in the prioritizing step 220 of the present invention, the depth priority step 410 is performed to determine a depth priority so that an area having a lower depth has a higher priority based on the depth of the hole area. Computing the sparsity of the hole area of the same depth map based on the priority (420) and determining the final priority based on the depth priority and the calculated scarcity (430).

Referring to FIG. 4, in the depth prioritization step 410, the hole filling apparatus according to an embodiment of the present invention sets a first depth degree ( X _{hole _ depth} ) for hole filling in consideration of the depth degree. The first depth means a depth diagram of a hole region and a background texture to be referred to when applying a sparsity calculation and inpainting variable. In the depth prioritization step 410, the hole filling apparatus needs to fill the hole in the hole area with the rearmost background (that is, the part with the lowest depth), so that the depth at the boundary between the hole and the area in which the pixel information exists is determined. Where the lowest degree (i.e. where the depth is 0) ( X _{hole _ depth} Depth priority is determined to have high priority = 0)). Since the priority based on the scarcity afterwards is also primarily executed based on the depth priority, it fills all the low-depth hole areas based on the depth priority, and _proceeds to the next depth level (e.g., X _{hole_depth} = 1). Hole filling can be performed by filling hole areas of the same depth. Therefore, the final priority determined based on the recall is established based on the depth priority. In other words, the final priority of the patch is determined by finding only the sparsity of the portion having a depth of 0, and the inpainting algorithm can be applied to the selected patch. When all hole regions of the same depth degree are filled, the first depth degree X _{hole_depth} is increased one by one, and the algorithm is repeatedly applied.

Next, in the sparsity calculation step 420, the hole filling apparatus calculates the sparsity of the hole areas of the same depth degree based on the priority determined in the depth prioritization step 410.

5 is a detailed flowchart illustrating the sparsity calculation step 420 of the hole filling method according to an embodiment of the present invention. As shown in FIG. 5, the sparsity calculation step 420 of the present invention includes a first patch definition step 510, a second patch definition step 520, a similarity calculation step 530, and a p point sparsity calculation step 540. And full pixel sparsity calculation step 550.

는 홀 주변의 화소 정보가 존재하는 영역을 의미한다. 그리고는 홀 채우기 장치는 ψ_p를 홀 가장자리의 p 점을 중심으로 하는 제 1 패치로 정의한다(510). 패치란 특정 화소를 포함하고 있는 하나의 블록을 의미한다. 또한, N(p)는 p점을 중심으로 하는 윈도우 영역을 의미한다. 윈도우 영역이란 패치보다 더 큰 영역을 나타내는 것으로 매크로 블록을 의미한다. 따라서, 제 1 패치는 상기 윈도우 영역에 포함되는 관계를 갖는다. 다음으로, 홀 채우기 장치는 ψ_pj를 상기 윈도우 영역 N(p) 내에 p_j점을 중심으로 하는 화소 정보가 존재하는 제 2 패치로 정의한다(520). 이때, N(p) 내의 화소 정보가 존재하는 영역 중 깊이도가 상기 제 1 깊이도(X _{hole_depth} )보다 낮은 영역을 N_s(P)라고 정의할 수 있다. 이를 토대로 제 1 패치(ψ_p)와 제 2 패치(ψ_pj) 사이의 유사성(

)을 다음의 수학식을 이용하여 구할 수 있다.FIG. 6 is a conceptual diagram illustrating a sparsity based inpainting algorithm in the sparsity calculation step 420 according to an embodiment of the present invention. First, the sparsity of point p, the edge of the hole, is calculated in the sparsity-based inpainting algorithm. Where Ω is the hole area,

Denotes an area in which pixel information around a hole exists. Then, the hole filling apparatus defines ψ _p as the first patch around the p point of the hole edge (510). A patch refers to one block including a specific pixel. In addition, N (p) means the window area centering on p point. The window area refers to an area larger than a patch, which means a macro block. Therefore, the first patch has a relationship included in the window area. Next, the hole filling apparatus defines ψ _pj as a second patch in which pixel information about the point p _j exists in the window region N (p) (520). In this case, a region having a depth lower than the first depth diagram X _{hole_depth} among the regions where pixel information exists in N (p) may be defined as N _s (P). Based on this, the similarity between the first patch ψ _p and the second patch ψ _pj (

) Can be obtained using the following equation.

는 패치(ψ)에서의 화소 정보가 존재하는 영역 중 깊이도가 제 1 깊이도(X _{hole _ depth} )보다 낮은 영역만을 추출하는 매트릭스를 의미하며, d(A,B)는 A,B 사이의 평균 제곱 오차(mean squared error)를, Z(p)는 정규화(normalization) 상수를 뜻한다. 또한, σ는 상수로써 0.5의 값을 갖는다. 이와 같이 제 1 패치(ψ_p)와 제 2 패치(ψ_pj) 사이의 유사성을 계산하는 과정을 기반으로 상기 유사성 계산을 N_s(P) 영역 내에 존재하는 모든 p_j점에 대해 구한다. 그리고 나서 다음의 수학식을 이용하여 p점의 희소성 ρ(p)를 계산한다(540).here,

Denotes a matrix that extracts only a region having a _depth lower than the first depth ( X _{hole _ depth} ) among the regions where the pixel information exists in the patch (ψ), and d (A, B) is a distance between A and B. Mean squared error, Z (p) means normalization constant. Also, sigma has a value of 0.5 as a constant. As described above, the similarity calculation is obtained for all p _j points existing in the N _s (P) region based on the process of calculating the similarity between the first patch ψ _p and the second patch ψ _pj . Then, the sparsity p (p) of the p point is calculated using the following equation (540).

는 A의 화소 수를 의미한다. 그리고는 p점의 희소성 계산 과정을 홀 영역의 가장자리의 모든 화소에 대해 반복 적용한다(550). 반복 적용을 통해 모든 홀 영역 가장자리의 화소에 대한 희소성을 확보할 수 있다. here,

Denotes the number of pixels in A. In operation 550, the sparsity calculation process of the p point is repeatedly applied to all the pixels at the edge of the hole area. Iterative application ensures sparsity for pixels at all edges of the hole region.

The scarcity is calculated by multiplying the value of the similarity normalized by the weight ratio of the window area and the area where the pixel information exists in the window area. A sparse patch means a high complexity, so it is likely to be the edge of the hole area. Therefore, a patch having a high scarcity based on the scarcity may be set to have a high patch priority, and thus the hole filling may be performed first, thereby achieving natural hole filling.

4, in the final prioritization step 430, the final priority is determined based on the depth priority and the scarcity.

7 is a detailed flowchart illustrating the final prioritization step 430 of the hole filling method according to an embodiment of the present invention. As shown in FIG. 7, the final prioritization step 430 calculates patch priority of all pixels at the edge of the hole based on the sparsity of all the pixels at the edge of the hole area 710 and patch priority. The determining of the final priority based on the step 720 may include.

Referring to FIG. 7, the patch priority may be calculated based on all the scarcity at the edge of the hole area calculated through the scarcity calculation step 420 (710). The patch priority at the point p can be calculated by the following equation.

는 ρ(p)의 최대값과 최소값의 간격을 0.2와 1 상이로의 선형 변환을 뜻하며, C(p)는 제 1 패치(ψ_p)에서 화소 정보를 알고 있는 화소 영역의 비융을 의미한다. p점의 패치 우선도를 계산하는 과정을 홀의 가장자리 화소들에 대해 모두 반복 적용하여 홀 영역의 가장자리의 모든 화소들의 패치 우선도를 산출한다. 그리고는 상기 패치 우선도를 기반으로 최종 우선 순위를 결정할 수 있다(720). 즉, 패치 우선도가 높을수록 최종 우선 순위도 높다.here,

Denotes a linear transformation of the interval between the maximum value and the minimum value of ρ (p) between 0.2 and 1, and C (p) refers to the melting of the pixel region where the pixel information is known in the first patch ψ _p . The process of calculating the patch priority of the p point is repeatedly applied to all edge pixels of the hole to calculate patch priority of all pixels of the edge of the hole area. In operation 720, the final priority may be determined based on the patch priority. In other words, the higher the patch priority, the higher the final priority.

2, in the hole filling step 230, the hole filling apparatus performs hole filling on the hole area of the 3D image based on the priority determined in the prioritizing step 220.

8 is a detailed flowchart illustrating the hole filling step 230 of the hole filling method according to an embodiment of the present invention. As shown in FIG. 8, the hole filling step 230 according to an embodiment of the present invention selects a patch having the highest patch priority and obtains an approximate patch through linear combination of patch candidates based on the selected patch. 810 and filling the unknown region of the selected patch based on the approximate patch using the approximate patch 820.

를 구한다(810).9 is a conceptual view illustrating a patch inpainting algorithm in the hole filling step according to an embodiment of the present invention. As shown in Fig. 9, the patch with the highest patch priority is selected according to the patch priority calculation. In the present embodiment, it is assumed that the first patch ψ _p is selected with the highest patch priority. Then, based on the selected patch (ψ _p ), an approximate patch is obtained through linear combination of patch candidates using the following equation.

Obtain 810.

은 제한된 최적화 문제(constrained optimization problem)의 최소화를 통해 구하게 되며, 이렇게 구해진 근사 패치(

)를 통해 선택된 패치(ψ_p)의 모르는 영역을 근사 패치(

)를 이용하여 채우게 된다(820). 홀 채우기는 우선 순위를 기반으로 낮은 깊이도의 홀 영역부터 홀 채우기를 실행하되, 같은 깊이도의 홀 영역이 모두 채우고 나서, 다음 깊이도의 홀 영역을 채우는 것이 바람직할 수 있다. Here, N denotes the number of patch candidate patch candidates, and the patch candidate means N patches most similar to the selected patch (ψ _p ), and among the areas where the pixels of the first patch (ψ _p ) exist. It is obtained by calculating a sum squared difference (SSD) with an area lower than the first depth degree X _{hole_depth} . Α denotes a weight value of each patch candidate. 9, phi _q1 , phi _q2 , phi _q3 is comprised as a patch candidate, and the weight (alpha) ₁ , (alpha) ₂ , (alpha) ₃ is respectively assigned to this patch candidate. Set of weights for each patch candidate (ψ _q1 , ψ _q2 , ψ _q3 )

Is obtained by minimizing the constrained optimization problem.

The unknown region of the patch (ψ _p ) selected through

In step 820, the data is filled using the. The hole filling may be performed based on the priority, and the hole filling may be performed from the hole area of the low depth degree, and then the hole area of the next depth degree may be filled after all the hole areas of the same depth degree are filled.

According to the hole filling method according to an embodiment of the present invention, since the inpainting is performed based on the scarcity, the hole region is filled using an approximate patch from the edge of the hole region, and the scarcity is introduced into the hole region from the edge. By repeatedly applying the based inpainting, a natural hole filling effect can be obtained.

Hole filling device

10 is a block diagram schematically illustrating a hole filling apparatus 1000 according to an exemplary embodiment of the present invention. As shown in FIG. 10, the hole filling apparatus 1000 of the present invention fills a hole area of a stereoscopic 3D image, and includes a depth map generator 1010, a priority determiner 1020, and a hole fill unit. 1030.

Referring to FIG. 10, the depth map generator 1010 generates a depth map by predicting a depth diagram of a 3D image generated by 3D warping. The depth map generator 1010 may generate the depth map of the 3D image by applying the horizontal shift mapping of the original color pixels and the horizontal shift mapping of the original depth map information.

In another embodiment, the depth map generator 1010 compares the depths of the left side and the right side of the hole area, and then determines a low depth degree as the depth of the hole area to predict the depth degree of the hole area. You can create a depth map.

Next, the priority determiner 1020 calculates the sparsity of the edge pixel of the hole area based on the depth map generated by the depth map generator 1010 and fills the hole based on the sparsity. Determine the priority to run.

11 is a detailed block diagram illustrating in detail the priority determiner 1020 of the hole filling apparatus according to the exemplary embodiment of the present invention. As shown in FIG. 11, the depth priority determiner 1110, the sparseness calculator 1120, and the final priority determiner 1130 may be included.

Referring to FIG. 11, the depth priority determiner 1110 determines the depth priority so that an area having a low depth level has a high priority based on the depth degree of the hole area.

The sparsity calculator 1120 calculates the sparsity of the hole areas of the same depth degree based on the depth priority.

FIG. 12 is a detailed block diagram of the sparsity calculator 1120 of the hole filling apparatus according to the exemplary embodiment. As shown in FIG. 12, the sparseness calculator 1120 includes a first patch definer 1210, a second patch definer 1220, a similarity calculator 1230, a p point sparseness calculator 1240, and an entirety. The pixel sparsity calculator 1250 may be included.

The first patch defining unit 1210 defines a first patch centering on an arbitrary pixel p point of an edge of the hole area. The second patch defining unit 1220 defines a second patch in which pixel information exists about a certain pixel p _j in the window area centering on the P point.

Then, the similarity calculator 1230 calculates the similarity between the first patch and the second patch. According to an embodiment of the present invention, the similarity calculator 1230 may calculate the similarity between the first patch and the second patch with respect to all p _j points in the region where the pixel information exists among the first patch and the window region. Calculate

(여기서, ψ_p는 p점을 중심으로 하는 패치, ψ_pj는 제 1 영역 내의 p_j점을 중심으로 하는 패치,

는 상기 패치(ψ)에서의 화소 정보가 존재하는 영역 중 깊이도가 특정 깊이도보다 낮은 영역만을 추출하는 매트릭스, d(A,B)는 A와 B 사이의 평균 제곱 오차, Z(p)는 정규화 상수를 나타내고, σ는 상수로서 0.5의 값을 갖음)에 의해 제 1 패치와 제 2 패치 사이의 유사도를 계산할 수 있다.The similarity calculator 1230

(Wherein ψ _p is a patch centering on p point, ψ _pj is a patch centering on p _j point in the first region,

Is a matrix that extracts only a region whose depth is lower than a certain depth among regions where pixel information exists in the patch (ψ), d (A, B) is the mean square error between A and B, and Z (p) is The normalization constant, sigma has a value of 0.5 as a constant), and the similarity between the first patch and the second patch can be calculated.

The p-point sparsity calculator 1240 calculates the scarcity of the p-point based on the similarity calculated by the similarity calculator 1230.

(여기서,

는 A의 화소 수를 의미함)에 의해 상기 p점의 희소성을 계산할 수 있다.According to one embodiment of the invention, the sparsity calculator 1230

(here,

Denotes the number of pixels of A).

Then, the entire pixel sparsity calculator 1250 calculates the sparsity of all the pixels at the edge of the hole area by repeatedly applying the p-sparity calculation process to all the pixels at the edge of the hole area.

11, the final priority determiner 1130 determines the final priority based on the scarcity calculated by the scarcity calculator 1120.

FIG. 13 is a detailed block diagram illustrating the final priority determiner 1130 of the hole filling apparatus according to the exemplary embodiment of the present invention. As shown in FIG. 13, the final priority determiner 1130 according to an embodiment of the present invention may include a patch priority calculator 1310 and a ranker 1320.

Referring to FIG. 13, the patch priority calculator 1310 calculates patch priorities of all pixels on the edge of the hole area based on the sparsity of all the pixels on the edge of the hole area. The rank determiner 1320 determines a final priority for determining which patch to first perform hole filling based on the patch priority calculated by the patch priority calculator 1310.

10, the hole filling unit 1030 performs hole filling on the hole area of the 3D image according to the priority determined by the priority determining unit 1020. The hole filling unit 1030 performs hole filling from the hole area of the low depth degree based on the priority, and after filling all the hole areas of the same depth degree, it is preferable to fill the hole area of the next depth degree.

14 is a detailed block diagram of the hole filling unit 1030 of the hole filling apparatus according to an embodiment of the present invention. As shown in FIG. 14, the hole filling unit 1030 according to an embodiment of the present invention may include an approximate patch calculating unit 1410 and a filling unit 1420.

Referring to FIG. 14, the approximate patch calculator 1410 selects a point having the highest patch priority and obtains an approximate patch through linear combination of patch candidates based on the selected patch. Here, the patch candidate means N patches most similar to the selected patch.

를 통해 근사 패치를 구할 수 있다. 여기서,

는 근사패치, α는 각 패치 후보의 가중치 값을 의미한다. According to an embodiment of the present invention, the approximate patch calculation unit 1410

An approximate patch can be obtained from. here,

Denotes an approximate patch, and α denotes a weight value of each patch candidate.

The fill unit 1420 fills an unknown region of the patch selected based on the approximate patch calculated by the approximate patch calculator 1410 using an approximate patch.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the inventions as defined by the following claims It will be understood that various modifications and changes may be made thereto without departing from the spirit and scope of the invention.

Claims (22)

In the method for filling a hole area of a stereoscopic 3D image,
A depth map generation step of generating a depth map by predicting a depth degree of the 3D image;
A priority determining step of calculating a sparsity of edge pixels of the hole area based on the depth map and determining a priority to execute hole filling based on the scarcity; And
And a hole filling step of filling the hole area of the 3D image according to the priority.
The method of claim 1, wherein generating the depth map
When the 3D image is an image generated through 3D image warping, generating a depth map of the 3D image by applying a horizontal shift mapping of an original color pixel and a horizontal shift mapping of original depth map information. Hole filling method characterized in that.
The method of claim 1, wherein generating the depth map
And comparing the depths of the left side and the right side of the hole area, and determining a low depth degree of the hole area.
The method of claim 1, wherein the prioritizing step
A depth priority determining step of determining a depth priority based on a depth degree of the hole area so that an area having a low depth level has a high priority;
A sparsity calculation step of calculating sparsity of hole areas of the same depth degree based on the depth priority; And
And determining a final priority based on the depth priority and the calculated sparsity.
5. The method of claim 4, wherein said calculating sparsity
A first step of defining a first patch centered at an arbitrary pixel point p of an edge of the hole area;
A second step of defining a second patch in which pixel information exists about an arbitrary pixel p _j in a window area centered on the p point;
Calculating a similarity degree between the first patch and the second patch with respect to all the p _j points in the region where the pixel information exists in the first patch and the window region;
A fourth step of calculating the sparsity of the p point based on the similarity; And
And a fifth step of calculating sparsity for all pixels of the edge of the hole region by repeatedly applying the sparsity calculation process of the p-point to all the pixels of the edge of the hole region.
6. The method of claim 5, wherein the final prioritization step is
Calculating patch priorities of all the pixels at the edge of the hole based on the sparsity of all the pixels at the edge of the hole area; And
And determining the final priority based on the patch priority.
The method of claim 5, wherein the third step

(Wherein ψ _p is a patch centering on p point, ψ _pj is a patch centering on p _j point in the window region,

Is a matrix for extracting only the regions where the pixel information in ψ is lower than a specific depth diagram, d (A, B) is the mean square error between A and B, and Z (p) is a normalization constant. , sigma has a value of 0.5 as a constant).
8. The method of claim 7, wherein the fourth step is

(here,

(A means the number of pixels of A); calculating the sparsity of the point P.
The method of claim 1, wherein the hole filling step
Performing hole filling from the hole areas of the low depth degree based on the priority, and filling all of the hole areas of the same depth degree, and then filling the hole areas of the next depth degree.
The method of claim 6, wherein the hole filling step
Selecting a patch having the highest patch priority to obtain an approximate patch based on a linear combination of patch candidates, wherein patch candidates represent N patches most similar to the selected patches; And
Filling the unknown region of the selected patch using the approximate patch based on the approximate patch.
11. The method of claim 10,
The approximate patch

(here,

Is an approximate patch, and α denotes a weight value of each patch candidate).
An apparatus for filling a hole area of a stereoscopic 3D image,
A depth map generator for predicting a depth degree of the 3D image to generate a depth map;
A priority determining unit configured to calculate a sparsity of edge pixels of the hole area based on the depth map and determine a priority to execute hole filling based on the scarcity; And
And a hole filling unit for filling a hole in the hole area of the 3D image according to the priority.
The depth map generator of claim 12, wherein the depth map generator
When the 3D image is an image generated through 3D image warping, a depth map of the 3D image is generated by applying a horizontal shift mapping of original color pixels and a horizontal shift mapping of original depth map information. Hole filling device.
The depth map generator of claim 12, wherein the depth map generator
And comparing the depths of the left side and the right side of the hole area, and determining a low depth degree as the depth of the hole area.
The method of claim 12, wherein the priority determiner
A depth priority determiner configured to determine a depth priority based on a depth degree of the hole area so that an area having a low depth degree has a high priority;
A sparsity calculator for calculating sparsity of the hole areas of the same depth degree based on the depth priority; And
And a final priority determiner configured to determine a final priority based on the depth priority and the calculated sparsity.
The method of claim 15, wherein the scarcity calculation unit
A first patch definition unit defining a first patch centering on an arbitrary pixel point P of an edge of the hole area;
A second patch definition unit defining a second patch having pixel information centered on an arbitrary pixel p _j point in a window area centered on the point P;
A similarity calculator configured to calculate a similarity degree between the first patch and the second patch with respect to all of the p _j points in the region where the pixel information exists among the first patch and the window area;
A p point sparseness calculator for calculating the sparsity of the p point based on the similarity; And
And a total pixel sparsity calculating unit for calculating sparsity for all pixels of the edge of the hole region by repeatedly applying the sparsity calculation process of the p point to all the pixels of the edge of the hole region.
17. The apparatus of claim 16, wherein the final priority determiner
A patch priority calculator configured to calculate patch priority of all pixels on the edge of the hole based on the sparsity of all the pixels on the edge of the hole area; And
And a ranking determiner configured to determine the final priority based on the patch priority.
The method of claim 16, wherein the similarity calculation unit

(Wherein ψ _p is a patch centering on p point, ψ _pj is a patch centering on p _j point in the first region,

Is a matrix for extracting only the regions where the pixel information in ψ is lower than a specific depth diagram, d (A, B) is the mean square error between A and B, and Z (p) is a normalization constant. , sigma has a value of 0.5 as a constant).
19. The apparatus of claim 18, wherein the p point sparsity calculation unit

(here,

Denotes the number of pixels of A), wherein the sparsity of the p point is calculated.
The method of claim 12, wherein the hole filling portion
The hole filling apparatus performs hole filling from the hole area of the low depth degree based on the said priority, but fills all the hole area of the same depth degree, and then fills the hole area of the next depth degree.
The method of claim 17, wherein the hole filling portion
Selecting the point with the highest patch priority and calculating an approximate patch based on a linear combination of patch candidates based on the selected patch, the patch candidate means N patches most similar to the selected patch ψ _p . part; And
And a fill portion filling the unknown region of the selected patch based on the approximate patch using the approximate patch.
22. The method of claim 21,
The approximate patch

(here,

Is an approximate patch, and α denotes a weight value of each patch candidate).

Images