KR20140102999A - Method and apparatus for image warping - Google Patents

Abstract

A method and apparatus for image warping are provided. The image warping apparatus comprises a shielded area management unit for generating shielded area information based on a stereo source image; and an image processing unit which interpolates variations of the shielded area information based on rare variation information of the stereo source image, and separates a shielded area and a non-shielded area when mesh warping the stereo source image to generate a virtual intermediate viewpoint image by using the result of the shielded area information reconstruction according to the variation-interpolated shielded area information.

Description

[0001] The present invention relates to a method and apparatus for image warping,

The present invention relates to an apparatus and method for image warping, and more particularly, to an apparatus and method for image warping that divides a shielded area and an unshielded area based on previously extracted shielded area information to reduce image distortion in the vicinity of a depth discontinuity point, And to an image warping apparatus and method capable of maintaining a stereoscopic effect.

Conventionally, the multi-view image generation technique calculates a variation from a stereo image by a stereo matching method and uses an intermediate image generated by using the variation information.

The multi-view image generation technique using the variation has a problem in that the variation information at the depth discontinuity point is inaccurate due to the shielding region of the depth discontinuity point, and artifacts that are uncomfortable in the hall interpolation process are generated.

Recently, in the stereo image, the variance information is calculated only for a partial region of the image through the feature point variation / optical flow calculation, and each term of the energy function is defined by using the constraint defined by the image saliency and the edge linear component And the weighted combination of them is used to minimize the energy function, and the warping method of the image mesh grid is used to generate multi - view images.

As described above, the multi-view image generation technique based on the image warping has a problem that the hole and the artificial structure are not generated due to the inaccuracy of the mutation information, but the stereoscopic effect of the stereo image is decreased due to the reduction of the depth difference due to the image distortion at the depth discontinuity point or near the shielding area .

Therefore, there is a need for techniques to minimize image distortion in the point of depth discomfort or in the vicinity of the shielding area in image warping.

Korean Patent Laid-Open No. 10-2012-0016439 ("Image Processing Apparatus and Method ", Samsung Electronics, published Feb. 24, 2012)

It is an object of the present invention to solve the above-described problems by providing an image processing apparatus and method for separating a shielded area and an unshielded area based on previously extracted shielded area information to reduce image distortion in the vicinity of a depth discontinuity point, And to provide an apparatus and method for warping.

According to an aspect of the present invention, there is provided an image warping apparatus including: a shielded area manager for generating shielded area information based on a stereo original image; And interpolating the variation of the shielded area information based on the rare variation information of the stereo original image and reconstructing the shielded area information according to the interpolated shielded area information, And an image processing unit for generating a virtual intermediate view image by separating the area and the non-shielded area.

According to the apparatus and method for image warping described above, the image of the stereo original image is divided into the shielded area and the non-shielded area based on the extracted shielded area information, and then image warping is performed using the reconstructed image of the shielded area information The image distortion in the vicinity of the shielding region can be reduced, and the image quality can be improved by maintaining the stereoscopic effect.

In addition, according to the embodiment of the present invention, a stereo original image is divided based on color information or divided into meaningful regions based on depth information, and mesh generation and image warping are applied to each region, Depth discontinuity maintenance and image distortion can be reduced.

1 is a view schematically showing an image warping apparatus according to an embodiment of the present invention.
2 is a diagram schematically illustrating an example of generating shielded area information according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating an example of dividing a stereo original image according to an embodiment of the present invention into a shielded area and a non-shielded area.
4 is a diagram showing an example of a rare variation image according to an embodiment of the present invention.
5 is a diagram illustrating an example of segmenting a stereo original image according to an embodiment of the present invention.
6 is a diagram illustrating an example of reconstructing shielded area information according to an embodiment of the present invention.
7 is a diagram illustrating an example of generating a virtual intermediate viewpoint multi-view image according to an embodiment of the present invention.
8 is a flowchart illustrating a method of performing image warping in consideration of shielded area information according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

1 is a view schematically showing an image warping apparatus according to an embodiment of the present invention. 2 is a diagram schematically illustrating an example of generating shielded area information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of dividing a stereo original image according to an embodiment of the present invention into a shielded area and a non-shielded area. 4 is a diagram showing an example of a rare variation image according to an embodiment of the present invention. 5 is a diagram illustrating an example of segmenting a stereo original image according to an embodiment of the present invention. 6 is a diagram illustrating an example of reconstructing shielded area information according to an embodiment of the present invention. 7 is a diagram illustrating an example of generating a virtual intermediate viewpoint multi-view image according to an embodiment of the present invention. 8 is a flowchart illustrating a method of performing image warping in consideration of shielded area information according to an embodiment of the present invention.

As shown in FIG. 1, the image warping apparatus 100 according to an embodiment of the present invention extracts shielded area information in advance and separates the shielded area from the non-shielded area, and performs mesh warping, The image distortion in the vicinity of the shielding area is reduced when the viewpoint image is generated, and the stereoscopic feeling is maintained.

The image warping apparatus 100 includes an image receiving unit 110, a shielded area management unit 120, and an image processing unit 130.

The image receiving unit 110 receives the stereo original image. The image receiving unit 110 transmits the inputted stereo original image to the shielded area management unit 120 and the image processing unit 130. The stereo original image according to the embodiment of the present invention includes a stereo original left image (hereinafter referred to as "original left image") and a stereo original right image (hereinafter referred to as "original right image").

The shielded area management unit 120 includes a shielded area information generation unit 121 and an area classification unit 122.

The shielded area information generating unit 121 receives the stereo original image from the image receiving unit 110. The shielded area information generation unit 121 extracts a mutual crossing between the original left image and the original right image. The shielded area information generation unit 121 generates a virtual viewpoint image (hereinafter, referred to as a "virtual viewpoint right view image") of the original right view image position using the original left view image based on the extracted cross transition, (Hereinafter referred to as "virtual viewpoint left image") of the original left image position.

The shielded area information generation unit 121 generates shielded area information using the difference between the original left / right image and the virtual viewpoint left / right image. In other words, the shielded area information generation unit 121 generates the left shielded area information by calculating the difference between the original left image and the virtual left view image. In the same manner, the shielded area information generation unit 121 generates right shielded area information by calculating the difference between the original right image and the virtual viewpoint right image. The shielded area information generation unit 121 transmits the generated left shielded area information and right shielded area information to the area division unit 122. [ The shielded area information according to an embodiment of the present invention includes left shielded area information and right shielded area information.

2, the shielded area information generating unit 121 generates a virtual viewpoint right image (FIG. 2) by using the variation information 210a of the original left image 200a as a virtual image at the position of the original right image 200b 300b. In the same manner, the shielded area information generation unit 121 generates the virtual viewpoint left image 300a as the virtual image at the position of the original left image 200a using the variation information 210b of the original right image 200b.

The shielded area information generation unit 121 generates the left shielded area information 310a by calculating the composite error between the original left image 200a and the virtual left view image 300a. The shielded area information generating unit 121 calculates the combined error between the original right image 200b and the virtual viewpoint right image 300b to generate the right shielded area information 310b.

At this time, the shielded region information generation unit 121 stores the colors of the positions of the left shielded region information 310a and the right shielded region information 310b in a binary image or an unshielded viewpoint color. For example, the shielded area information generating unit 121 fills the color of the shielded area in the virtual viewpoint left image 300a with the color of the same area of the original right image 200b, which is the opposite viewpoint, Is filled with the image color for the same area of the original left image 200a, which is the opposite view, and stored.

The shielded area according to the embodiment of the present invention is not limited to the above-described method, and other methods capable of generating a shielded area may be applied to generate shielded area information.

Referring again to FIG. 1, the region dividing unit 122 receives the original left image and the original right image from the image receiving unit 110. The area division unit 122 receives the left shielded area information and the right shielded area information from the shielded area information generation unit 121. [ The region dividing unit 122 generates the left region dividing image as a result of dividing the original left image into the shielded region and the non-shielded region using the left shielded region information. In the same manner, the region dividing unit 122 generates the right region divided image as a result of dividing the original right image into the shielded region and the non-shielded region using the right shielded region information. The region dividing unit 122 transmits the left region classification image and the right region classification image to the image processing unit 130. [

For example, referring to FIG. 3, the region dividing unit 122 receives the original left image 200a and the original right image 200b from the image receiving unit 110. FIG. The region dividing section 122 receives the left shielded region information 320a and the right shielded region information 320b from the shielded region information generation section 121. [ The region dividing unit 122 divides the original left image 200a into a shielded region and a non-shielded region based on the left shielded region information 320a to generate a left regionized image 400a. The region dividing unit 122 divides the original right image 200b into a shielded region and a non-shielded region based on the right shielded region information 320b to generate a right regionized image 400b. The non-shielded area according to the embodiment of the present invention is defined as the remaining area except for the shielded area.

Referring again to FIG. 1, the image processing unit 130 includes a feature detection unit 131, a division unit 132, a reconstruction unit 134, a variation interpolation unit 133, and an image generation unit 135.

The feature detection unit 131 receives the stereo original image from the image receiving unit 110. The feature detection unit 131 detects scene feature information such as image importance, sparse variation, linear component, etc. from the stereo original image. Here, an example of the rare variation image is shown in Fig. FIG. 4A is an example of a left unshielded rare displacement image, and FIG. 4B is an example of a right unshielded rare displacement image. At this time, the detected scene feature is used as a limiting condition when generating the virtual point multi-view image using the mesh warping technique in the image generating unit 135.

 The division unit 132 receives the stereo original image from the image receiving unit 110. The segmentation unit 132 divides the stereo original image into meaningful regions through region segmentation for the stereo original image to generate a segmented segmented image, and an example of the segmented segmented image is shown in FIG. 5 (a) is a stereo original image, and FIG. 5 (b) is a region divided image generated by dividing FIG. 5 (a) into meaningful regions. At this time, the segmentation information can be generated by dividing the stereo original image into meaningful regions based on the color information or based on the color and depth information.

1 and 3, the transition interpolator 133 receives the original left image 200a and the original right image 200b from the image receiving unit 110 to generate a reconstructed image at a virtual viewpoint, And receives the rare variation image from the feature detection unit 131. The transition interpolation unit 133 receives the left region classification image 400a and the right region classification image 400b from the region classification unit 122 of the shielded region management unit 120. [ The disparity interpolation unit 133 performs disparity interpolation on the left screen area information 320a of the left area classification image 400a to generate a left screen area disparity interpolation image. The transition interpolating unit 133 interpolates the sides of the right screen region information 320b of the right region screen image 400b to generate a right screen region variance interpolated image. At this time, the disparity interpolation unit 133 performs disparity interpolation using pixels having an effective disparity around the disparity, and performs interpolation by weighting a disparity near a disparate background. The transition interpolating unit 133 selects a pixel having a high mutual reliability with a high weight when selecting a pixel to be interpolated. When the area division information is transferred from the partitioning unit 132, the transition interpolating unit 133 performs the displacement interpolation for each divided area.

Referring again to FIG. 1, the reconstruction unit 134 reconstructs the left and right original images generated by dividing the original left and right images into a shielded area and a non-shielded area from the region dividing unit 122 of the shielded area management unit 120, And the right region classification image. The reconstruction unit 134 receives the scene feature information from the feature detection unit 131. The reconstructing unit 134 generates the left reconstructed image and the right reconstructed image by reconstructing the shielded region information of the left region classification image and the right region classification image using the rare variation information of the scene feature information to fit the virtual viewpoint position to be generated .

For example, as shown in FIG. 6, the reconstructing unit 134 receives the left region classification image 400a and the right region classification image 400b from the region classification unit 122. FIG. The reconstructing unit 134 receives the left sparse image 410a and the right sparse image 410b from the feature detecting unit 131. The reconstruction unit 134 reconstructs 330a in such a manner that the left shielded region information 320a interpolated at the virtual viewpoint position to be generated in the left region classification image 400a is projected based on the left sparse image 410a, Thereby generating a reconstructed image 500a. The right shield region information 320b is interpolated to the virtual point position to be generated in the right region discrimination image 400b based on the right sparse deviation image 410b of the reconstruction unit 134, ) To generate a right reconstructed image 500b.

)(0<

<1)와 좌측 차폐영역 정보(320a)의 길이(N1)를 곱하여 좌측 차폐영역 정보(320a)의 길이(N1

)를 재구성한다. 그리고, 재구성부(134)는 변이 보간된 좌측 차폐영역 정보(320a)를 가상시점 위치(

)(0<

<1)와 곱하여 좌측 차폐영역 정보(320a)의 이동위치를 결정하여 좌측 재구성 차폐정보(330a)를 생성한다. More specifically, when the reference time point is the time point of the left region classification image 400a and the length of the left shielded region information 320a calculated beforehand when the shielded region information is input is 'N1', the reconstruction unit 134 Based on the left sparse mutation image 410a, a virtual viewpoint position from the left direction to the right direction (

) (0 <

<1) and the length N1 of the left shielded area information 320a to calculate the length N1 of the left shielded area information 320a

). The reconstructing unit 134 reconstructs the interpolated left shielded area information 320a from the virtual viewpoint position (

) (0 <

<1) to determine the movement position of the left shielded area information 320a to generate the left reconstructed shielded information 330a.

)(0<

<1)와 우측 차폐영역 정보(320b)의 길이(N2)를 곱하여 우측 차폐영역 정보(320b)의 길이(N2

(1-

))를 재구성한다. 그리고, 재구성부(134)는 변이 보간된 우측 차폐영역 정보(320b)를 가상시점 위치(1-

)(0<

<1)와 곱하여 우측 차폐영역 정보(320b)의 이동위치를 결정하여 우측 재구성 차폐정보(330b)를 생성한다.In the same way, when the reference time point is the time point of the right region classification image 400b and the length of the previously calculated right shielded region information 320b is 'N2' when the shielded region information is inputted, the reconstruction unit 134 Based on the right sparse mutation image 410b, the virtual viewpoint position (1-

) (0 <

<1) and the length N2 of the right shielded area information 320b to calculate the length N2 of the right shielded area information 320b

(One-

)). Then, the reconstructing unit 134 obtains the right-side shielded area information 320b interpolated from the virtual viewpoint position 1-

) (0 <

<1) to determine the movement position of the right shielded area information 320b to generate the right reconstructed shielded information 330b.

At this time, in order to remove the boundary noise in the shielding region in the process of reconstruction, post-processing such as extending the boundary of the shielding region in the direction of the background may be performed.

As described above, in the embodiment of the present invention, since the unshielded object is shifted from the shielded region by a large amount, the shielded region is blocked or newly appeared.

Referring back to FIG. 1, the image generating unit 135 receives the left shielded region shifted interpolation image and the right shielded region shifted interpolated image from the transition interpolation unit 133. The image generating unit 135 receives the left side rare image and the right rare side image from the feature detecting unit 131. The image generating unit 135 generates a virtual viewpoint image by using a mesh warping based on the left shielded region side interpolation image, the right shielded region side-by-side interpolation image, the left rare side image, and the right side rare image, The blending weight for each input image is calculated. In other words, the image generation unit 135 determines a blending weight value between a value of "0" and a value of 1 according to the distance between the input image and the virtual intermediate point multi-view image to be generated, The blending weight is calculated for each region using the segmentation information for the original left image and the original right image. Meanwhile, the image generating unit 135 calculates the blending weight in units of the mesh grid when the region dividing information is not used, calculates the blending weight only in the unshielded region, and calculates the blending weight only for the unshielded view image Determine the blending weight to use.

The image generating unit 135 generates the original left image, original right image, left shielded region shifted interpolated image and right shielded region shifted interpolated image, left sparse shifted image, and left shadowed region image using the blending weight calculated by dividing into the shielded region and the non- The virtual intermediate viewpoint multi-view image is generated by mesh warping and blending of the right sparse image. In this case, when the segmentation information is selectively available, the mesh warping is divided into a shielded area and a non-shielded area in units of divided areas to generate a virtual intermediate viewpoint multi-view image. Also, the image generating unit 135 may selectively perform mutation-based interpolation for the shielded area instead of mesh warping. At this time, the boundary post-processing for the shielded area is selectively performed.

7, the image generating unit 135 receives the original left image 200a and the original right image 200b from the image receiving unit 110 and receives the left rare image 200b from the feature detecting unit 131. [ (410a) and the right sparse deviation image (410b). The image generating unit 135 receives the left reconstructed image 500a and the right reconstructed image 500b from the reconstructing unit 134 and receives the left side shielded area side-transition image 600a and the right side shielded image 500b from the side- And receives the disparity interpolation image 600b. The image generating unit 135 generates the virtual intermediate viewpoint multi-view image 700 (see FIG. 1) through mesh warping and blending of the original left image 200a and the original right image 200b according to the blending weights calculated by dividing into the shielded area and the non- ).

More specifically, when the blending weight of the original left image 200a is "0" and the blending weight of the original right image 200b is 1 ", the image generating unit 135 generates a virtual intermediate point multi- 700 is determined to be a value between "0" and "1" according to the distance between the original left / right images 200a and 200b and the virtual intermediate viewpoint multi view image 700. In FIG. 7, It is assumed that the weight of the original left / right images 200a and 200b is determined to be "0.5 " when generating the virtual intermediate viewpoint multi-view image 700 divided into the shielded regions. The shielded area of the original left image 200a, that is, the left reconstruction shielding information 330a, is generated from the original right image 200b based on the right shielded area variation interpolated image 600b, Is determined by using the mesh warping of the area corresponding to the shielding area of the original, and the non- When generating the virtual intermediate viewpoint multi-view image 700, the image generation unit 135 determines the shadowed area of the original right image 200b using the mesh warping for each region of the image 200a, The reconstruction blocking information 330b is determined using the mesh warping of the area corresponding to the shielded area of the original left image 200a based on the left shielded area variation interpolated image 600a, Using the mesh warping for each region of the original right image 200b.

8 is a flowchart illustrating a method of performing image warping in consideration of shielded area information according to an embodiment of the present invention.

As shown in FIG. 8, the image receiving unit 110 of the image warping apparatus 100 according to the embodiment of the present invention receives the stereo original image (S100). Stereo source video includes source left video and source right video. The image receiving unit 110 transmits the inputted stereo original image to the shielded area management unit 120 and the image processing unit 130.

The shielded area information generation unit 121 of the shielded area management unit 120 generates the left shielded area information and the right shielded area information using the stereo original image (S110). The shielded area information generation unit 121 delivers the left shielded area information and the right shielded area information to the area classification unit 122. The region dividing unit 122 divides the original left image and the original right image into a shielded area and a non-shielded area using the left shielded area information and the right shielded area information (S120). That is, the region dividing unit 122 divides the original left image into the shielded region and the non-shielded region based on the left shielded region information to generate the left region divided image 400a. The region dividing unit 122 divides the original right image into a shielded region and a non-shielded region based on right shielded region information to generate a right regionized image 400b (see FIG. 3).

The side interpolation unit 133 of the image processing unit 130 receives the left side region classification image 400a and the right side region classification image 400b from the region division unit 122. [ The disparity interpolation unit 133 performs disparity interpolation on the left screen area information 320a of the left area classification image 400a to generate a left screen area disparity interpolation image. The transition interpolating unit 133 interpolates the sides of the right-side shielded region information 320b of the right-side region-separated image 400b to generate a right-side shielded-region interpolated image (S130).

The reconstruction unit 134 receives the left region classification image 400a and the right region classification image 400b from the region classification unit 122. [ The feature detection unit 131 receives the left sparse mutation image 410a and the right sparse mutation image 410b from the feature detection unit 131. [ The reconstruction unit 134 reconstructs 330a in such a manner that the left shielded region information 320a interpolated at the virtual viewpoint position to be generated in the left region classification image 400a is projected based on the left sparse image 410a, The reconstructed image 330a is generated in such a manner that the reconstructed image 500a is generated and the right shielded region information 320b is interpolated and interpolated to the virtual viewpoint position to be generated in the right region divided image 400b based on the right sparsely located image 410b. To generate a right reconstructed image 500b (S140) (see FIG. 6).

The image generation unit 135 divides the image into a shielded area and a non-shielded area, and calculates a blending weight for each input image (S150). The image generating unit 135 generates a virtual intermediate viewpoint multi-view image by mesh warping and blending of the original left image 200a and the original right image 200b according to the blending weight calculated by dividing into the shielded area and the non- (S160) (refer to FIG. 7).

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, And various modifications may be made by those skilled in the art without departing from the scope of the present invention.

100: image warping device 110: image receiving unit
120: shielded area management unit 121: shielded area information generation unit
122: area division unit 130: image processing unit
131: feature detecting unit 132:
133: Transition interpolation section 134: Reconstruction
135:

Claims (1)

A shielded area management unit for generating shielded area information based on a stereo original image; And
The method of claim 1, further comprising: interpolating the variation of the shielded area information based on the sparse disparity information of the stereo original image, and reconstructing the shielded area information according to the interpolated shielded area information, And a non-shielded area to generate a virtual intermediate view image
Wherein the image warping device comprises:

Images