KR102455843B1 - Apparatus for reconstructing image and method using the same - Google Patents

Abstract

An image reconstruction apparatus and method are disclosed. An image reconstruction apparatus according to an embodiment of the present invention includes: an image segmentation unit for segmenting input images into segments using an image segmentation (IMAGE SEGMENTATION) technique; a mesh component that configures a mesh corresponding to the size of the segment; a feature point matching unit for extracting feature points of the mesh and matching the feature points between the input images; a mesh warping unit calculating a position of a feature point of a target image based on the corresponding feature points, and warping the mesh at the calculated position; and a texture mapping unit for texture mapping the pixels of the input image to the target image, and an image generating unit for generating a final image by interpolating an empty area of the target image with reference to neighboring pixels.

Description

Image reconstruction apparatus and method {APPARATUS FOR RECONSTRUCTING IMAGE AND METHOD USING THE SAME}

The present invention relates to a technique for reconstructing an image while minimizing image distortion.

IMAGE WARPING is to change the shape of an image by changing the positions of pixels of the original image in the image. Image warping may change the shape of the image, such as the size, length, and thickness of the input image, according to a specific rule. Such image warping was initially used to restore distortion of images received from satellites and spacecraft, but is now widely used for transformation and restoration of images captured by cameras or special effects of movies. This image warping technique includes mesh image warping. Mesh image warping is useful for warping based on feature points of an image, and is a method of performing image warping in groups by forming a mesh group in a triangular or square shape in a specific part of an input image.

However, such mesh image warping has problems in that, when the area of the mesh group is subdivided too small, the image warping processing speed decreases, distortion occurs in the image, and noise occurs at the edge of the object.

On the other hand, Korean Patent Application Laid-Open No. 2014-0102999 "Apparatus and method for image warping" interpolates the disparity of the shielding area information based on the sparse disparity information of the stereo original image, and the stereo original image according to the disparity interpolated shielding area information. Disclosed is a technology for generating a virtual mid-view image by dividing a shielded area and a non-occluded area during mesh warping.

However, Korean Patent Application Laid-Open No. 2014-0102999 is silent on the process of transforming and optimizing a mesh with respect to image distortion that occurs during mesh image warping.

An object of the present invention is to minimize image distortion that occurs when image warping is used to reconstruct an image.

Another object of the present invention is to optimize a warped mesh on a target image in order to minimize image distortion.

An image reconstruction apparatus according to an embodiment of the present invention for achieving the above object includes: an image segmentation unit for segmenting input images into segments using an image segmentation (IMAGE SEGMENTATION) technique; a mesh component that configures a mesh corresponding to the size of the segment; a feature point matching unit for extracting feature points of the mesh and matching feature points of a first input image with feature points of a second input image; a mesh warping unit calculating a position of a feature point of a target image based on the corresponding feature points, and warping the mesh of the first input image to the calculated position; and a texture mapping unit for texture mapping the pixels of the first input image to the target image, and an image generating unit for generating a final image by interpolating the pixels in the blank area of the target image with reference to neighboring pixels.

In this case, the image divider may group neighboring pixels having a high similarity to the pixels of the input images.

In this case, the mesh warping unit may calculate the first feature point of the target image based on the corresponding feature points and the position of the target image located between the input images.

In this case, the mesh warping unit may calculate relation coefficients from barycentric coordinates of the vertices of the mesh corresponding to the vertices of the mesh of the first feature point.

In this case, the mesh warping unit may calculate vertices of the target image by using the first feature point of the target image and the relation coefficients.

In this case, the mesh warping unit may calculate the second feature point of the target image by using the vertices of the target image and the relation coefficients.

In this case, the mesh warping unit calculates vertices of the mesh having the minimum error between the first feature point and the second feature point of the target image, and applies the mesh of the first input image to the vertices of the mesh having the minimum error. can warp

At this time, the mesh warping unit is used to minimize the error value between one optimal vertex calculated using the vertices of the mesh warped in the target image and the vertex of the warped mesh corresponding to the calculated one optimal vertex. Optimal vertices may be calculated, and the warped mesh may be deformed based on all the calculated optimal vertices.

In this case, the texture mapping unit may calculate a transformation matrix that reflects changes in vertices before and after warping by using a transformation technique based on the number of vertices of the mesh.

In this case, the texture mapping unit may perform the texture mapping of the pixels of the first input image to the target image using an additive transformation matrix.

In addition, an image reconstruction method according to an embodiment of the present invention for achieving the above object is a method using an image reconstruction apparatus, comprising: segmenting input images into segments using an image segmentation (IMAGE SEGMENTATION) technique; constructing a mesh corresponding to the size of the segment; extracting the feature points of the mesh and matching the feature points of the first input image with the feature points of the second input image; calculating a location of a feature point of a target image based on the corresponding feature points, and warping the mesh of the first input image at the calculated location; and texture mapping the pixels of the first input image to the target image, and generating a final image by interpolating pixels in the blank area of the target image with reference to neighboring pixels.

In this case, the dividing may include grouping neighboring pixels having a high similarity to the pixels of the input image.

In this case, the warping may include calculating a first feature point of the target image based on the corresponding feature points and a position of the target image located between the input images.

In this case, the warping may include calculating relation coefficients from barycentric coordinates of the vertices of the mesh corresponding to the vertices of the mesh of the first feature point.

In this case, the warping may include calculating vertices of the target image by using the first feature point of the target image and the relation coefficients.

In this case, the warping may include calculating the second feature point of the target image by using the vertices of the target image and the relation coefficients.

In this case, the warping includes calculating vertices of the mesh having the minimum error between the first feature point and the second feature point of the target image, and changing the mesh of the first input image to the vertices of the mesh having the minimum error. can warp on

In this case, the warping may include minimizing an error value between one optimal vertex calculated using the vertices of the mesh warped on the target image and the vertex of the warped mesh corresponding to the calculated one optimal vertex. The remaining optimal vertices may be calculated, and the warped mesh may be deformed based on all the calculated optimal vertices.

In this case, the texture mapping may use a transformation technique based on the number of vertices of the mesh to calculate a transformation matrix reflecting changes in vertices before and after warping.

In this case, the texture mapping may include performing the texture mapping of the pixels of the first input image to the target image using an additive transformation matrix.

The present invention can minimize image distortion that occurs when image warping is used to reconstruct an image.

In addition, the present invention may optimize a warped mesh on a target image in order to minimize image distortion.

1 is a block diagram illustrating an image reconstruction apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating an image reconstruction method according to an embodiment of the present invention.
3 is a diagram illustrating an image divided into blocks according to an embodiment of the present invention.
4 is a diagram illustrating a mesh generated in a segmented segment according to an embodiment of the present invention.
5 is a view showing a feature point inside a mesh according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating a central coordinate system of a triangle for calculating the feature point shown in FIG. 5 .
7 is a diagram illustrating a process of optimizing a shape by warping a mesh according to an embodiment of the present invention.
8 is a view showing that the mesh is warped and the shape of the mesh is deformed according to an embodiment of the present invention.
9 is a diagram illustrating warping of a mesh using a transformation matrix according to an embodiment of the present invention.
10 is a block diagram illustrating a computer system according to an embodiment of the present invention.

The present invention will be described in detail with reference to the accompanying drawings as follows. Here, repeated descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations will be omitted. The embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an image reconstruction apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , an image reconstruction apparatus according to an embodiment of the present invention includes an image segmentation unit 110 , a mesh construction unit 120 , a feature point matching unit 130 , a mesh warping unit 140 , and a texture mapping unit ( 150 ) and an image generator 160 .

The image segmentation unit 110 may receive images and divide the images into segments by using an image segmentation (IMAGE SEMEENTATION) technique. In this case, the images may correspond to the two first input images and the second input images having different viewpoints. In this case, the image segmentation technique may group neighboring pixels having a high similarity to the pixels of the input images to segment the image segments. Referring to FIG. 3 , it can be seen that the input image 301 is divided into segments.

The mesh configuration unit 120 may configure a mesh corresponding to the size of the segment. In this case, the mesh composing unit 120 may configure a mesh for each segment constituting the image in consideration of the sizes of segments into which the image is divided. A mesh may be configured per segment, or may be configured independently of meshes configured in other segments. Referring to FIG. 4 , it can be seen that the mesh 302 is configured to correspond to one segment size.

The keypoint matching unit 130 may extract the keypoint and match the keypoints between the input images.

First, the keypoint matching unit 130 may extract a keypoint of an input image. Feature point extraction may be input through a user input, and an automatic extraction technique may be used. As the automatic extraction technique, a generally well-known stereo matching (STEREO MATCHING), a feature point extraction (FEATURE POINT EXTRACTION), or a feature point matching (FEATURE POINT MATCHING) technique may be used. Referring to FIG. 5 , it can be seen that the feature point 303 is extracted inside the mesh corresponding to the segment size.

In this case, the keypoint matching unit 130 may match the extracted keypoints of the first input image with the extracted keypoints of the second input image in a 1:1 ratio using an automatic extraction technique.

The mesh warping unit 140 may calculate a feature point of the target image based on the corresponding feature points, and warp the mesh on the feature point of the target image.

First, the mesh warping unit 140 may select a mesh of a corresponding feature point, and calculate center coordinates and relationship coefficients using vertices of the selected mesh. In this case, the vertices of the mesh may correspond to coordinates of the vertices of the triangular mesh including the feature point.

In this case, the mesh warping unit 140 may calculate relation coefficients from the center coordinates (BARYCENTRIC COORDINATES) of the vertices. Referring to FIG. 6 , it can be seen that the vertices are represented by v ₁ , v ₂ , and v ₃ , the central coordinates are p, and the distances from the central coordinates to the vertices are represented by A ₁ , A ₂ , and A ₃ . In this case, the center coordinate p and the relation coefficients α, β, and γ may be calculated by Equation 1 below.

[Equation 1]

The feature point X _i (x _i ,y _i ) in the mesh may have coordinates of (x _i ,y _i ) in the image coordinate system, and based on the coordinates of the vertices v _i1 , v _i2 , v _i3 Equation 1 It can be calculated using Equation 2 below.

[Equation 2]

The mesh warping unit 140 may calculate the position of the feature point of the mesh of the target image by using Equations 1 and 2. The position of the feature point of the target image may be calculated using a proportional relationship between the feature point of the first input image and the feature point of the second input image.

For example, when the feature point of the first input image is p ₁ (x ₁ ,y ₁ ) and the feature point of the second input image is p ₂ (x ₂ ,y ₂ ), the feature point of the target image j is p _j (x _j ,y _j ), and the feature point of the target image may be calculated using Equation 3 below. Here, j may have a real value of 0<j<1.

[Equation 3]

The feature point calculated using Equation 3 may correspond to the first feature point.

In this case, the mesh warping unit 140 may calculate vertices of the target image by using the calculated first feature points and relation coefficients. All the feature points may be expressed by the following determinant based on Equation (2).

The determinant can be expressed as Ac=b if A is the matrix representing the relation coefficients, c is the matrix representing the coordinates of the vertices, and b is the matrix representing the coordinates of the feature points. Here, the mesh warping unit 140 may substitute the coordinates b′ of the first feature point of the target image into the determinant. The determinant can be transformed into c'=A ^{- 1} b' to obtain the vertices (c') of the target image. That is, the mesh warping unit 140 may warp the mesh on the target image by using the transformed determinant. At this time, the mesh warping unit 140 may warp all meshes constituting one segment rather than warping only one mesh, and the meshes constituting one segment are not independent but share each other's vertices Therefore, it is possible to optimize the deformation of the entire mesh using Equation (4).

[Equation 4]

부분은 목표 영상으로 워핑된 정점들의 좌표와 관계 계수들을 이용하여 생성된 목표 영상의 제2 특징점을 나타낸 것이다. 이 때,

부분은 제1 입력 영상의 특징점(x_i,y_i)이 목표 영상으로 전방 사상(FORWARD WARPING)된 제1 특징점에 상응할 수 있다. 이 때, 메시 워핑부(140)는 수학식 4에서 E₁이 최소값을 갖는 메시의 정점들(v`<sub>i1</sub>, v`_i2, v`_i3)을 산출할 수 있다.In Equation 4,

The portion indicates the second feature point of the target image generated using the coordinates and relation coefficients of the vertices warped into the target image. At this time,

The portion may correspond to the first feature point where the feature point (x _i , y _i ) of the first input image is forward-warped to the target image. At this time, the mesh warping unit 140 may calculate the vertices v` <sub>i1</sub> , v` _i2 , v` _i3 of the mesh having the minimum value of E ₁ in Equation 4 .

Also, the mesh warping unit 140 may perform a process of optimizing the shape of the mesh. Referring to FIG. 7 , it can be seen that the shape of the mesh is transformed from the mesh 304 before warping to the mesh 305 after warping. In this case, the mesh warping unit 140 may calculate optimal vertices using Equations 5 and 6, and configure the optimized mesh 306 .

[Equation 5]

를 산출할 수 있다.The mesh warping unit 140 is a vertex optimized using Equation 5 based on the two vertices v _m1 and v _m2 in the mesh 305 after warping.

can be calculated.

[Equation 6]

,

,

)을 산출할 수 있다.At this time, the mesh warping unit 140 uses Equation ₆ to determine the optimal vertices (

,

,

) can be calculated.

That is, the mesh warping unit 140 may optimize the deformation of the warped mesh by using optimal vertices in the target image that minimizes both E ₁ and E ₂ .

The texture mapping unit 150 may TEXTURE MAPPING the pixels of the input image to the target image. In this case, the texture mapping unit 150 may forward-map the pixels of the first input image using the transformation matrix, and may perform the backward mapping using the inverse transformation matrix. In this case, the texture mapping unit 150 may perform texture mapping by mapping pixels. In this case, the transformation matrix may be generated using an affine transformation (AFFINE TRANSFORMATION) or a viewpoint transformation (PERSPECTIVE TRANSFORMATION) based on the number of vertices. In this case, the affine transformation may calculate a feature point using three vertices, and may be expressed as a determinant of the following HOMOGENEOUS form.

In this case, the viewpoint transformation may calculate a feature point using four vertices, and may be expressed by the following homogeneous determinant.

Referring to FIG. 9 , the vertices v ₁ , v ₂ , v ₃ , v ₄ of the original mesh 311 are vertices v` <sub>1</sub> , v` ₂ , v` <sub>3</sub> , v using the transformation matrix T. It can be seen that it is composed of a deformed mesh 312 that has been mapped forward to ` ₄ .

In addition, the vertices v` <sub>1</sub> , v` ₂ , v` <sub>3</sub> , v` ₄ of the deformed mesh 312 are obtained by using the transformation inverse matrix (T ^-1 ) to obtain the vertices v ₁ , v ₂ , v ₃ , v It can be seen that it is back mapped to ₄ ) and consists of the original mesh 311 .

The image generator 160 may generate a final image. That is, the image generator 160 may generate the final image by using an image completion (IMAGE COMPLETION0) technique that interpolates a blank area of the target image. In this case, when an unwarped empty area (HOLE) exists in the target image after texture mapping, the image generator may interpolate the empty area by referring to similar pixels in the adjacent area based on the spatial axis/time axis.

2 is a flowchart illustrating an image reconstruction method according to an embodiment of the present invention.

Referring to FIG. 2 , in the image reconstruction method according to an embodiment of the present invention, an image is first divided (S210).

That is, in step S210 , the images may be received and divided into segments by using an image segmentation technique. In this case, the images may correspond to the two first input images and the second input images having different viewpoints. In this case, the image segmentation technique may group neighboring pixels having a high similarity to the pixels of the input images to segment the image segments. Referring to FIG. 3 , it can be seen that the input image 301 is divided into segments.

In addition, the image reconstruction method may configure a mesh (S220).

That is, in step S220 , a mesh may be constructed for each segment constituting the image in consideration of the sizes of segments into which the image is divided. A mesh may be configured per segment, or may be configured independently of meshes configured in other segments. Referring to FIG. 4 , it can be seen that the mesh 302 is configured to correspond to one segment size.

In addition, the image reconstruction method may correspond to the feature points (S230).

That is, in step S230, the feature points may be extracted and the feature points between the input images may be matched.

First, in step S230, a feature point of an input image may be extracted. Feature point extraction may be input through a user input, and an automatic extraction technique may be used. As the automatic extraction technique, a generally well-known stereo matching (STEREO MATCHING), a feature point extraction (FEATURE POINT EXTRACTION), or a feature point matching (FEATURE POINT MATCHING) technique may be used. Referring to FIG. 5 , it can be seen that the feature point 303 is extracted inside the mesh corresponding to the segment size.

In this case, in step S230, the extracted feature points of the first input image and the extracted feature points of the second input image may be matched 1:1 by using an automatic extraction technique.

Also, the image reconstruction method may warp the mesh (S240).

That is, in step S240 , the feature points of the target image may be calculated based on the corresponding feature points, and the mesh may be warped on the feature points of the target image.

First, in step S240 , a mesh of corresponding feature points may be selected, and center coordinates and relationship coefficients may be calculated using vertices of the selected mesh. In this case, the vertices of the mesh may correspond to coordinates of the vertices of the triangular mesh including the feature point.

In this case, the feature step S240 may calculate a relationship coefficient from the center coordinates (BARYCENTRIC COORDINATES) of the vertices. Referring to FIG. 6 , it can be seen that the vertices are represented by v ₁ , v ₂ , and v ₃ , the central coordinates are p, and the distances from the central coordinates to the vertices are represented by A ₁ , A ₂ , and A ₃ . In this case, the center coordinate p and the relation coefficients α, β, and γ may be calculated by Equation 1 above. The feature point X _i (x _i ,y _i ) in the mesh may have coordinates of (x _i ,y _i ) in the image coordinate system, and based on the coordinates of the vertices v _i1 , v _i2 , v _i3 Equation 1 It can be calculated using Equation 2 above.

In this case, in step S240, the position of the feature point of the mesh of the target image may be calculated using Equations 1 and 2. The position of the feature point of the target image may be calculated using a proportional relationship between the feature point of the first input image and the feature point of the second input image.

For example, when the feature point of the first input image is p ₁ (x ₁ ,y ₁ ) and the feature point of the second input image is p ₂ (x ₂ ,y ₂ ), the feature point of the target image j is p _j (x _j , y _j ), and the feature point of the target image may be calculated using Equation 3 above. Here, j may have a real value of 0<j<1. The feature point calculated using Equation 3 may correspond to the first feature point.

In this case, in step S240, vertices of the target image may be calculated using the calculated first feature points and relation coefficients. All the feature points may be expressed by the following determinant based on Equation (2).

부분은 목표 영상으로 워핑된 정점들의 좌표와 관계 계수들을 이용하여 생성된 목표 영상의 제2 특징점을 나타낸 것이다. 이 때,

부분은 제1 입력 영상의 특징점(x_i,y_i)이 목표 영상으로 전방 사상(FORWARD WARPING)된 제1 특징점에 상응할 수 있다. 이 때, 단계(S240)는 수학식 4에서 E₁이 최소값을 갖는 메시의 정점들(v`<sub>i1</sub>, v`_i2, v`_i3)을 산출할 수 있다.The determinant can be expressed as Ac=b if A is the matrix representing the relation coefficients, c is the matrix representing the coordinates of the vertices, and b is the matrix representing the coordinates of the feature points. Here, in step S240, the coordinates b′ of the first feature point of the target image may be substituted into the determinant. The determinant can be transformed into c'=A ^{- 1} b' to obtain the vertices (c') of the target image. That is, in step S240, the mesh may be warped on the target image using the transformed determinant. At this time, in step S240, not only one mesh is warped, but all meshes constituting one segment can be warped. Since the meshes are not independent but share each other's vertices, Equation 4 is used to The deformation of the entire mesh can be optimized. In Equation 4,

The portion indicates the second feature point of the target image generated using the coordinates and relation coefficients of the vertices warped into the target image. At this time,

The portion may correspond to the first feature point in which the feature point (x _i , y _i ) of the first input image is forward-warped to the target image. In this case, step S240 may calculate the vertices v` <sub>i1</sub> , v` _i2 , v` _i3 of the mesh having the minimum value of E ₁ in Equation 4 .

를 산출할 수 있다. 이 때, 단계(S240)는 수학식 6을 이용하여 E₂를 최소화하는 목표 영상의 최적 정점들(

,

,

)을 산출할 수 있다. In addition, step S240 may perform a process of optimizing the shape of the mesh. Referring to FIG. 7 , it can be seen that the shape of the mesh is transformed from the mesh 304 before warping to the mesh 305 after warping. In this case, in step S240 , optimal vertices may be calculated using Equations 5 and 6 and an optimized mesh 306 may be constructed. At this time, step S240 is a vertex optimized using Equation 5 based on the two vertices v _m1 and v _m2 in the mesh 305 after warping.

can be calculated. _At this time, step S240 is performed using the optimal vertices (

,

,

) can be calculated.

That is, in step S240 , deformation of the warped mesh may be optimized using optimal vertices in the target image that minimizes both E ₁ and E ₂ .

Also, the image reconstruction method may perform texture mapping ( S250 ).

That is, in step S250 , the pixels of the input image may be texture mapped to the target image. In this case, in step S250 , pixels of the first input image may be forward mapped using a transformation matrix, and may be backward mapped using an inverse transformation matrix. In this case, in step S250 , the pixels may be mapped to texture mapping. In this case, the transformation matrix may be generated using an affine transformation (AFFINE TRANSFORMATION) or a viewpoint transformation (PERSPECTIVE TRANSFORMATION) based on the number of vertices. In this case, the affine transformation may calculate a feature point using three vertices, and may be expressed as a determinant of the following HOMOGENEOUS form.

In this case, the viewpoint transformation may calculate a feature point using four vertices, and may be expressed by the following homogeneous determinant.

Referring to FIG. 9 , the vertices v ₁ , v ₂ , v ₃ , v ₄ of the original mesh 311 are vertices v` <sub>1</sub> , v` ₂ , v` <sub>3</sub> , v using the transformation matrix T. It can be seen that it is composed of a deformed mesh 312 that has been mapped forward to ` ₄ .

In addition, the vertices v` <sub>1</sub> , v` ₂ , v` <sub>3</sub> , v` ₄ of the deformed mesh 312 are obtained by using the transformation inverse matrix (T ^-1 ) to obtain the vertices v ₁ , v ₂ , v ₃ , v It can be seen that it is back mapped to ₄ ) and consists of the original mesh 311 .

In addition, the image reconstruction method may generate a final image (S260).

That is, in step S260 , a final image may be generated using an image completion (IMAGE COMPLETION0) technique that interpolates a blank area of the target image. In this case, in step S260, after texture mapping, if an unwarped empty area (HOLE) exists in the target image, the empty area can be interpolated by referring to similar pixels in the adjacent area based on the spatial axis/time axis. have.

3 is a diagram illustrating an image divided into blocks according to an embodiment of the present invention.

Referring to FIG. 3 , it can be seen that the input image 301 is divided into segments. In this case, the segment unit may correspond to an image segmented into blocks using an image segmentation technique. In this case, the image segmentation technique may group neighboring pixels having a high similarity to the pixels of the input image 301 .

4 is a diagram illustrating a mesh generated in a segmented segment according to an embodiment of the present invention.

Referring to FIG. 4 , it can be seen that the mesh 302 is configured to correspond to the segment size. The mesh 302 is formed in the entire input image, and the size of the mesh 302 may be determined based on the size of segments divided in the entire image.

5 is a view showing a feature point inside a mesh according to an embodiment of the present invention.

Referring to FIG. 5 , it can be seen that the feature point 303 is extracted from the inside of the mesh. In this case, extraction of key points may be input through a user input, and an automatic extraction technique may be used. As the automatic extraction technique, a generally well-known stereo matching (STEREO MATCHING), a feature point extraction (FEATURE POINT EXTRACTION), or a feature point matching (FEATURE POINT MATCHING) technique may be used. In this case, the feature point 303 may be a reference point for warping the mesh to the target image. When a target image is generated from two images, all feature points of the two images may be extracted, all the feature points corresponded 1:1, and the feature points of the target image may be calculated based on the corresponding feature points.

FIG. 6 is a diagram illustrating a central coordinate system of a triangle for calculating the feature point shown in FIG. 5 .

Referring to FIG. 6 , it can be seen that the vertices of the triangle are represented by v ₁ , v ₂ , v ₃ , the central coordinates are p, and the distances from the central coordinates to the vertices are A ₁ , A ₂ , and A ₃ . In this case, the center coordinate p and the relation coefficients α, β, and γ may be calculated by Equation 1 above.

7 is a diagram illustrating a process of optimizing a shape by warping a mesh according to an embodiment of the present invention.

Referring to FIG. 7 , it can be seen that the shape of the mesh is transformed from the mesh 304 before warping to the mesh 305 after warping. In this case, the mesh warping unit 140 may calculate optimal vertices using Equations 5 and 6, and configure the optimized mesh 306 .

8 is a view showing that the mesh is warped and the shape of the mesh is deformed according to an embodiment of the present invention.

Referring to FIG. 8 , it can be seen that the mesh 309 of the input image is warped to the target image to generate a mesh 310 with a deformed shape.

9 is a diagram illustrating warping of a mesh using a transformation matrix according to an embodiment of the present invention.

Referring to FIG. 9 , the vertices v ₁ , v ₂ , v ₃ , v ₄ of the original mesh 311 are vertices v` <sub>1</sub> , v` ₂ , v` <sub>3</sub> , v using the transformation matrix T. It can be seen that it is composed of a deformed mesh 312 that has been mapped forward to ` ₄ .

In addition, the vertices v` <sub>1</sub> , v` ₂ , v` <sub>3</sub> , v` ₄ of the deformed mesh 312 are obtained by using the transformation inverse matrix (T ^-1 ) to obtain the vertices v ₁ , v ₂ , v ₃ , v It can be seen that it is back mapped to ₄ ) and consists of the original mesh 311 .

10 is a block diagram illustrating a computer system according to an embodiment of the present invention.

Referring to FIG. 10 , an embodiment of the present invention may be implemented in a computer system 1100 such as a computer-readable recording medium. As shown in FIG. 10 , computer system 1100 includes one or more processors 1110 , memory 1130 , user input device 1140 , user output device 1150 and storage that communicate with each other via bus 1120 . (1160) may be included. In addition, the computer system 1100 may further include a network interface 1170 coupled to the network 1180 . The processor 1110 may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory 1130 or the storage 1160 . The memory 1130 and the storage 1160 may be various types of volatile or non-volatile storage media. For example, the memory may include a ROM 1131 or a RAM 1132 .

As described above, in the image reconstruction apparatus and method according to the present invention, the configuration and method of the embodiments described above are not limitedly applicable, but the embodiments are all of the embodiments so that various modifications can be made. Alternatively, some may be selectively combined and configured.

110: image segmentation unit 120: mesh construction unit
130: feature point matching unit 140: mesh warping unit
150: texture mapping unit 160: image generating unit
301: input image 302: mesh
303: feature point 304: mesh before warping
305: mesh after warping 306: optimized mesh
309: mesh 310: mesh
311: original mesh 312: warped mesh
1100: computer system 1110: processor
1120: bus 1130: memory
1131: rom 1132: ram
1140: user input device 1150: user output device
1160: storage 1170: network interface
1180: network

Claims (20)

an image segmentation unit for segmenting input images into segments using an image segmentation technique;
a mesh component that configures a mesh corresponding to the size of the segment;
a feature point matching unit for extracting feature points of the mesh and matching feature points of a first input image with feature points of a second input image;
a mesh warping unit calculating a position of a feature point of a target image based on the corresponding feature points, and warping the mesh of the first input image to the calculated position;
a texture mapping unit for texture mapping the pixels of the first input image to the target image; and
an image generator for generating a final image by interpolating the blank area of the target image with reference to neighboring pixels;
including,
The mesh warping unit
calculating vertices of the mesh having the minimum error between the first feature point and the second feature point of the target image, and warping the mesh of the first input image to the vertices of the mesh having the minimum error;
Calculating and calculating the remaining optimal vertices so that an error value between one optimal vertex calculated by using the vertices of the warped mesh in the target image and the vertex of the warped mesh corresponding to the one calculated optimal vertex is minimized; Image reconstruction apparatus, characterized in that the warped mesh is deformed based on all the optimal vertices. The method according to claim 1,
The image division unit
Image reconstruction apparatus, characterized in that grouping the neighboring pixels having a high similarity to the pixels of the input images. 3. The method according to claim 2,
The mesh warping unit
Image reconstruction apparatus, characterized in that for calculating the first feature point of the target image based on the corresponding feature points and the position of the target image located between the input images. 4. The method of claim 3,
The mesh warping unit
The image reconstruction apparatus of claim 1, wherein the relation coefficients are calculated from barycentric coordinates of the vertices of the mesh corresponding to the vertices of the mesh of the first feature point. 5. The method according to claim 4,
The mesh warping unit
The image reconstruction apparatus of claim 1, wherein the vertices of the target image are calculated by using the first feature point of the target image and the relation coefficients. 6. The method of claim 5,
The mesh warping unit
and calculating a second feature point of the target image by using the vertices of the target image and the relation coefficients. delete delete 7. The method of claim 6,
The texture mapping unit
Image reconstruction apparatus, characterized in that by using a transformation method based on the number of vertices of the mesh to calculate a transformation matrix reflecting the change of the vertices before and after warping. 10. The method of claim 9,
The texture mapping unit
The image reconstruction apparatus according to claim 1, wherein the texture mapping is performed on the pixels of the first input image to the target image by using an additive transformation matrix. In the method of using the image reconstruction apparatus,
segmenting the input images into segments using an IMAGE SEGMENTATION technique;
constructing a mesh corresponding to the size of the segment;
extracting the feature points of the mesh and matching the feature points of the first input image with the feature points of the second input image;
calculating a location of a feature point of a target image based on the corresponding feature points, and warping the mesh of the first input image at the calculated location;
texture mapping the pixels of the first input image to the target image; and
generating a final image by interpolating the blank area of the target image with reference to neighboring pixels;
including,
The warping step
calculating vertices of the mesh having the minimum error between the first feature point and the second feature point of the target image, and warping the mesh of the first input image to the vertices of the mesh having the minimum error;
Calculating and calculating the remaining optimal vertices so that an error value between one optimal vertex calculated by using the vertices of the warped mesh in the target image and the vertex of the warped mesh corresponding to the one calculated optimal vertex is minimized; An image reconstruction method, characterized in that the warped mesh is deformed based on all optimal vertices. 12. The method of claim 11,
The dividing step
An image reconstruction method, characterized in that grouping neighboring pixels having a high similarity to the pixels of the input images. 13. The method of claim 12,
The warping step
The image reconstruction method, characterized in that calculating the first feature point of the target image based on the corresponding feature points and the position of the target image located between the input images. 14. The method of claim 13,
The warping step
The image reconstruction method, characterized in that calculating the relation coefficients from the barycentric coordinates of the vertices of the mesh corresponding to the vertices of the mesh of the first feature point. 15. The method of claim 14,
The warping step
The image reconstruction method of claim 1, wherein the vertices of the target image are calculated by using the first feature point of the target image and the relation coefficients. 16. The method of claim 15,
The warping step
and calculating a second feature point of the target image by using the vertices of the target image and the relation coefficients. delete delete 17. The method of claim 16,
The texture mapping step
An image reconstruction method comprising calculating a transformation matrix reflecting changes in vertices before and after warping by using a transformation technique based on the number of vertices of the mesh. 20. The method of claim 19,
The texture mapping step
The image reconstruction method, characterized in that the texture mapping of the pixels of the first input image to the target image using an additive transformation matrix.

Images