KR101105675B1 - Method and apparatus of inpainting for video data - Google Patents

Abstract

The present invention relates to an image data inpainting method and apparatus for inpainting an image of an inpainting target area included in a plurality of consecutive image data in the same manner.

An image data inpainting method according to the present invention includes a first step of receiving a plurality of consecutive image data and stitching into one image data; Performing a band matching on the selected inpainting target area in the stitched image data; Removing a boundary of an edge of the inpainting target region with respect to the stitched image data in which the inpainting target region is band-matched; And dividing the stitched image data subjected to the boundary elimination of the edge into a plurality of consecutive image data.

Image data, inpainting, stitching, band matching, border elimination

Description

Image data inpainting method and apparatus {METHOD AND APPARATUS OF INPAINTING FOR VIDEO DATA}

The present invention relates to an image data processing technique, and more particularly, to an image data inpainting method and apparatus for inpainting the same inpainting target areas included in a plurality of consecutive image data.

In recent years, there are no movies without special effects, so special effects are widely used in movies. Among many special effects techniques, inpainting is widely used in various fields as a technique for filling a specific region removed from the live image to match the surrounding background. For example, after shooting a movie, if there is an object or person in the background that is out of date, the movie may be retaken, but inpainting technology is used in consideration of cost and time.

However, when a plurality of continuous image data such as a movie is reproduced in succession, when inpainting is performed on each of the plurality of image data, the degree of processing of the filled image or the image for inpainting differs for each image data. Could be done

The different images of the same area for each of a plurality of consecutive video data to be reproduced in succession are a cause of deterioration of the image quality when the plurality of consecutive video data are reproduced successively.

Therefore, in the related art, the inpainting processing is performed on the inpainting target areas for each of a plurality of consecutive image data in the same way, so that the inpainting target region can be consistent for each of the plurality of consecutive image data. It was desperately desired.

The present invention provides an image data inpainting method and apparatus for inpainting target areas included in a plurality of consecutive image data in the same manner so that the inpainting target regions of each of the plurality of continuous image data can be consistent. Its purpose is to provide.

According to an exemplary embodiment of the present invention, an image data inpainting method includes: a first step of receiving a plurality of continuous image data and stitching the image data into one image data; Performing a band matching on the selected inpainting target area in the stitched image data; Removing a boundary of an edge of the inpainting target region with respect to the stitched image data in which the inpainting target region is band-matched; And dividing the stitched image data subjected to the boundary elimination of the edge into a plurality of consecutive image data.

In addition, the image data inpainting processing apparatus according to the present invention includes a memory unit for storing a plurality of consecutive image data; Stitching the plurality of consecutive image data into one image data, performing band matching on an inpainting target region selected from the stitched image data, and applying the stitched image data to which the inpainting target region is band-matched. And a processor that removes a boundary of an edge of the inpainting target region and separates the stitched image data subjected to the boundary removal of the edge into a plurality of consecutive image data.

According to the present invention, the inpainting target area included in the plurality of continuous image data is inpainted equally, so that the inpainting result of each of the plurality of consecutive image data can be made consistent. When the video data of the picture is reproduced in sequence, there is an effect that the high quality video can be output.

In addition, the present invention has an effect of significantly reducing the inpainting processing time by automatically stitching a plurality of consecutive image data and converting the same image data into one image data.

In addition, the present invention has an effect of maximizing the user's convenience by automatically setting the inpainting target area in a plurality of consecutive image data.

In addition, the present invention has an effect that can be applied to a variety of image data by allowing the user to receive a feature point for the stitching for the image data that can not be automatically stitched.

In addition, the present invention has an effect of improving the inpainting result by dividing the inpainting target region to perform inpainting.

The present invention processes the inpainting for the inpainting target region included in the plurality of consecutive image data in the same manner so that the inpainting result for each of the plurality of image data can be consistent.

<Configuration of Inpainting Processing Unit>

A brief configuration of the inpainting processing apparatus according to a preferred embodiment of the present invention will be described with reference to FIG. 1.

The inpainting processor includes a processor 100, a memory unit 102, and a user interface 104.

The processor 100 receives a plurality of consecutive frames and stores the plurality of consecutive frames in the memory unit 102, and performs inpainting on a plurality of consecutive image data stored in the memory unit 102.

The memory unit 102 stores a variety of information including a processing program of the processor 100 and provides a storage area for performing inpainting according to a preferred embodiment of the present invention.

The user interface 104 receives various information from a user and provides the same to the processor 100.

<Inpainting Processing Method According to First Embodiment>

An inpainting processing method according to a first preferred embodiment of the present invention will be described in detail with reference to the flowchart of FIG. 2.

When the processor 100 is requested to operate in the first painting mode in which the inpainting target area is input by the user (step 200), the processor 100 detects feature points between consecutive image data (step 202).

The detection of the feature points is in accordance with the Kanade-Lucas-Tomasi Feature Tracker (KLT) method. This is disclosed in Birchfield, S., An implementation of the Kanade-Lucas-Tomasi feature tracker, http://www.ces.clemson.edu/stb/klt.

When feature points are detected according to the KLT method, the processor 100 detects a homography matrix for feature points of continuous image data (step 204).

Here, the homography matrix will be described in more detail based on two consecutive image data.

The homography matrix defines a relationship between two image data and is calculated based on corresponding feature points in the two image data.

FIG. 3 illustrates two adjacent image data, and the second image data b includes feature points X1 ′, X2 corresponding to each of the feature points X1, X2, X3, and X4 of the first image data a. ', X3', X4 ') exists. The relationship between the first feature points (X1, X2, X3, X4) and the second feature points (X1 ', X2', X3 ', X4') may be expressed as in Equation (1).

In Equation 1, H is a homography matrix, and in the case of a two-dimensional image, H may be represented by a 3 * 3 matrix, which may be developed in terms of H as in Equation 2.

When the preceding matrix is simplified to A in Equation 2, A is a matrix of (2 * n) x 9, and n is the number of feature points. That is, Equation 2 shows a matrix when four corresponding feature points exist in two adjacent frames, and as the corresponding feature points increase, rows of A increase. And h is calculated by S singular value decomposition (SINGULAR VALUE DECOMPOSITION, SVD).

The singular value decomposition of A can be decomposed into UDV ^T , where the last column of the V matrix is the h value, that is, the homographic matrix H according to the present invention.

The homography matrix may be obtained through a RANSAC process.

The RANSAC process is performed in the following order.

1. Select four arbitrary feature points between the two image data.

2. Calculate H between four feature points.

를 산출하며, 상기 P₁은 제1영상 데이터의 특징점, P₂는 제2영상 데이터의 특징점, K는 특징점이 수이다. 여기서, 상기 수학식 1에 따르면 P₂=HP₁이나, 상기 H 계산 상에 오차가 있을 수 있으므로, P₂와 HP₁은 동일한 값이 나오지 않는다. 상기 d_k는 상기 P₂와 HP₁사이의 오차를 나타낸다. 3. For all feature points

Where P ₁ is the feature point of the first image data, P ₂ is the feature point of the second image data, and K is the number of feature points. Here, according to Equation 1, P ₂ = HP ₁ or, since there may be an error in the H calculation, P ₂ and HP ₁ does not come the same value. D _k represents the error between P ₂ and HP ₁ .

4. The number of points whose d _k is equal to or less than a predetermined value is called NIi.

5. Perform steps 1 to 4 a predetermined number of times.

6. Recalculate H with the feature points when the NI i is maximum to obtain the final homography matrix H.

The RANSAC process is to calculate the best H through repetition, and the processor 100 repeats a predetermined number of times to select a random feature point to calculate H, and to determine the best H among them, d _k H is determined to be the best when the number NIi of the feature points that is equal to or less than the predetermined value is the best, and the corresponding H is determined as the homography matrix of the present invention.

When the detection of the homography matrix is completed, the processor 100 stitches consecutive image data into one according to the homography matrix. That is, the processor 100 obtains stitching image data by bilinear interpolation using a homography matrix. At this time, the area where the images overlap is calculated as an average value.

4 illustrates image data stitched according to the present invention, wherein the first image data and the second image data of FIG. 3 are bilinearly interpolated based on four feature points.

In operation 208, the processor 100 receives information on an inpainting target area input by the user through the user interface 104 with respect to the image data stitched into one.

When the information on the inpainting target region is input, the processor 100 performs band matching on the inpainting target region on the image data stitched into the one (step 210).

The band matching process will be described in more detail.

The processor 100 defines a target band having a constant thickness surrounding the target region for band matching.

FIG. 5A illustrates an inpainting target region, FIG. 5B illustrates an enlarged input image, a white region represents a target region, a red line represents a boundary of the target region, and FIG. 5C has a predetermined thickness based on the boundary. The target band is set and the blue area is the target band.

The target band is used for comparison with various candidate source bands in the image, and the source band refers to the remaining portion of the input image except for the target region in the same shape and thickness as the target band.

In FIG. 6A, the green portions indicate a portion of the candidate source band for comparison with the target band.

The processor 100 calculates a sum of a difference of gradients between the target band and the candidate source bands according to Equation 3 below.

는 각 색상 채널의 화소값이고,

는 각 색상 채널의 그래디언트이다.In Equation 3, k denotes an index of candidate source bands, Ck denotes a sum of differences of three color channels, and Gk denotes a difference of gradients. T denotes a target band, s denotes a source band, N denotes the total number of pixels of the band, and in c, R, G, and B are red, green, and blue color channels.

Is the pixel value for each color channel,

Is the gradient of each color channel.

Dk between the target band and the Kth source band may be calculated as Equation 4 as the sum of Ck and Gk.

과

는

와

의 가중치를 나타내기 위한 상수이다. In Equation 4

and

Is

Wow

Constant for indicating the weight of.

When the source band having the minimum Dk is detected through the above process, the processor 100 fills the internal color information of the source band in the inpainting target region, thereby completing band matching.

That is, in FIG. 6B, the green portion indicates the optimal source band with the minimum Dk, FIG. 6C is filled with the internal color information of the optimal source band in the target area, and FIG. 6D shows the result image data. .

When the band matching is completed, the processor 100 performs boundary elimination of an edge of the band matched inpainting target area (operation 212).

Here, the boundary removal process will be described in more detail.

When inpainting is performed by band matching according to a preferred embodiment of the present invention, a boundary appears between the target area and the peripheral area. 7B illustrates a boundary appearing between the target area and the peripheral area after band matching. In order to remove the boundary, the processor 100 calculates a ratio of pixel values of the target band and the optimal band, and recalculates and updates each pixel value of the inpainting target area using the ratio.

The first step of the boundary elimination is to calculate the ratio h (x, y) between the pixel value of the target band and the pixel value of the optimal source band according to Equation 5.

In Equation 5, f (x, y) is a target band, and g (x, y) is a color value of an optimal source band. The initial h (x, y) of the target area is set to one.

In the second step, h (x, y) is updated by repeatedly applying Laplacian of h (x, y).

는 라플라시안을 나타내는 것이다. 여기서, 2차 미분을 라플라시안이라 한다. 이는 밴드영역의 값을 채워넣은 인페인팅 영역으로 전파하기 위한 기능을 합니다. In Equation 6, n represents a repetition number,

Represents laplacian. Here, the second derivative is called Laplacian. This serves to propagate to the inpainting area filled with the value of the band area.

값과 최적의 소스밴드 내부의 화소값을 곱하여 갱신하며, 이는 수학식 7에 따른다. Thereafter, the processor 100 updates the last pixel value of the inpainting target region by repetition.

The value is multiplied and updated by the pixel value inside the optimal source band, which is according to Equation (7).

When the boundary removal is completed through the above process, the processor 100 separates the stitched image data into a plurality of image data according to the homography inverse (step 214).

Here, the inpainting process according to the preferred embodiment of the present invention will be briefly described with reference to FIG.

FIG. 8 (a) shows a plurality of continuous image data, and FIG. 8 (b) shows a result of stitching the plurality of continuous image data.

As shown in FIG. 8 (c), when the inpainting target region is defined for the region marked with white in the stitched image data, FIG. 8 (d) shows the inpainted image data.

FIG. 8E illustrates the separation of the inpainted image data into a plurality of image data.

As described above, the present invention stitches a plurality of continuous image data into one, and then divides the plurality of image data into a plurality of image data, thereby making the inpainting result for each of the plurality of continuous image data consistent.

Second Embodiment

A second preferred embodiment of the present invention calculates a homography matrix by manually selecting a feature point for projection data having a lot of noise, such as an old film, and having poor image quality, which will be described with reference to FIG. 9.

When the processor 100 of the inpainting processing apparatus receives a feature point directly through the user interface 102 and requests an operation in the second painting mode in which inpainting is performed (step 300), the processor 100 of the inpainting processing apparatus receives the feature point. In step 302 and 304, a homogeneous matrix is detected for the feature points of the continuous image data.

Thereafter, the processor 100 stitches consecutive image data into one according to the homography matrix, and receives an inpainting target region with respect to the stitched image data (steps 306 and 308).

Thereafter, the processor 100 performs band matching on the inpainted target area on the stitched image data, and removes the boundary of the edge of the band-matched inpainted target area (steps 310 and 312).

The inpainted stitched image data according to the band matching and boundary elimination is separated into stitched image data according to the homography inverse (step 314).

Third Embodiment

A third preferred embodiment of the present invention will be described in detail with reference to FIG. 10 by automatically tracking and inpainting moving objects when the camera is fixed and the photographing is performed.

The processor 100 of the inpainting processing apparatus tracks the inpainting target region with respect to successive image data when an operation for a third painting mode for automatically tracking the inpainting target region is requested. Here, the inpainting target area is set as an inpainting target area by tracking a moving area based on a background when an in-flight object or person is to be painted while the camera is fixed.

Thereafter, the processor 100 extracts the image data for the inpainting target region from the image data not including the inpainting target region and fills the inpainting target region of the image data including the inpainting target region, Inpainting is performed through band matching and boundary removal on the inpainting target region (step 404).

FIG. 11 illustrates an inpainting result according to a third exemplary embodiment of the present invention, (a) shows an original image, (b) shows tracking and extracting an inpainting target area, and (c) The image data of the inpainting target region is extracted from the image data not including the inpainting target region and filled into the inpainting target region of the image data including the inpainting target region, and (d) is the inpainting target. It illustrates the implementation of inpainting through band matching and boundary elimination for the region.

Fourth Embodiment

A fourth preferred embodiment of the present invention will be described with reference to FIG. 12 by automatically tracking and inpainting an object changing at a different speed from image data changing at a constant speed.

When the processor 100 of the inpainting processing apparatus is requested to operate in the fourth painting mode for inpainting an object moving at a different speed from a background moving at a constant speed, the processor 100 of the inpainting processing apparatus changes from another image data changing at a constant speed to another speed. The inpainting target area is tracked (steps 500 and 502).

Then, the processor 100 detects a feature point between consecutive image data, detects a homography matrix for the feature points of the continuous image data, and converts the continuous image data into one image data according to the homography matrix. Stitch (steps 504-508).

In operation 512, band matching on the inpainting target region is performed on the image data stitched with the single image data, and border elimination is performed on an edge of the band-matched inpainting target region.

In operation 514, the processor 100 separates the stitched image data according to the homography inverse matrix.

In the above-described preferred embodiment of the present invention, only the band matching and boundary elimination are performed for the inpainting target region, but the inpainting target region is divided into a plurality of segments to improve the inpainting result. Band matching and boundary elimination can also be implemented for the region of.

1 is a block diagram of an inpainting processing apparatus according to a preferred embodiment of the present invention.

2 is a flowchart of an inpainting processing method according to a first preferred embodiment of the present invention.

3 to 8 and 11 illustrate image data.

9 is a flowchart of an inpainting processing method according to a second preferred embodiment of the present invention.

10 is a flowchart of an inpainting processing method according to a third preferred embodiment of the present invention.

12 is a flowchart of an inpainting processing method according to a fourth preferred embodiment of the present invention.

Claims (22)

In the video data inpainting method, A first step of receiving a plurality of consecutive image data and stitching into one image data; Performing a band matching on the selected inpainting target area in the stitched image data; Removing a boundary of an edge of the inpainting target region with respect to the stitched image data in which the inpainting target region is band-matched; A fourth step of separating the stitched image data subjected to the boundary elimination of the edge into a plurality of consecutive image data; Image data inpainting method comprising a. The method of claim 1, The first step, Detecting feature points corresponding to each other between the image data; Detecting a homography matrix which is position change information between the feature points; Generating one image data by stitching the continuous image data based on the homography matrix, The fourth step, And stitching the stitched image data into a plurality of image data according to the homography inverse matrix. 3. The method of claim 2, Detection of the homography matrix, A first feature point is selected from the first image data with respect to continuous first and second image data among a plurality of image data, and a second feature point corresponding to each of the first feature points in the second image data. Select them, Calculating a homography matrix between the first feature points and the second feature points respectively corresponding thereto; Each of the second feature points calculates errors that are differences between a value obtained by applying a homography matrix to corresponding first feature points and its value, Detecting the number of errors that the calculated error value is less than or equal to a predetermined value is repeated a predetermined number of times, Acquiring the number of errors less than or equal to a predetermined value according to the repetition of the predetermined number of times, And determining the homography matrix from the first and second feature points, which yield errors corresponding to the largest number among the numbers, as the final homography matrix.  3. The method of claim 2, The feature point is detected according to the Kanade-Lucas-Tomasi Feature Tracker (KLT) method.  3. The method of claim 2, The feature point is input by the user, characterized in that the image data inpainting method. The method of claim 1, The inpainting target area is A video data inpainting method, wherein a background data moving at a constant speed is selected as an image data area moving at a speed different from that of the background. The method of claim 1, The inpainting target area is Image data inpainting method, characterized in that the selected area moving relative to the background. The method of claim 1, The inpainting target area is Image data inpainting method, characterized in that selected by the user. The method of claim 1, The third step, Setting a target band having a predetermined thickness at an edge of the selected inpainting target area; Setting a plurality of source bands having the same shape and thickness as the target band in the image data portion except for the target band; Calculating a difference between gradients between each of the plurality of source bands and the target band, selecting a source band having a minimum gradient difference as an optimal source band, and inserting color information in the source band into the inpainting target region step; Image data inpainting method comprising a. 10. The method of claim 9, The fourth step, The final pixel value of the inpainting target area; And multiplying the ratio of pixel values between the target band and the optimal source band by the pixel value inside the optimal band. In the image data inpainting processing apparatus, A memory unit for storing a plurality of consecutive image data; Stitching the plurality of consecutive image data into one image data, Performing band matching on the selected inpainting target region from the stitched image data, Remove a boundary of an edge of the inpainting target region with respect to the stitched image data in which the inpainting target region is band-matched, A processor for separating the stitched image data subjected to the boundary elimination of the edge into a plurality of consecutive image data; Image data inpainting processing apparatus comprising a. The method of claim 11, The processor, Stitching for the plurality of image data, Detect feature points corresponding to each other between the image data, Detecting a homography matrix which is position change information between the feature points; Transition by generating one image data by stitching the continuous image data based on the homography matrix, Separation of the stitched image data, And the stitched image data is divided into a plurality of image data according to the homography inverse matrix. The method of claim 12, Detection of the homography matrix, A first feature point is selected from the first image data with respect to continuous first and second image data among a plurality of image data, and a second feature point corresponding to each of the first feature points in the second image data. Select them, Calculating a homography matrix between the first feature points and the second feature points respectively corresponding thereto; Each of the second feature points calculates errors that are differences between a value obtained by applying a homography matrix to corresponding first feature points and its value, Detecting the number of errors that the calculated error value is less than or equal to a predetermined value is repeated a predetermined number of times, Acquiring the number of errors less than or equal to a predetermined value according to the repetition of the predetermined number of times, And determining the homography matrix from the first and second feature points as the final homography matrix, which calculates the errors corresponding to the largest number among the numbers.  The method of claim 12, And the feature point is detected according to the Kanade-Lucas-Tomasi Feature Tracker (KLT) method.  The method of claim 12, And the feature point is input by a user. The method of claim 11, The inpainting target area is And an image data area moving at a different speed from the background with respect to a background moving at a constant speed. The method of claim 11, The inpainting target area is The image data inpainting processing apparatus, wherein the image data is selected as a moving region based on the background. The method of claim 11, The inpainting target area is Image data inpainting processing apparatus, characterized in that selected by the user. The method of claim 11, The band matching is, Set a target band with a certain thickness on the edge of the selected inpainting target area, Set a plurality of source bands having the same shape and thickness as the target band in the image data portion except for the target band, Calculating a difference between gradients between each of the plurality of source bands and the target band, selecting a source band having a minimum gradient difference as an optimal source band, and inserting color information in the source band into the inpainting target region Image data inpainting processing apparatus, characterized in that implemented. The method of claim 19, The boundary removal is The final pixel value of the inpainting target area; And performing a determination by multiplying a ratio of pixel values between the target band and the optimal source band by a pixel value within the optimal band. In the video data inpainting method, A first step of receiving a plurality of consecutive image data and stitching into one image data; A second step of selecting an inpainting target region from the stitched image data and dividing the region into a plurality; Performing band matching on each of the plurality of divided inpainting target regions, and removing a boundary of an edge of the band-matched regions; A fourth step of separating the stitched image data subjected to the boundary elimination of the edge into a plurality of consecutive image data; Image data inpainting method comprising a. In the image data inpainting processing apparatus, A memory unit for storing a plurality of consecutive image data; Stitching the plurality of consecutive image data into one image data, Dividing the selected inpainting target area from the stitched image data into a plurality; Performing band matching on the divided inpainting target region, removing a boundary of an edge of the band matching regions, A processor for separating the stitched image data subjected to the boundary elimination of the edge into a plurality of consecutive image data; Image data inpainting processing apparatus comprising a.

Images