KR101693643B1 - Apparatus and method for inpainting image - Google Patents

Abstract

The present invention relates to an apparatus and a method for restoring an image. The image restoration apparatus according to the present invention includes: a patch priority detection unit configured to detect a priority of a target patch having a predetermined pixel region including a target region which is set by a target region setting unit and a source region in the form of an original image; a comparison patch selection unit for selecting a candidate patch to be compared with similar images by using an average value and a standard deviation of known pixels of each target patch detected by the patch priority detection unit and an average value and a standard deviation calculated by a candidate patch calculation unit; a similar image patch detection unit for detecting a most similar candidate patch by stopping calculation and setting a patch difference value with respect to a next similar image candidate patch when the patch difference value between the known pixels of each target patch detected by the patch priority detection unit and the corresponding pixels of the similar image comparison candidate patch selected by the comparison patch selection unit is greater than the minimum patch difference value, and resetting the minimum patch difference value when the patch difference value is less than the previous patch difference value; and a patch restoration processing unit for restoring the patch by copying corresponding pixels of the candidate patch, which is detected by the similar image patch detection unit, to an unknown area of the target patch. The image can be restored by detecting the most similar candidate patch at a high speed.

Description

[0001] APPARATUS AND METHOD FOR INPAINTING IMAGE [0002]

The present invention relates to an image restoration apparatus and method, and more particularly, to an image restoration apparatus and method for restoring an image by rapidly detecting a candidate patch that is the most similar by reducing a candidate patch of a comparison object in an original image restoration method.

Image restoration technology is a technology related to filling in the unknown area of an image, which removes scratches and stains, restores the parts of the image that have been removed, and removes unwanted objects and replaces them with a plausible looking background ≪ / RTI >

This image restoration method largely restores an image based on PDE (Partial Difference Equation) or based on an original (Exemplar). The PDE-based image reconstruction method fills an unknown area by spreading neighboring known pixels. In the case of filling small gaps such as scratches, it is excellent, but it is not possible to obtain a good result when the areas where the image is removed are wide.

On the other hand, Criminisi et al. Proposed an image-based image restoration method in Non-Patent Document 1. This image restoration method is a method of filling the unknown region of the image from the base image. First, the unknown region including the boundary and the base region are selected as the target patches, and similarity judgment is performed for all the patches of the same size from the base region. To fill the unknown area repeatedly. However, this image restoration method has a problem that the processing time is too long in a small device such as a smart phone because the throughput related to image restoration is too large.

Accordingly, the present applicant filed a patent application No. 1539013 for an image restoration apparatus and method for recovering an image by detecting the most similar candidate patch at high speed. However, faster restoration without degradation of image quality is still required.

Patent Document 1: Korea Patent No. 1539013

Non-Patent Document 1: IEEE TRANSACTIONS ON IMAGE PROCESSING VOL.13 NO.9, SEP 2004 (Region Filling and Object Removal by Exemplar-Based Image Inpainting)

In order to solve the above-described problems, it is an object of the present invention to provide an image restoration apparatus and method for restoring an image by detecting a candidate patch of a comparative object at a high speed by reducing a candidate patch to be compared in an original- .

According to an aspect of the present invention, there is provided an image restoration apparatus including a target region setting unit configured to set a target region in an original image to be restored, A patch priority calculation unit for calculating a priority of a target patch having a predetermined pixel area including a target area set in the target area setting unit and a source area in the original image, A comparison patch selection unit for selecting a candidate patch to be compared with a similar image by using an average value and a standard deviation of known pixels of each target patch detected by the patch priority detection unit and an average value and a standard deviation calculated by the candidate patch calculation unit, And comparing the known pixels of each target patch detected by the patch priority detector with the known pixels If the patch difference value of the corresponding pixels of the similar image comparison candidate patch selected by the value selection unit is larger than the minimum patch difference value, the calculation is stopped and then the patch difference value is performed for the next similar image comparison candidate patch, A similar image patch detection section for detecting the most similar candidate patch by storing the same as the minimum patch difference value, and a corresponding pixel of the candidate patch detected by the similar image patch detection section in the unknown area of the target patch, And provides a patch restoration processing unit.

The comparison patch selection unit may calculate a lower bound using an average value and a standard deviation of known pixels of each target patch detected by the patch priority detection unit, an average value and a standard deviation calculated by the candidate patch calculation unit, If the bound is greater than the current minimum patch difference value, the candidate patch is skipped, and if it is smaller than the current minimum patch difference value, the similar image comparison candidate patch can be selected.

The similar image patch detector may calculate the patch difference value in the order of the pixels having the low frequency value among the known pixels of the target source by using the histogram for the pixels of the source area, have.

The similar image patch detection unit may calculate the patch difference value in the order of closest proximity to the target patch among the candidate patches in the source region.

According to another aspect of the present invention, there is provided an image restoration method including: setting an area requiring restoration in an original image as a target area; calculating an average value and a standard deviation of candidate patches of a source area as an original image; The method comprising the steps of: detecting a priority of a target patch having a predetermined pixel area including a target area set as an original image and a source area set as an original image; and detecting a priority of each target patch detected in the step of detecting the priority Selecting candidate patches to be compared with similar images using the average value and standard deviation of the pixels of the candidate patches and the average value and standard deviation calculated in the step of calculating the average and standard deviation of the candidate patches, Selecting the candidate patches from the known pixels of the detected target patches, If the patch difference value of the corresponding pixels of the image comparison candidate patch is larger than the minimum patch difference value, the calculation is stopped and then the patch difference value is performed for the next similar image comparison candidate patch. If the patch difference value is smaller than the previous patch difference value, Detecting the most similar candidate patch by storing the difference value as a difference value and detecting the most similar hatching patch in the unknown region of the target patch by providing the step of copying and restoring the corresponding pixels of the detected candidate patch, The above-described object can be achieved.

With the above-described configuration, the present invention can detect the most similar candidate patch at high speed and restore the image.

In addition, the present invention can alleviate the burden on the similar image patch detection unit which takes a long time.

In addition, the present invention can reduce the error in restoring the image by detecting the priority of the target patch.

1 is a block diagram of an image restoration apparatus according to an embodiment of the present invention.
2 is a diagram illustrating an image restoration process by the image restoration apparatus according to an embodiment of the present invention.
3 is a diagram illustrating test images used in an image restoration apparatus according to an embodiment of the present invention.
4 is a diagram showing the cumulative distribution function of the Manhattan distance between the centers of the candidate patches most similar to the center of the target patch.
5 is a flowchart illustrating an image restoration method according to another embodiment of the present invention.

Hereinafter, preferred embodiments of an image restoration apparatus and method according to the present invention will be described with reference to the accompanying drawings. In the following description of the present invention, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the technical scope of the present invention. Will be.

FIG. 1 is a block diagram of an image restoration apparatus according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a process of restoring an image by an image restoration apparatus according to an embodiment of the present invention. 3 is a view showing test images used in an image restoration apparatus according to an embodiment of the present invention, and FIG. 4 is a diagram showing a cumulative distribution function of Manhattan distance between centers of a candidate patch most similar to a center of a target patch .

1, the image restoration apparatus includes a target area setting unit 110, a candidate patch calculation unit 120, a patch priority detection unit 130, a comparison patch selection unit 140, a similar image patch detection unit 150 A patch restoration processing unit 160, and a restored image output unit 170.

The target area setting unit 110 sets an area required to be restored in the input original image as a target area. The target area may be set via a user interface (not shown). In FIG. 2A, a target region (?) Is displayed and an original image to be filled in the target region (?) Is indicated as a source region (?). And the boundary line of the target area? Is indicated by??.

The candidate patch calculation unit 120 calculates an average value and a standard deviation for each candidate patch of a source region that is an original image other than the target region set in the target region setting unit 110. [ The candidate patch calculation unit 120 can calculate an average value and a standard deviation of L (lightness), a (red-green region) and b (yellow-blue region) according to the CIE color system.

In order to store the average value and the standard deviation of the candidate patches calculated by the candidate patch calculation unit 120, four bytes (two bytes mean value and two bytes standard deviation) are required. Standard deviation can be stored.

The patch priority detector 130 determines the priority of the patch area to be restored first in the set target area. If the detection of the priority in the patch priority detector 130 is incorrect, a considerable error may occur in the restored image have. In Fig. 2 (b), the target patch (? P) to be filled from the known image of the source region is shown. The size of the target patch? P used in the present invention is 9 X 9 pixels.

That is, the patch priority detector 130 first detects a target patch? P to be restored, wherein the priority of the patch is defined as in Equation (1).

Where C (p) is the confidence term and D (p) is the data term, C (p) and D (p) Is defined as Equation (2).

는 픽셀(p)에서의 등광도선(isophote)이다.Where? P is the area of? P, and? Is the normalization factor (for example,? = 255 for the gray level image). n _p is a unit vector orthogonal to the boundary line (delta OMEGA) at p,

Is an isophote in the pixel p.

The details of the patch priority detector 130 are described in Non-Patent Document 1, and Non-Patent Document 1 is incorporated by reference into the present invention for understanding of the patch priority detector 130. [

The comparison patch selection unit 140 selects the similarity image and the standard deviation using the average value and the standard deviation of the known pixels of each target patch detected by the patch priority detection unit 130 and the average value and the standard deviation calculated by the candidate patch calculation unit 120, The candidate patch to be compared is selected.

The comparison patch selection unit 140 can obtain a lower bound on the difference between the target patch and the target patch by using the average value and the standard deviation of L, a, b calculated by the candidate patch calculation unit 120. [ If the subbound exceeds a predetermined value, e. G. A minimum patch difference value, it is removed from the similar image comparison candidate patch.

는 타겟 패치의 기지의 픽셀들에 대응하는 후보 패치의 픽셀들의 세트이다. 이 경우

의 평균값과 표준편차는 <수학식 3>과 같은 범위를 가질 수 있다.Let d be the number of pixels in a patch, and let k be the number of known pixels for the target patch, then in a 9 x 9 patch, d is 81 and k is less than 81. X is a set of all pixels of the candidate patch,

Is a set of pixels of the candidate patch corresponding to known pixels of the target patch. in this case

The mean value and the standard deviation of the average value of the three-dimensional image data may have a range as shown in Equation (3).

및

는

의 평균 및 표준 편차이다.Where μ _c and σ _c are the mean and standard deviation of X,

And

The

&Lt; / RTI &gt;

The mathematical proof for this is described in the present inventor's article, which is incorporated herein by reference, and is incorporated herein by reference.

및

라고 하고, 후보 패치 계산부(120)에서 계산된 각 후보 패치의 모든 픽셀의 평균값 및 표준편차를 μ_q 및 σ_q 라고 하면, 평균값 및 표준편차의 각 최소 차이 값을 <수학식 4>에서 얻을 수 있다.Further, the average value and the standard deviation of the known pixels of the target patch

And

And an average value and a standard deviation of all the pixels of each candidate patch calculated by the candidate patch calculation unit 120 are denoted by μ _q and σ _q , The minimum difference value between the average value and the standard deviation can be obtained from Equation (4).

Then, the lower bound using the difference values may be calculated using Equation (5).

Here, c ∈ {L, a, b} means to calculate for each of L, a, and b.

When the minimum patch difference value is initially set in the comparison patch selection unit 140, a large number can be set. The comparison patch selection unit 140 calculates a lower bound using the average value and the standard deviation of all the pixels of each candidate patch calculated by the candidate patch calculation unit 120. [ If the subbound is greater than the current minimum patch difference value, the candidate patch is skipped, and if it is smaller than the current minimum patch difference value, it is selected as the similar image comparison patch candidate.

The similar image patch detecting unit 150 detects the most similar image patch to be filled in the target patch of the priority order detected by the patch priority detecting unit 130. Specifically, the similar image patch detector 150 calculates and adds a difference value of pixel values using SSD (Sum of Squared Difference) for each candidate patch q in the source region corresponding to the target patch and the target patch, And a candidate patch having a minimum patch difference value among these patch difference values is selected as the most similar patch. The candidate patch qq of the source region is shown in Fig. 2c.

The similar image patch detecting section 150 detects the pixel difference value delta _i using the pixel value of the pixel c ' _i of the candidate patch qq from the pixel value of each known pixel c _i of the target patch p This difference value is defined as Equation (6).

In this equation, c _{i, R} , c _{i, G} , c _{i, and B} are the R, G, and B values of the pixel (ci).

On the other hand, the patch difference value (Dp, q) between? P and? Q is defined as follows.

Equation (7) expresses that the pixel difference value? Should be calculated k times for each candidate patch? Q in the known region if the number of known pixels in the target patch? P is k, for example.

If the patch difference value of the candidate patch (? Q ') calculated using Equation (7) is smaller than the patch difference value of the other candidate patch (? Q' '), the similar image patch detection unit (150) It is judged that the candidate patch? Q 'is more suitable for the target patch? P.

On the other hand, the similar image patch detecting section 150 includes a bounding algorithm for detecting a similar image patch at a higher speed. Here, if the patch difference value of the currently executed candidate patch (qq) exceeds the patch difference value that is the smallest based on the minimum patch difference value among the patch difference values of the candidate patches (qq) calculated so far, The calculation is performed for the next candidate patch qq without proceeding abnormally and if the patch difference value of the candidate patch qq currently being executed does not exceed the smallest patch difference value, it is stored as the minimum patch difference value after calculation.

These bounding algorithms can be displayed as shown in Table 1.

Using this bounding algorithm, as shown in Table 1, the calculation process can be considerably reduced. In this bounding algorithm, the candidate patches are selected in a sequential order, and can start from candidate patches in the upper-left corner. However, it is desirable that the execution of the bounding algorithm includes a comparison order of subsequent pixels and patches.

The similar image patch detection unit 150 can perform a calculation process at a high speed while selecting a candidate patch having the smallest patch difference value and the known image of the target patch among the calculated patch difference values as the similar image patch by the bounding algorithm have.

In addition, the similar image patch detecting section 150 can detect the similar patch image at a higher speed by changing the pixel order to be calculated in calculating the patch difference value between the target patch and the candidate patch.

In general, it is not known which pixel of the target patch has a color difference with the corresponding pixel of the candidate patch. Therefore, the similar image patch detecting unit 150 uses the following statistical method to determine the pixel having the largest color difference in the candidate patch.

The similar image patch detecting section 150 may first use a gray-scale value (0 to 255) to generate a histogram for each pixel of the source region in which the image exists. Whereby the frequency can be obtained for each gray-scale value of the known image. Here, if the calculation is first performed on a pixel having a low frequency, a similar image patch can be detected more quickly by using the above bounding algorithm.

These pixel order algorithms can be expressed as shown in Table 2.

Where H (cm) is the number of pixels (I-OMEGA) with gray-scale hue equal to pixel (cm), and argmin H (cm) is the smallest frequency.

The gray-scale values (0 to 255) are used when counting the number of pixels for each color above. Alternative methods for selecting the pixels include (1) (2) counting the number of pixels individually for RGB, adding a count, and selecting pixels with a minimum sum, (3) counting the number of pixels individually for RGB, and A pixel having a minimum count regardless of RGB can be selected.

The similar image patch detecting section 150 can change the order of comparing candidate patches of the source region to detect an image patch similar to the target patch more quickly. That is, similar image patches can be detected more quickly if the minimum patch difference value can be detected more quickly through the calculation process.

To this end, the similar image patch detection section 150 may order the candidate patches near the target patch to be processed earlier than the candidate patches that are farther away.

Such a change in the patch order may take more time than limiting the search size to the neighborhood of the target area, but it may also reduce the possibility of detecting false images, as well as detect similar images faster by combining with the bounding algorithm.

This patch sequence algorithm can be expressed as in Table 3.

Basically, the candidate patches are ordered such that the patch closest to the target patch is processed first, where M (p, q) is the Manhattan distance of the pixel p and pixel q, do.

Where px and py are the x and y coordinates of the pixel p.

The cumulative distribution function (CDF) of the Manhattan distance between the center of the target patch and the center of the best candidate patch is shown in FIG. As can be seen from FIG. 3, it can be seen that in most cases it is desirable to first compare candidate patches closer to the target patch.

The patch restoration processing unit 160 fills the target patch using the most similar image patch detected by the similar image patch detection unit 150. [ The contents of restoring the target patch are shown in Fig. 2D.

The patch restoration processing unit 160 delivers the restored image to the restored image output unit 170 when the image restoration of the target area is completed. However, if the image restoration is not completed, the patch restoration processing unit 160 transmits the restored image to the patch priority detection unit 130, and continues the image restoration operation until the similar image is filled in the target area.

The restored image output unit 170 displays the restored final image on the display unit.

The images restored by the above-described configuration are shown in Fig. It can be seen that there is no difference between the reconstructed images. It can be seen from Table 4 that the processing speed according to the present invention is considerably improved as compared with the case where it is processed by non-Patent Document 1 (Criminisi) and Patent Document 1 (Method A + B).

5 is a flowchart illustrating an image restoration method according to another embodiment of the present invention.

The image restoration device receives an image to be restored (S502). The target area setting unit 110 sets an area required to be restored in the input original image as a target area (S504). The target area may be set via a user interface (not shown).

The candidate patch calculation unit 120 calculates an average value and a standard deviation for each candidate patch of the source region of the original image (S506). The patch priority detector 130 detects a target patch by obtaining a priority for a patch area to be restored first in the set target area (S508).

The comparison patch selection unit 140 selects the similarity image and the standard deviation using the average value and the standard deviation of the known pixels of each target patch detected by the patch priority detection unit 130 and the average value and the standard deviation calculated by the candidate patch calculation unit 120, A candidate patch to be compared is selected (S510). The comparison patch selection unit 140 can obtain a lower bound on the difference between the target patch and the target patch by using the average value and the standard deviation of L, a, b calculated by the candidate patch calculation unit 120. [

The similar image patch detecting section 150 can change the order of comparing the patches of the source region in order to detect the image patches similar to the target patch more quickly. That is, the similar image patch detecting unit 150 may order the patches near the target patch to be processed earlier in order than the patches farther away (S512).

In addition, the similar image patch detecting section 150 can detect the similar patch image at a higher speed by changing the pixel order to be calculated in calculating the patch difference value between the target patch and the candidate patch. The similar image patch detecting section 150 first uses a gray-scale value (0 to 255) to generate a histogram for each pixel of the source region in which the image exists. Whereby the frequency can be obtained for each gray-scale value of the known image. If a calculation is first performed on a pixel having a low frequency (S514), similar image patches can be detected more quickly using the above bounding algorithm.

In addition, the similar image patch detecting unit 150 detects a patch difference value of the candidate patch qq currently being executed based on the minimum patch difference value among the patch difference values of the candidate patches qq calculated so far If the difference value is exceeded (S516), the calculation is performed on the next candidate patch (qq), and if the patch difference value of the candidate patch (qq) currently being executed does not exceed the smallest patch difference value As a minimum patch difference value (S518).

If all the candidate patches for the target patch have been compared (S520), the patch restoration processing unit 160 fills the target patch using the most similar image patch detected by the similar image patch detecting unit 150 (S522).

The patch restoration processing unit 160 delivers the restored image to the restored image output unit 170 when the image restoration of the target area is completed (S524). However, if the image restoration process is not completed, the patch restoration process unit 160 transmits the restored image to the patch priority detection unit 130 and continuously performs the image restoration operation until the similar image is filled in the target area .

The restored image output unit 170 displays the restored final image on the display unit (S524).

The embodiments of the present invention described above are merely illustrative of the technical idea of the present invention, and the scope of protection of the present invention should be interpreted according to the claims. It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention as defined by the appended claims. It should be interpreted that it is included in the scope of right.

110: target area setting unit 120: candidate patch calculation unit
130: patch priority detection unit 140: comparison patch selection unit
150: Pseudo image patch detection unit 160: Patch restoration processing unit
170: restored image output unit

Claims (7)

A target area setting unit for setting an area requiring restoration in the original image as a target area,
A candidate patch calculation unit for calculating an average value and a standard deviation of candidate patches of the source region of the original image,
A patch priority detector for detecting a priority of a target patch having a target area set in the target area setting part and a predetermined pixel area including a source area in the original image;
A comparison patch selection unit for selecting a candidate patch to be compared with a similar image by using an average value and a standard deviation of known pixels of each target patch detected by the patch priority detection unit and an average value and a standard deviation calculated by the candidate patch calculation unit; ,
If the patch difference value of the known pixels of each target patch detected by the patch priority detector and the corresponding pixels of the similar image comparison candidate patch selected by the comparison patch selector is greater than the minimum patch difference value, A similar image patch detection section for detecting the most similar candidate patch by storing the patch difference value for the similar image comparison candidate patch and storing it as a minimum patch difference value if the patch difference value is smaller than the previous patch difference value;
And a patch recovery processing unit for copying and restoring corresponding pixels of the candidate patch detected by the similar image patch detecting unit in a unknown area of the target patch,
The comparison patch selection unit may calculate a lower bound using an average value and a standard deviation of known pixels of each target patch detected by the patch priority detection unit, an average value and a standard deviation calculated by the candidate patch calculation unit, If the bound is larger than the current minimum patch difference value, the candidate patch is skipped, and if it is smaller than the current minimum patch difference value, the similar image comparison candidate patch is selected. delete The method according to claim 1,
The similar image patch detection unit calculates a patch difference value in the order of pixels having a low scale value among the known pixels of the target source by obtaining the frequency of each scale value using the histogram for the pixels of the source region The image restoration device comprising: The method of claim 3,
Wherein the similar image patch detection unit calculates the patch difference value in the order of closest distance to the target patch among the candidate patches in the source region. Setting an area requiring restoration in the original image as a target area,
Calculating an average value and a standard deviation of candidate patches of the source region in the original image,
Detecting a priority of a target patch having a predetermined pixel area including a target area set in the target and a source area in the original image;
A candidate to be compared with the similar image using the average value and the standard deviation calculated in the step of calculating the average value and the standard deviation of the known pixels of each target patch detected in the step of detecting the priority and the average value and the standard deviation of the candidate patches Selecting a patch,
If the patch difference value of the known pixels of the target patch detected in the step of detecting the priority and the corresponding pixels of the similar image comparison candidate patch selected in the step of selecting the candidate patch is larger than the minimum patch difference value, Detecting the most similar candidate patch by performing a patch difference value for the next similar image comparison candidate patch and storing the difference value as a minimum patch difference value when the difference value is smaller than the previous patch difference value;
And copying and restoring the corresponding pixels of the candidate patch detected in the step of detecting the most similar candidate patch in the unknown region of the target patch,
Wherein the step of selecting the candidate patch comprises the steps of calculating an average value and a standard deviation of known pixels of each target patch detected in the step of detecting the priority and a mean value and standard deviation calculated in the step of calculating an average value and a standard deviation of the candidate patches, If the calculated lower bound is larger than the current minimum patch difference value, the candidate patch is skipped. If the calculated lower bound is smaller than the current minimum patch difference value, it is selected as the similar image comparison candidate patch . 6. The method of claim 5,
Wherein the step of detecting the most similar candidate patch comprises: obtaining a frequency of each scale value by using a histogram for pixels of the source region; calculating a difference between the number of known pixels of the target source and the number of pixels having a low- And calculating a value of the image. The method according to claim 5 or 6,
Wherein the step of detecting the most similar candidate patch calculates a patch difference value in the order of closest proximity to the target patch among the candidate patches in the source region.

Images