CN102999887B - Sample based image repairing method - Google Patents

Abstract

The invention relates to a sample based image repairing method. The sample based image repairing method includes steps of 1), calculating priority of any one pixel point p on a boundary of an area to be repaired; 2), determining a target block to be repaired firstly according to the calculated priority of the pixel point p in the step 1); 3) searching all matched blocks in the known area, and finding the optimally matched block of the target blocks to be repaired; 4) extracting pixel value of the optimally matched block, calculating confidence coefficient value of the central pixel point of the optimally matched block; 5) copying the pixel value corresponding to the optimally matched block to the corresponding position of the target block to be repaired, updating the confidence coefficient of the point p into confidence coefficient value of the central pixel point of the optimally matched block and forming a new area to be repaired; 6) executing the steps 1) to 5) until the area to be repaired is completely filled. The sample based image repairing method can be widely applied to image repairing process.

Description

A kind of image repair method based on sample

Technical field

The present invention relates to a kind of image repair and texture synthesis method, particularly about a kind of image repair method based on sample being applicable to repair Incomplete image.

Background technology

Along with continuous progress and the development of science and technology, universal and the people of computer network are to the increase of digital image information demand, image repair technology is subject to people and pays close attention to more and more, in the information recovering to lose in old photo, remove in the problem such as video text and hiding video error and be used widely, therefore study image repair method and there is great theory value and using value.Image repair method mainly in conjunction with the unknown images information of the information recovery area to be repaired around area to be repaired in degraded image, thus reaches the visual psychology requirement of people.It is preferential consistent with texture etc. that the reparation rule of visual psychology is mainly structural, similarity, structure.Image repair method mainly comprises two large classes: a class is the image repair of small scale defect (structural information), main thought is the mathematical model set up about degraded image and true picture, reparation problem is converted into a functional and asks the variational problem of extreme value to solve; Another kind of is the image repair of inner bulk drop-out (texture information) of blank map picture, main thought is the image information utilized around area to be repaired, method based on Block-matching fills area to be repaired, and these class methods mainly utilize picture breakdown technology and Future Opportunities of Texture Synthesis to realize image repair.Complete image repair needs to take into account structure and texture repairing simultaneously, could keep the unitarity of the continuity of image structure information, the consistance of texture information and structure repair and texture repairing.But in existing image repair process, above-mentioned two class methods all can not realize synchronous reparation to the structural information in image defect area and texture information.

In prior art, completed the filling of area to be repaired by the blocks and optimal matching blocks of searching for target to be repaired in known region based on the image repair method of sample, the method first fills priority by the block calculating area to be repaired, determine the multiblock to be repaired border, area to be repaired with highest priority, then in sample district exhaustive search blocks and optimal matching blocks, finally the pixel value of blocks and optimal matching blocks correspondence position is copied to the correspondence position of multiblock to be repaired, because the method considers unitarity and the constructional rationality of repairing model of structure repair and texture repairing, the method is removed in scratch reparation and object and obtains all well and good result, but there is a general problem in said method, namely damaged area information needed can not be generated intelligently.In other words, when information required for damaged area cannot find in image Given information, so damage zone just can not get gratifying reparation result.

Summary of the invention

For the problems referred to above, the object of this invention is to provide and a kind ofly can generate damaged area information needed intelligently and the high image repair method based on sample of precision repaired by picture.

For achieving the above object, the present invention takes following technical scheme: a kind of image repair method based on sample, comprise the following steps: the right of priority of arbitrary pixel p on the border, area to be repaired 1) calculating image to be repaired, comprise the following steps: 1. image to be repaired is divided into known region Φ and area to be repaired Ω two parts, δ Ω is the border of area to be repaired Ω; 2. the confidence value R of arbitrary pixel p on the δ Ω of border, area to be repaired is calculated _c(p):

R _C(p)=(1-ω)C(p)+ω

$C\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈{\mathrm{\Ψ}}_p\∩\mathrm{\Φ}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_p\right|}$

In formula, ω is for regulating operator, Ψ _pfor multiblock to be repaired, q is Ψ _pwith arbitrary pixel of Φ intersecting area; 3. the data item D (p) of pixel p is calculated; 4. right of priority P (the p)=R of pixel p is calculated _c(p) D (p); 2) calculate the right of priority of pixel p according to described step 1), determine the object block to be repaired of repairing at first; 3) search for all match block in known region, according to the matching condition of setting, find the best matching blocks of object block to be repaired; 4) extract the pixel value of best matching blocks, and calculate the confidence value of best matching blocks central pixel point; 5) pixel value corresponding for best matching blocks is copied to the relevant position of object block to be repaired, and the degree of confidence of p point is updated to the confidence value of best matching blocks central pixel point, form new area to be repaired; 6) above-mentioned steps 1 is performed) ~ 5), until area to be repaired is all filled complete.

In described step 1), the span of ω is [0.1,0.7].

3. described step 1) calculates the data item D (p) of pixel p:

$D\left(p\right)=\frac{|{\▿I}_p^{\⊥}\·n_p|}{\mathrm{\α}}$

In formula, represent the isophote vector at pixel p place, n _prepresent the unit normal vector at some p place, α represents normalized factor.

All match block in described step 3) search known region, according to the matching condition of setting, find the best matching blocks of object block to be repaired, comprise the following steps: the average SSD distance 1. calculating each match block in object block to be repaired and known region, if the minimum value of the average SSD distance calculated is unique, using this blocks and optimal matching blocks as object block Ψ to be repaired _pbest matching blocks, if the minimum value of the average SSD distance calculated is not unique, then enter step 2) mate again; 2. object block Ψ to be repaired is calculated _pthe NCC distance of the simplification fast with each Optimum Matching, the blocks and optimal matching blocks of the NCC distance of the simplification that chosen distance is minimum is as object block Ψ to be repaired _pbest matching blocks.

Described object block Ψ to be repaired _pwith the computing formula of the average SSD distance of arbitrary match block be:

${\stackrel{\‾}{d}}_{\mathrm{SSD}}({\mathrm{\Ψ}}_p,{\mathrm{\Ψ}}_{\mathrm{qi}})=\mathrm{\Σ}[{({\stackrel{\‾}{R}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{R}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2+{({\stackrel{\‾}{G}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{G}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2+{({\stackrel{\‾}{B}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{B}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2]$

In formula, represent the average of different color channels brightness in each pixel in object block to be repaired and match block respectively, the described match block in this formula is the rectangular block put centered by a certain pixel qi in known region.

The NCC distance computing formula of described simplification is:

$d_{\mathrm{NCC}}({\mathrm{\Ψ}}_p,{\mathrm{\Ψ}}_{\mathrm{qi}})=\frac{{[{\mathrm{\ΣG}}_{{\mathrm{\Ψ}}_P}\·G_{{\mathrm{\Ψ}}_{\mathrm{qi}}}]}^2}{\mathrm{\Σ}{\left[G_{{\mathrm{\Ψ}}_P}\right]}^2\·\mathrm{\Σ}{\left[G_{{\mathrm{\Ψ}}_{\mathrm{qi}}}\right]}^2}$

In formula, G represents the gray-scale value of each pixel in image.

The present invention is owing to taking above technical scheme, it has the following advantages: 1, operator is regulated because the present invention introduces when calculating the degree of confidence of pixel, by periodically calculating the difference between confidence value minimum in the confidence value of current iteration and previous ones, if this difference is greater than the threshold value of setting in advance, illustrate that " dropping effect " occurs confidence value curve, then can reselect the value regulating operator, thus the priority valve calculated can be enable to represent correct block fill order, make in limited iterations, reach repairing effect more accurately, compared with prior art, damaged area information needed can not only be generated intelligently, and effectively avoid the effect of " dropping effect ", make the more realistic reparation order of the right of priority of the pixel calculated, ensure that the accuracy that picture is repaired.2, have employed two kinds of matching conditions when the present invention finds the fast match block of target to be repaired in known region, first average SSD distance is adopted to find the blocks and optimal matching blocks of object block to be repaired as first fit condition, the blocks and optimal matching blocks of multiple object block to be repaired may be there is as matching condition owing to adopting minimum average SSD distance, now adopt the NCC distance of simplification as this matching condition, in blocks and optimal matching blocks, find best matching blocks, further ensure precision and the accuracy of image repair.3, the present invention is owing to setting sizeable to be matched piece of region, fast and effeciently can generate and fill the information needed for damaged area.The present invention can be widely used in image repair process.

Accompanying drawing explanation

Fig. 1 is image repair principle schematic of the present invention;

Fig. 2 is the reparation result schematic diagram of Fig. 1;

Fig. 3 is the schematic flow sheet of image repair method of the present invention;

Fig. 4 (a) is embodiments of the invention image schematic diagram;

Fig. 4 (b) is the schematic diagram of arbitrary pixel p on the δ Ω of embodiments of the invention restoring area border;

Fig. 5 is the new area to be repaired that the object block reparation to be repaired of the embodiment of the present invention completes formation;

Fig. 6 adopts repairing effect schematic diagram of the present invention, and Fig. 6 (a) is preprosthetic pictorial diagram; Fig. 6 (b) is the effect schematic diagram after repairing;

Fig. 7 (a) is the degree of confidence curve synoptic diagram of prior art, and horizontal ordinate is iterations, and ordinate is degree of confidence;

Fig. 7 (b) is the degree of confidence curve synoptic diagram adopting method of the present invention to calculate, and horizontal ordinate is iterations, and ordinate is degree of confidence;

Fig. 8 (a) is that the block of prior art fills right of priority schematic diagram, and horizontal ordinate is iterations, and ordinate is that block fills right of priority;

Fig. 8 (b) is the filling right of priority schematic diagram adopting method of the present invention to calculate, and horizontal ordinate is iterations, and ordinate is that block fills right of priority.

Embodiment

Below in conjunction with drawings and Examples, the present invention is described in detail.

Object removal is a typical apply in figure recovery technique field, its task is the loss graphical information of as far as possible naturally repairing appointed area in image, the present invention is that the detailed process of embodiment to image repair method is described with object removal, object removal is for the information defect area in image, the structure of other known region in image or texture information is utilized to fill, make the nature of the image after reparation, truly, meet the visual psychology requirement of people.

As shown in Figure 1, I is image to be repaired, and Ω is the defect area (area to be repaired) of specifying, and Φ is the region that in image, information is intact.As shown in Figure 2, from the angle of object removal, the target of image repair utilizes the intact region Φ of information repeatedly to repair defect area Ω exactly, thus image I to be repaired.

As shown in Figure 3, the image repair method based on sample of the present invention, comprises the following steps:

1, the right of priority (area to be repaired of image to be repaired can reduce gradually in the process of constantly repairing) of arbitrary pixel p on border, the area to be repaired δ Ω calculating image to be repaired, comprises the following steps:

1) as shown in Figure 4 (a), image to be repaired is divided into known region Φ and area to be repaired Ω two parts, δ Ω represents the border of area to be repaired Ω;

2) the degree of confidence R of arbitrary pixel p on the δ Ω of border, area to be repaired as shown in Figure 4 (b), is calculated _c(p):

R _C(p)=(1-ω)C(p)+ω （1）

$C\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈{\mathrm{\Ψ}}_p\∩\mathrm{\Φ}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_p\right|}---\left(2\right)$

In formula, ω is for regulating operator, for controlling the smooth degree of degree of confidence, ω can according to actual needs [0.1,0.7] select with the change controlling confidence value flexibly in, by periodically calculating the difference between confidence value minimum in the confidence value of current iteration and previous ones, if this difference is greater than the threshold value set in advance, (threshold value of the embodiment of the present invention is set to 0.02, can select according to actual needs, in this no limit), then reselect the ω value regulating operator, Ψ _pbe centered by p point, size is the region of n × n, and n can choose according to actual needs, in this no limit, and the embodiment of the present invention chooses n=9, then Ψ _pthe size of multiblock to be repaired is the rectangular block of 9 pixel × 9 pixels, and q is Ψ _pwith arbitrary pixel of Φ intersecting area, when initialization, $C\left(p\right)=0(\∀p\∈\mathrm{\Ω}),C\left(s\right)=1$ $(\∀s\∈\mathrm{\Φ}).$

3) the data item D (p) of pixel p is calculated:

$D\left(p\right)=\frac{|{\▿I}_p^{\⊥}\·n_p|}{\mathrm{\α}}---\left(3\right)$

In formula, D (p), for weighing the edge strength at pixel p place, represents the isophote direction and intensity at p point place, represent the isophote vector at pixel p place, n _prepresent the unit normal vector at some p place, α represents normalized factor.

4) according to the degree of confidence R of pixel p _cp the product of () and data item D (p) calculates the right of priority P (p) of pixel p:

P(p)=R _C(p)D(p) (4)

2, calculate the right of priority of pixel p on the δ Ω of border, area to be repaired according to above-mentioned steps 1, determine the object block to be repaired of repairing at first.

For area to be repaired borderline arbitrary pixel p, R _cp the pixel around () larger pixels illustrated point p with high confidence level is more, the degree of confidence of some p is also higher, should preferentially repair, D (p) value larger pixels illustrated point p place is the isophote of image known region and the intersection on border, area to be repaired, in order to keep the marginal texture in image, Ψ _pshould preferentially repair, therefore the right of priority P (p) of pixel p is larger, and representative needs to repair in advance.

3, search for all match block in known region, according to the matching condition of setting, find the best matching blocks of object block to be repaired, comprise the following steps:

1) each match block (Ψ in object block to be repaired and known region is calculated _q1Ψ _q2Ψ _qm) average SSD distance (Sum of Squared Differences, the sum of squares of deviations), using this average SSD distance as measuring object block Ψ to be repaired _pand the similarity between each match block, select with the average SSD of object block to be repaired apart from minimum match block as with object block Ψ to be repaired _poptimum Matching fast, if the minimum value of the average SSD distance calculated is unique, using this blocks and optimal matching blocks as object block Ψ to be repaired _pbest matching blocks, if the minimum value of the average SSD distance calculated is not unique, namely there is more than one blocks and optimal matching blocks, then enter step 2) again mate.

Wherein, average SSD distance adopts the average of each Color Channel brightness to reflect average similarity between object block to be repaired and match block, object block Ψ to be repaired _pwith arbitrary match block Ψ _qi(i=1 ... m) average SSD distance is:

${\stackrel{\‾}{d}}_{\mathrm{SSD}}({\mathrm{\Ψ}}_p,{\mathrm{\Ψ}}_{\mathrm{qi}})=\mathrm{\Σ}[{({\stackrel{\‾}{R}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{R}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2+{({\stackrel{\‾}{G}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{G}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2+{({\stackrel{\‾}{B}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{B}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2]---\left(5\right)$

In formula, represent the average of different color channels brightness in each pixel in object block to be repaired and match block respectively, described match block in this formula be in known region centered by a certain pixel qi point, size is the rectangular block of 9 pixel × 9 pixels, can calculate object block Ψ to be repaired respectively according to formula (5) _pwith the average SSD distance of other arbitrary match block.

2) object block Ψ to be repaired is calculated _pnCC (Normalized CrossCorrelation) distance of the simplification fast with each Optimum Matching, to select with the NCC of the simplification of object block to be repaired apart from minimum blocks and optimal matching blocks as object block Ψ to be repaired _pbest matching blocks.

NCC is a kind of image matching method based on pixel grey scale, and the method anti-white noise disturbance ability is strong, and when grey scale change and geometric distortion little precision very high, but computation complexity is higher, NCC distance utilizes object block Ψ to be repaired in image _pand the correlativity between blocks and optimal matching blocks half-tone information measures similarity degree each other.The present invention adopts the reduced form of NCC, eliminates to subtract equal value part in traditional NCC method, and reduce computation complexity, the NCC distance of simplification is:

$d_{\mathrm{NCC}}({\mathrm{\Ψ}}_p,{\mathrm{\Ψ}}_{\mathrm{qi}})=\frac{{[{\mathrm{\ΣG}}_{{\mathrm{\Ψ}}_P}\·G_{{\mathrm{\Ψ}}_{\mathrm{qi}}}]}^2}{\mathrm{\Σ}{\left[G_{{\mathrm{\Ψ}}_P}\right]}^2\·\mathrm{\Σ}{\left[G_{{\mathrm{\Ψ}}_{\mathrm{qi}}}\right]}^2}---\left(6\right)$

In formula, G represents the gray-scale value of each pixel in image, Ψ _qifor the plural blocks and optimal matching blocks calculated in step 5), object block Ψ to be repaired can be calculated respectively according to formula (6) _pwith the NCC distance of the simplification of each blocks and optimal matching blocks.

4, extract the pixel value of best matching blocks, and calculate the confidence value of best matching blocks central pixel point.

5, as shown in Figure 5, pixel value corresponding for best matching blocks is copied to object block Ψ to be repaired _prelevant position, and the degree of confidence at p point place is updated to the confidence value of best matching blocks central pixel point, now forms new area to be repaired.

6, above-mentioned steps 1 ~ 5 is performed, until area to be repaired is all filled complete (being depicted as the effect schematic diagram repairing picture as Fig. 6 (a) with as schemed (b)).

As shown in Fig. 7 ~ 8, in sum, through process of the present invention, can find out, " dropping effect " effect is made to obtain effective suppression through process of the present invention, because along with the carrying out of iteration, by periodically calculating the difference between confidence value minimum in the confidence value of current iteration and previous ones, if this difference is greater than the threshold value of setting in advance, illustrate that " dropping effect " occurs confidence value curve, then can reselect the value regulating operator ω, thus the priority valve calculated can be enable to represent correct block fill order, make in limited iterations, reach repairing effect more accurately.

The various embodiments described above are only for illustration of the present invention, and wherein each step of method etc. all can change to some extent, and every equivalents of carrying out on the basis of technical solution of the present invention and improvement, all should not get rid of outside protection scope of the present invention.

Claims (3)

1., based on an image repair method for sample, comprise the following steps:

1) right of priority of arbitrary pixel p on the border, area to be repaired calculating image to be repaired, comprises the following steps:

1. image to be repaired is divided into known region Φ and area to be repaired Ω two parts, δ Ω is the border of area to be repaired Ω;

2. the confidence value R of arbitrary pixel p on the δ Ω of border, area to be repaired is calculated _c(p):

R _C(p)＝(1-ω)C(p)+ω

$C\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈{\mathrm{\Ψ}}_p\∩\mathrm{\Φ}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_p\right|}$

In formula, ω is for regulating operator, Ψ _pfor multiblock to be repaired, q is Ψ _pwith arbitrary pixel of Φ intersecting area;

3. the data item D (p) of pixel p is calculated;

4. right of priority P (the p)=R of pixel p is calculated _c(p) D (p);

2) according to described step 1) calculate the right of priority of pixel p, determine the object block to be repaired of repairing at first;

3) search for all match block in known region, according to the matching condition of setting, find the best matching blocks of object block to be repaired, comprise the following steps:

A) calculate the average sum of squares of deviations distance of each match block in object block to be repaired and known region, if the minimum value of the average sum of squares of deviations distance calculated is unique, namely there is a blocks and optimal matching blocks, using this blocks and optimal matching blocks as object block Ψ to be repaired _pbest matching blocks, if the minimum value of the average sum of squares of deviations distance calculated is not unique, namely there is plural blocks and optimal matching blocks, then enter step B) mate again;

B) object block Ψ to be repaired is calculated _pwith the normalized crosscorrelation distance of the simplification of each blocks and optimal matching blocks, the blocks and optimal matching blocks of the normalized crosscorrelation distance of the simplification that chosen distance is minimum is as object block Ψ to be repaired _pbest matching blocks;

Described object block Ψ to be repaired _pwith arbitrary match block Ψ _qithe computing formula of average sum of squares of deviations distance be:

${\stackrel{\‾}{d}}_{\mathrm{SSD}}({\mathrm{\Ψ}}_p,{\mathrm{\Ψ}}_{\mathrm{qi}})=\mathrm{\Σ}[{({\stackrel{\‾}{R}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{R}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2+{({\stackrel{\‾}{G}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{G}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2+{({\stackrel{\‾}{B}}_{{\mathrm{\Ψ}}_p}-{\stackrel{\‾}{B}}_{{\mathrm{\Ψ}}_{\mathrm{qi}}})}^2]$

In formula, represent object block Ψ to be repaired respectively _pwith arbitrary match block Ψ _qithe average of different color channels brightness in each pixel, the described match block in this formula is the rectangular block put centered by a certain pixel qi in known region;

The normalized crosscorrelation distance computing formula of described simplification is:

$d_{\mathrm{NCC}}({\mathrm{\Ψ}}_p,{\mathrm{\Ψ}}_{\mathrm{qi}})=\frac{{[{\mathrm{\ΣG}}_{{\mathrm{\Ψ}}_P}.G_{{\mathrm{\Ψ}}_{\mathrm{qi}}}]}^2}{\mathrm{\Σ}{\left[G_{{\mathrm{\Ψ}}_P}\right]}^2.\mathrm{\Σ}{\left[G_{{\mathrm{\Ψ}}_{\mathrm{qi}}}\right]}^2}$

In formula, G represents the gray-scale value of each pixel in image;

4) extract the pixel value of best matching blocks, and calculate the confidence value of best matching blocks central pixel point;

5) pixel value corresponding for best matching blocks is copied to the relevant position of object block to be repaired, and the degree of confidence of p point is updated to the confidence value of best matching blocks central pixel point, form new area to be repaired;

6) above-mentioned steps 1 is performed) ~ 5), until area to be repaired is all filled complete.

2. a kind of image repair method based on sample as claimed in claim 1, is characterized in that: described step 1) in the span of ω be [0.1,0.7].

3. a kind of image repair method based on sample as claimed in claim 1 or 2, is characterized in that: described step 1) 3. calculate the data item D (p) of pixel p:

$D\left(p\right)=\frac{|{\▿I}_p^{\⊥}.n_p|}{\mathrm{\α}}$

In formula, represent the isophote vector at pixel p place, n _prepresent the unit normal vector at some p place, α represents normalized factor.