CN101847255B - Structural information synthesis-based image completion method - Google Patents

Abstract

The invention belongs to the technical field of image inpainting, and relates to a structural information synthesis-based image completion method, which comprises the following steps of: for an image, determining an inpainting border area; calculating the cost energy of nodes per se; calculating the correlation energy of the nodes; determining an integral target function; initializing and updating message values between every two adjacent nodes; and calculating the optimal solution of the nodes. The method has the advantages of achieving the effect of simultaneously reserving structural information and texture information without human intervention and greatly improving image inpainting effect.

Description

A kind of image completion method of compages information

Technical field

The invention belongs to the image repair technical field, relate to a kind of image completion method of compages information.

Background technology

The background technology that relates among the present invention has:

(1) TV image repair algorithm: the inpainting model based on the total variational method is by Chan﹠amp the earliest; Shen is called the TV model based on Variational Principle, and it is one of best model that keeps structural information based on partial differential equation, namely utilize diffusion equation in the physics with the Information Communication of known region in zone to be repaired.Breakage has reasonable effect to TV image repair algorithm to small scale, can effectively keep its structural information.If but the area to be repaired area more greatly then can produce significantly fuzzyly, not too meets people's visual sensory.

(2) based on the synthetic image completion algorithm of texture: the image completion algorithm that Criminisi has proposed patch-based texture synthesis solves the problem that removes the large tracts of land object from image.Utilize this algorithm, can fill the zone that object removes with a kind of visually reasonable manner.This algorithm combines the advantage based on the ﹠ Future Opportunities of Texture Synthesis of sample and small scale image repair technology effectively.This algorithm has proposed the concept of confidence level, and it is similar to the information propagation pattern in the small scale image repair technology.Although this algorithm has been eliminated the blurring effect in the former algorithm, but easily cause matching error, therefore for the abundant Incomplete image of structural information, repairing effect is not good.

Summary of the invention

The object of the invention is to overcome the above-mentioned deficiency of prior art, a kind of more efficient image completion method is provided.Do not need human intervention to reach the effect that keeps simultaneously structural information and texture information.

For this reason, the present invention adopts following technical scheme: a kind of image completion method of compages information comprises the following steps:

1. determine the target area of image, read in the image of the target area excessively to be repaired of mark.Judge the target area according to color value, draw mask images, represent with a logic matrix, wherein element value is that 1 this pixel of expression needs to repair in the matrix; Element value is 0 to be expressed as known pixels.Then this logic matrix and Laplace operator are done convolution, find out the border of this target area.

2. the cost energy of computing node self, E ₁(xi)=C (i) d (x _i, φ) wherein,

The expression node credibility, and when overlapping with known region, d (x _i, φ) expression piece x _iWith the matching degree of known region, it is that SSD weighs with the difference quadratic sum of color value between the pixel of overlapping region.

3. the relevance energy function of computing node, E ₂(x _i, x _j)=λ ₁E ₂ ^t(x _i, x _j)+λ ₂E ₂ ^s(x _i, x _j), E ₂ ^t(x _i, x _j)=d (x _i, x _j), wherein, d (x _i, x _j) expression adjacent block x _iAnd x _jThe SSD of overlapping region.

D wherein _Gx(x _i, x _j) and d _Gy(x _i, x _j) represent respectively adjacent block x _iAnd x _jGradient difference in X and Y-direction.

4. determine the overall goals function $E\left(x\right)=\underset{v_i\∈}{\mathrm{\Σ}}{E}_1\left(x_i\right)+\underset{(i,j)\∈E}{\mathrm{\Σ}}{E}_2(x_i,x_j).$

5. the message value between the initialization adjacent node, and upgrade.With any two node n among the bayesian network structure figure G _iAnd n _jBetween message be initially 0, namely

Upgrade the internodal value of information.During each iteration, for separating x _i, from n _iPropagate into n _jThe computing formula of information be:

Wherein N (j) i represent n _jNon-n _iThe field collection.m _{J → i} ^t(x _i) expression n _jNode is to x _iBe n _iDegree of belief.After T iteration, match block is x _iNode n _iThe reliability computing formula be: $b_i\left(x_i\right)=E_1\left(x_i\right)+\underset{j\∈N\left(i\right)}{\mathrm{\Σ}}{m}_{j\→i}^T\left(x_i\right).$

6. the optimum solution of computing node finds to make b _i(x _i) minimized x _i ^*Be exactly node n _iBest matching blocks.That is:

$x_i^{*}=\arg \underset{x_i}{\min }\{E_1\left(x_i\right)+\underset{j\∈N\left(i\right)}{\mathrm{\Σ}}{m}_{j\→i}^T\}.$

The present invention is on the basis of traditional image completion algorithm that synthesizes based on texture and TV image repair algorithm, has proposed the image completion algorithm of compages information, has greatly improved the effect of image completion.This fundamental idea of the invention is to propose corresponding energy object function for characteristics of image, then uses the reliability propagation algorithm with its optimization, and the match block that chooses is copied to image after defect area draws reparation.This paper has proposed new energy function, and it has comprised structural energy, texture energy and correlativity energy.Combine structure and texture information just because of new energy function, so that this invention has remedied the deficiency of two class classic algorithm, can avoid blurring effect again so that can keep structural information, and proved superiority of the present invention from theoretical and experiment two aspects.

Description of drawings

Fig. 1 is based on the image completion method overview flow chart of compages information.

Fig. 2 chooses target.

The initial pictures that Fig. 3 is to be repaired.

Fig. 4 has marked the image that needs removing objects.

Fig. 5 (a), (b), (c) are respectively the image completion arithmetic result figure of the compages information that proposes with the as a result figure of the as a result figure of TV image repair algorithm reparation, Criminisi method, the present invention.

Embodiment

The present invention proposes the piece disappearance image repair algorithm frame based on structural information, this algorithm utilizes Bayesian Network will be applicable to the TV algorithm of repair structure information and the Texture Synthesis combination that is applicable to repair texture information, its essence is to propose corresponding energy object function for characteristics of image, then with its optimization, the coupling texture block that then will choose copies to defect area, draws the image after the reparation.This invention does not need human intervention to reach the effect that keeps simultaneously structural information and texture information.

The present invention is based on the image completion method of compages information, and Fig. 1 is overview flow chart, specifically may further comprise the steps:

1. determine the target area of image.

As shown in Figure 4, read in the image of the target area excessively to be repaired of color mark.Judge the target area according to color value, draw mask images, represent with a logic matrix, wherein element value is that 1 this pixel of expression needs to repair in the matrix; Element value is 0 to be expressed as known pixels.Then this logic matrix and Laplace operator are done convolution, find out the border of this target area.Judge boundary value, if be non-zero, expression also has the pixel that needs reparation, then down carries out; If 0, expression reparation is finished, and withdraws from.

2. the cost energy of computing node self

If I represents whole image, Ω is target area (namely treating the completion zone).The same with ﹠ Future Opportunities of Texture Synthesis, need the first size of definite sample window Ψ, default value is 9 * 9 pixels.Fig. 2 shows the algorithm pattern that mates energy function.In order to define the node in the algorithm, the image lattice is by horizontal space pixel gap _xWith vertical space pixel gap _yForm.Known observation label is comprised of all w of regional φ * h piece, is designated as X={x ₁, x ₂... x _k.And N block of pixels in the expression target area is node.

According to the cost energy of each node self in formula (1) calculating chart 2 target areas, the i.e. rehabilitation cost of node.Wherein, C (i) represents node credibility, and its computing formula is (2), and C (i) value that the more outer field pixel in target area contains is larger, namely needs more early to fill; And it is less in C (i) value of the pixel at the center of target area.In the formula | Ψ _i| be piece | Ψ _i| area.

E ₁(x _i)＝C(i)d(x _i，φ) (1)

$C\left(i\right)=\frac{{\mathrm{\Σ}}_{q\∈\mathrm{\Ψi}\∩\stackrel{\‾}{\mathrm{\Ω}}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_i\right|}---\left(2\right)$

And when overlapping with known region, d (x _i, φ) expression piece x _iWith the matching degree of known region, it is that SSD weighs with the difference quadratic sum of color value between the pixel of overlapping region.Work as x _iAnd there is not when overlapping E between known region ₁(x _i)=0.As shown in Figure 2, E ₁(x _i) then be that the P piece is put into x _iThe SSD value of time domain 1 (red line mark) on the position.

3. the relevance energy function of computing node

Such as publicity (3), E ₂(x _i, x _j) expression (x _i, x _j) the consistance cost.Wherein, E ₂ ^t(x _i, x _j) the using texture homogeneity cost of adjacent block in the presentation video, E ₂ ^s(x _i, x _j) expression adjacent block the structural integrity cost.λ ₁And λ ₂Weight coefficient for balance texture cost and infrastructure cost.When containing more significant structural information in the target area, λ ₂Then larger; When texture information in the target area relatively enriches, λ ₁Then larger.

E ₂(x _i，x _j)＝λ ₁E ₂ ^t(x _i，x _j)+λ ₂E ₂ ^s(x _i，x _j) (3)

E ₂ ^t(x _i, x _j) be defined as:

E ₂ ^t(x _i，x _j)＝d(x _i，x _j) (4)

Wherein, d (x _i, x _j) expression adjacent block x _iAnd x _jThe SSD of overlapping region.As shown in Figure 2: E ₂ ^t(x _i, x _j) be that the piece P in any known region and P ' are put into respectively x _iAnd x _jThe time, lap is the SSD value in Green Marker zone.

E ₂ ^s(x _i, x _j) be defined as:

${E_2}^s(x_i,x_j)=\sqrt{d_{\mathrm{gx}}^2(x_i,x_j)+d_{\mathrm{gy}}^2(x_i,x_j)}---\left(5\right)$

D wherein _Gx(x _i, x _j) and d _Gy(x _i, x _j) represent respectively adjacent block x _iAnd x _jGradient difference in X and Y-direction.

4. determine the overall goals function.

The overall goals function definition is (6).The overall goals function has represented the Global Information in the entire image, such as: structure, texture, global consistency etc.The overall goals function be all nodes of target area energy and.In this algorithm, the energy function of node comprises the cost ENERGY E of self ₁With with the relevance energy function E of field node ₂Determining of overall goals function is the key of whole algorithm, and it is related to whether Global Information can keep effectively in the repair process.

$E\left(x\right)=\underset{v_i\∈v}{\mathrm{\Σ}}{E}_1\left(x_i\right)+\underset{(i,j)\∈E}{\mathrm{\Σ}}{E}_2(x_i,x_j)---\left(6\right)$

Wherein, E ₁(x _i) expression node rehabilitation cost, E ₂(x _i, x _j) expression (x _i, x _j) the consistance cost.

5. the message value between the initialization adjacent node, and upgrade.

With any two node n among the bayesian network structure figure G _iAnd n _jBetween message be initially 0, namely

Upgrade the internodal value of information

During each iteration, for separating x _i, from n _iPropagate into n _jThe computing formula of information be:

$m_{j\→i}^t\left(x\right)=\min [E_2(x_i,x_j)+E_1\left(x_j\right)+\underset{s\∈N\left(j\right)\backslash i}{\mathrm{\Σ}}{m}_{s\→j}^{t-1}\left(x_j\right)]---\left(7\right)$

Wherein N (j) i represent n _jNon-n _iThe field collection.m _{J → i} ^t(x _i) expression n _jNode is to x _iBe n _iDegree of belief.After T iteration, match block is x _iNode n _iThe reliability computing formula be:

$b_i\left(x_i\right)=E_1\left(x_i\right)+\underset{j\∈N\left(i\right)}{\mathrm{\Σ}}{m}_{j\→i}^T\left(x_i\right)---\left(8\right)$

6. the optimum solution of computing node.

Find and make b _i(x _i) minimized x _i ^*Be exactly node n _iBest matching blocks.That is:

$x_i^{*}=\arg \underset{x_i}{\min }\{E_1\left(x_i\right)+\underset{j\∈N\left(i\right)}{\mathrm{\Σ}}{m}_{j\→i}^T\}---\left(9\right)$

Fig. 3 is original image, Fig. 4 is for marking the image that needs removing objects, and Fig. 5 (a), (b), (c) are respectively the image completion arithmetic result figure of the compages information that proposes with the as a result figure of the as a result figure of TV image repair algorithm reparation, Criminisi method, the present invention.Difference to three width of cloth figure is carried out mark with red rectangle, can find out: (a) occur large-area fuzzy among the figure; (b) more serious matching error has appearred, the roof line in the red rectangle of mark and discontinuous among the figure; (d) figure has remedied the defective among front two width of cloth figure significantly.The color on roof is very complete in two rectangles, and lake surface completion effect is fine.

Claims (1)

1. the image completion method of a compages information comprises the following steps:

(1) determine the target area of image: read in the image that is labeled, judge the target area according to color value, draw mask images, represent with a logic matrix, wherein element value is that 1 expression needs the pixel of repairing in the matrix; Element value is 0 to be expressed as known pixels, then this logic matrix and Laplace operator is done convolution, finds out the border of this target area;

(2) the cost energy of computing node self, E ₁(x _i)=C (i) d (x _i, φ) wherein, The expression node credibility, and when overlapping with known region, d (x _i, φ) expression piece x _iWith the matching degree of known region, it is that SSD weighs with the difference quadratic sum of color value between the pixel of overlapping region, and Ω is the target area;

(3) the relevance energy function E of computing node ₂(x _i, x _j)=λ ₁E ₂ ^t(x _i, x _j)+λ ₂E ₂ ^s(x _i, x _j), E ₂ ^t(x _i, x _j)=d (x _i, x _j), wherein, d (x _i, x _j) expression adjacent block x _iAnd x _jThe SSD of overlapping region; ${E_2}^s(x_i,x_j)=\sqrt{d_{\mathrm{gx}}^2(x_i,x_j)+d_{\mathrm{gy}}^2(x_i,x_j)},$ Wherein, d _Gx(x _i, x _j) and d _Gy(x _i, x _j) represent respectively adjacent block x _iAnd x _jGradient difference in X and Y-direction; λ ₁And λ ₂Weight coefficient for balance texture cost and infrastructure cost;

(4) determine that overall goals function E (x) is the rehabilitation cost E of all nodes of target area ₁(x _i) and consistance cost E ₂(x _i, x _j) energy and;

(5) message value between the initialization adjacent node, and upgrade: with any two node n among the bayesian network structure figure _iAnd n _jBetween message be initially 0, namely Upgrade the internodal value of information, during each iteration, for separating x _i, from n _iPropagate into n _jThe computing formula of information be: $m_{j\→i}^t\left(x_i\right)=\min [E_2(x_i,x_j)+E_1\left(x_j\right)+\underset{s\∈N\left(j\right)\backslash i}{\mathrm{\Σ}}{m}_{s\→j}^{t-1}\left(x_j\right)],$ Wherein, N (j) i represent n _jNon-n _iThe neighborhood collection; Expression n _jNode is to x _iBe n _iDegree of belief; After t iteration, match block is x _iNode n _iThe confidence level computing formula be: $b_i\left(x_i\right)=E_1\left(x_i\right)+\underset{j\∈N\left(i\right)}{\mathrm{\Σ}}{m}_{j\→i}^t\left(x_i\right);$

(6) optimum solution of computing node finds to make b _i(x _i) minimized

Be exactly node n _iBest matching blocks, that is:

$x_i^{*}=\arg \underset{x_i}{\min }\{E_1\left(x_i\right)+\underset{j\∈N\left(i\right)}{\mathrm{\Σ}}{m}_{j\→i}^t\left(x_i\right)\}.$

Images