CN103578085A - Image empty region inpainting method based on variable blocks - Google Patents

Abstract

Provided is an image empty region inpainting method based on variable blocks. The image empty region inpainting method based on the variable blocks comprises the steps of (1) setting the initial dimensions of the blocks, and searching for the boundary of a region to be inpainted in an input image; (2) calculating priorities of the blocks, and selecting the block with the maximum priority as a current block to be inpainted, wherein the blocks use pixels, without being inpainted, on the boundary of the region to be inpainted as the centers, the size of each block is equal to the initial dimension, calculation of the priorities of the blocks is carried out through combination of credibility, structure conditions and texture conditions of the blocks; (3) searching for a best matched block of the current block to be inpainted, wherein the dimension of the current block to be inpainted is adjusted according to the texture conditions of the current block to be inpainted, and then the best matched block is searched on the basis of a color gradient histogram; (4) using the best matched image to inpaint the current block to be inpainted on the basis of an image segmentation technology, and repeatedly carrying out the step (2) to the step (4) until inpainting of the region to be inpainted in the input image is accomplished. According to the image empty region inpainting method based on the variable blocks, texture details can be well reserved under the condition that the texture conditions are complex or stable, and texture information and structural information can be repaired at the same time.

Description

Image hole region method for repairing and mending based on variable-block

Technical field

The invention belongs to image processing, computer vision, computer graphics and virtual reality field, specifically a kind of image hole region method for repairing and mending based on variable-block.

Background technology

As far back as the Renaissance, people just start to repair more medieval artworks, object is by filling up some cracks, more medieval pictures to be renovated, and such work is known as " Retouching(retouching) " or " Inpainting(repairs or repairs) ".Repair at that time is mainly repaired Shi Shougong by specialty and is completed, and requires a great deal of time and workload.

Be subject to the inspiration of the artificial repair of museum artist, the scholar of field of Computer Graphics attempts the method for coming imitate manual to repair with graphics algorithm, utilize Given information in present image to remove on target image unwanted scenery or to there being the damaged image of information to recover, make the image after repairing seem harmonious, nature, guarantee that the target image picture after repairing is continuous in consistent in structure, texture and brightness, image mending technology that Here it is.

In general, image mending technology mainly comprises three steps, 1) first input target image to be repaired; 2) by region to be repaired on user interactions intended target image; 3) using and treat that repairing area Given information is around as constraint, the Given information area monitoring from target image and the available information of copy are carried out the pixel in filling information lost regions.

Mainly there are at present three class image repair technology: the image repair based on structure (inpainting) technology, the technology based on the synthetic image completion of texture (completion) technology and integrated structure and texture.

Image repair technology based on structure is to be incorporated in image processing by people such as Bertalmio the earliest, they use 2D Laplacian method to estimate the smoothness of local color on each channel, then utilize smoothness by the outside information of the isoline of zone boundary to be repaired (Isophotes) along consistency profiles to being anisotropically diffused in the pixel to be repaired of inside, guaranteed the border continuity at place, image border.Afterwards, they were applied to image repair by the Navier-Stokes equation in fluid mechanics again, propose a kind of based on Navier-Stokes graph of equation as patch algorithm.Tony F.Chan and JianhongShen, by solving the third-order non-linear Diffusion Partial Differential Equations under structure Curvature-driven, reach the object of the target image of non-grain being carried out to visual continuity reparation.

The people such as Chan also propose the Variation based on total variational TV(Total) the image mending algorithm of model, by constraint and the diffusion of the anisotropy based on the curvature CDD(Curvature Driven Diffusion of Euler-Lagrange equation) process guarantees the seamless spliced of restoring area and original image.

Image repair technology based on structure, under the slickness hypothesis of image, obtains the colouring information of hole region by the resampling of fitting function.It can repair undersized information dropout region preferably; Yet it easily produces blur effect when repairing larger information dropout region, and helpless especially for the larger affected area with height texture features.

Based on the synthetic method of texture, be to regard Given information region around, cavity as texture sample, then by the synthetic method of texture, produce new image block and fill up drop-out region.Image mending method based on dough sheet adopts mode from coarse to fine, gathers the dough sheet that neighborhood with pixel to be repaired has Best similarity and fills, progressively the whole affected area of iterative approach in the ten-four region from image.First the people's such as Criminisi method sorts the dough sheet to be repaired on drop-out zone boundary by certain priority, area to be repaired is progressively filled up in the region that then the search neighborhood of pixels high with priority mated most in entire image.Owing to having considered the importance of structural information in mending course, strengthened having the repairing priority of the regional area of stronger structural information, this method has obtained extraordinary effect, becomes a milestone in image mending technical development process.

The technology of integrated structure and texture is mainly to consider that the method for utilizing based on structure is but difficult to process texture information to structural information sensitivity, and be easy to repair texture information based on the synthetic method of texture, structural information is lacked to control, so two kinds of methods have been combined to better repairing effect.The people such as Bertalmio propose the method based on picture breakdown, first algorithm is decomposed into structural drawing and texture maps input picture, then adopt the method preparing structure figure based on structure, employing is repaired texture maps based on the synthetic method of texture, finally merges the reparation result that the two width subimages that repair obtain original image.The method is separately processed the structure in image and texture information, avoided interfering with each other, so repair process only need consider the single characteristic of image, can have efficient recovery with the image of certain structural information.But algorithm complex is higher, whole repair process is consuming time larger.

Summary of the invention

The object of this invention is to provide a kind of image hole region method for repairing and mending based on variable-block, the method can all retain grain details preferably at texture complicated condition or more constant in the situation that, can repair texture information and structural information simultaneously.

The technical solution used in the present invention is as follows:

An image hole region method for repairing and mending for variable-block, comprises the following steps:

Step 1, the original dimension of setting piece is found the border for the treatment of repairing area on input picture;

Step 2, centered by each pixel that calculates respectively on zone boundary to be repaired not repair, size equals the priority of each piece of original dimension, chooses piece that priority is the highest as current patch to be repaired; The calculating of piece priority comprises: by the confidence level of the interior total proportion of pixel of known pixels point in piece and piece and the confidence level computing block of known pixels point, the structural condition of the structural information computing block of locating with zone boundary to be repaired, by the texture condition of the texture information computing block in piece, the confidence level of combined block, structural condition and texture condition are determined the priority of piece;

Step 3, the match block of searching for current patch to be repaired, comprising: according to the texture condition of current patch to be repaired, adjust the size of current patch to be repaired, then based on color gradient histogram, search for match block;

Step 4, repairs current patch to be repaired based on the figure technology of cutting by match block;

Repeated execution of steps 2-4, until the repairing area for the treatment of on input picture has been repaired.

In above-mentioned image hole region method for repairing and mending, preferably, the method for the structural condition of computing block is

$D\left(p\right)=\frac{|\▿I_p^{\⊥}\·n_p|}{\mathrm{\α}}$

The number of greyscale levels of input picture, n _pbe perpendicular to zone boundary to be repaired

vector of unit length, be the gradient of image, ⊥ is orthogonal operators,

intensity while being isophote arrival border.

In above-mentioned image hole region method for repairing and mending, preferably, the method for the texture condition of computing block is

ψ _pthe calibrated bolck of expression point centered by p point, β is normalized factor, p ' is piece ψ _pinterior other pixels of ordering except p, I (p ') is the color value of p ', (I (p ') μ) is I (p ') and piece ψ to f _paverage color μ between colour-difference, g (p, p ') is the Gaussian function based on p point and p ' space of points distance.

In above-mentioned image hole region method for repairing and mending, preferably, the confidence level C of combined block (p), structural condition D (p) and texture condition T (p) determine that the method for the priority P (p) of piece is

When the confidence level of piece is less than confidence level threshold value, the priority of putting piece is 0;

When the confidence level that the confidence level of piece is not less than confidence level threshold value or all is all less than confidence level threshold value, put priority P (the p)=C of piece ³(p)+D ³(p)-0.5 ³t ³(p).

In above-mentioned image hole region method for repairing and mending, preferably, if there is the priority of a plurality of the highest, from these a plurality of, select piece that confidence level is the highest as current patch to be repaired.

In above-mentioned image hole region method for repairing and mending, preferably, in step 3, the method of adjusting the size of current patch to be repaired according to the texture condition of current patch to be repaired comprises: when texture condition is less than corresponding texture threshold value, expand chunk sizes, contracting tile size when texture condition is greater than corresponding texture threshold value.

In above-mentioned image hole region method for repairing and mending, preferably, described texture condition is the texture condition after normalization, described texture threshold value comprises 1/3 and 2/3, when texture condition is less than 1/3, the size of piece is extended to 9x9 pixel, when texture condition be greater than 1/3 and while being less than 2/3 piece size remain unchanged, when texture condition is greater than 2/3, piece size is contracted to 3x3 pixel.

In above-mentioned image hole region method for repairing and mending, preferably, based on color gradient histogram, search for match block and comprise:

More current patch to be repaired and the last position relationship of having repaired piece, when the two approaches using last match block region around as region of search, when the two is kept off using whole known region as region of search;

By color gradient histogram, in corresponding region of search, search for match block.

In above-mentioned image hole region method for repairing and mending, preferably, step 4 also comprises: by the confidence level of inpainted pixels in this piece of belief update of piece, the confidence level of piece is less than the confidence level of pixel in known region.

The present invention has the following advantages:

(1), according to the size of the texture condition adjustment block of patch to be repaired, the in the situation that of texture complicated condition, select less piece can better retain the details of texture; The minimum unit of the reservation texture that the larger piece of selection can be complete in the situation that texture is more constant.

(2) in repair process, by confidence level, calculate and get rid of piece with a low credibility, can prevent on the one hand patch to be repaired around known pixels seldom and by force repairs the obvious repairing mark of generation, can accelerate on the other hand the computing velocity of piece priority.

(3) complexity by judgement texture information whether, preferentially selects texture information better simply, the part reparation of easily repairing.Judge the isophote intensity of borderline region to be repaired, the preferential informative part of choice structure, to reach the effect of repairing texture information and structural information simultaneously simultaneously.

(4) in piece matching process, added a decision condition, the region of search that can reduce greatly piece in matching process while having piece to meet this decision condition.

(5) by decision condition (relatively the histogram of two pieces, gets rid of if be less than threshold value), can get rid of fast two pieces that texture condition is different.The histogrammic method of color gradient that is used in field of image search is mated for piece simultaneously, can to object, block and chaotic interference robust to a certain extent, and improve the accuracy of coupling.

(6) utilize the figure technology of cutting to carry out the renewal process of piece, reduce greatly the artificial vestige of repairing, improved the effect of repairing.By the renewal of confidence level, to the pixel after repairing, give the confidence level lower than known region pixel simultaneously, when mending course normally carries out after having guaranteed, can pay the utmost attention to the many pieces of known region pixel around.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the image hole region method for repairing and mending of embodiment based on variable-block;

Fig. 2 wherein chooses the process flow diagram of current patch to be repaired by calculating priority level;

Fig. 3 is the process flow diagram of wherein searching for the match block of current patch to be repaired;

Fig. 4 wherein repairs the process flow diagram of current patch to be repaired by match block.

Embodiment

The present invention relates to the image hole region method for repairing and mending based on variable-block, for improving repair efficiency.The major technique adopting in the present invention comprises according to the confidence level of area to be repaired piece centered by pixel on border, the priority that structural information and texture condition judgment are repaired, utilize texture condition in piece to select block size, piece by mating most with current block in color gradient histogram search known pixels region, cuts the piece update strategy of technology based on figure.

Below in conjunction with drawings and Examples, the present invention is described further.

As shown in Figure 1, this image hole region method for repairing and mending based on variable-block comprises the following steps:

Step 1, pre-treatment step, this pre-treatment step can be divided into following two steps: one, input image l to be repaired, and select to treat repairing area (being target area) Ω.Definition known region Φ=l-Ω.Setting initial block size (being the original dimension of piece) is 5 * 5 pixels.Two, find the border for the treatment of repairing area Ω the original dimension that should be appreciated that piece is not limited to 5 * 5 pixels, can set flexibly according to actual conditions, as being set as 4 * 4 pixels or 6 * 6 pixels etc. when application.

Step 2, calculates respectively zone boundary to be repaired

on centered by each pixel of not repairing, size equals the priority of each piece of original dimension, chooses piece that priority is the highest as current patch to be repaired.Wherein, the calculating of priority has considered confidence level, structural condition and the texture condition of piece.

Step 3, the match block of searching for current patch to be repaired.

Step 4, repairs current patch to be repaired based on the figure technology of cutting by match block.

Step 5, judgement border

whether be zero, border

be that zero repairing completes, output image; Border

non-vanishingly go to step 2, repeated execution of steps 2-4, until the repairing area for the treatment of on input picture has been repaired.

As shown in Figure 2, above-mentioned steps 2 can be divided into following step:

Step 2-1, on border

upper selected pixels point p,

Step 2-2, calculate take centered by a p, piece ψ that size is 5 * 5 pixels _pconfidence level C (p).In preferred embodiment, with the confidence level C (p) of the interior total proportion of pixel of known pixels point in piece and piece and the confidence level computing block of known pixels point, formula is as follows

$C\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈{\mathrm{\ψ}}_p\∩(I-\mathrm{\Ω})}C\left(q\right)}{\left|{\mathrm{\ψ}}_p\right|}$

│ Ψ wherein _p│ represents piece Ψ _parea, q represents piece Ψ _pinterior known pixels point.When mending course starts, to make the some confidence level in region to be filled up be 0 in initialization, and the some confidence level of known region is 1.

Step 2-3, calculate take centered by a p, piece Ψ that size is 5 * 5 pixels _pstructural condition D (p).In preferred embodiment, the computing method of structural condition are

$D\left(p\right)=\frac{|\▿I_p^{\⊥}\·n_p|}{\mathrm{\α}}$

The number of greyscale levels of input picture, n _pbe perpendicular to border

vector of unit length,

be the gradient of image, ⊥ is orthogonal operators,

intensity while being isophote arrival border.

Step 2-4, calculate take centered by a p, piece Ψ that size is 5 * 5 pixels _ptexture condition T (p).In preferred embodiment, the computing method of texture condition are

Be normalized factor, p ' is other pixels of ordering except p in piece, and I (p ') is the color value of p ', and (I (p ') μ) is I (p ') and piece ψ to f _paverage color between colour-difference, g (p, p ') is the Gaussian function based on p point and p ' space of points distance.T (p) value greatly has more high-frequency informations in illustrated block, and the texture information that T (p) is worth in little illustrated block is likely constant color value.

Step 2-5, calculate take centered by a p, piece ψ that size is 5 * 5 pixels _ppriority P (p).In preferred embodiment, the computing method of priority P (p) are: if confidence level C (p) is <0.5, put priority P (p)=0.If confidence level C (p)>=0.5, or border

do not make the p point of C (p)>=0.5, put priority P (p)=C ³(p)+D ³(p)-0.5 ³t ³(p).

Step 2-6, border

on point whether calculated, do not calculated and returned to step 2-1; Calculated and selected the piece that priority is the highest.

Step 2-7, judges that whether the piece that priority is the highest is unique, unique using it as current patch to be repaired, not uniquely chooses piece with a high credibility in the piece that priority is the highest as current patch to be repaired.

As shown in Figure 3, above-mentioned steps 3 can be divided into following step:

Step 3-1, the size of adjusting current patch to be repaired according to the texture condition T (p) of current patch to be repaired, comprising: when texture condition is less than corresponding texture threshold value, expand chunk sizes, contracting tile size when texture condition is greater than corresponding texture threshold value.In preferred embodiment, texture condition T (p) is the texture condition after normalization, and texture threshold value comprises 1/3 and 2/3, and method of adjustment comprises

If a) texture condition T (p) <

putting block size is 9 * 9 pixels, expands current patch to be repaired;

B) if texture condition

putting block size is 5 * 5 pixels, and piece size remains unchanged;

C) if texture condition

putting block size is 3 * 3 pixels, dwindles current patch to be repaired.

Step 3-2, searches for match block based on color gradient histogram.Comprise:

More current patch to be repaired and the last position relationship of having repaired piece, when the two approaches using last match block region around as region of search, when the two is kept off using whole known region as region of search;

By color gradient histogram, in corresponding region of search, search for match block.First by histogram intersection (histogram intersection), as decision condition, in region of search, select the more similar piece of colouring information.Then introduce the concept of color gradient (color ration gradient): process respectively each Color Channel, the sum of change color can be estimated by cumulative Grad.Use I _kdefine a specific Color Channel, the color gradient of input picture l can obtain by Euclidean distance:

$\&dtri;F={\left(\underset{k=1}{\overset{N}{\mathrm{\&Sigma;}}}\right[{\left(\frac{{\&PartialD;I}_k}{\&PartialD;x}\right)}^2+{\left(\frac{{\&PartialD;I}_k}{\&PartialD;y}\right)}^2\left]\right)}^{\frac{1}{2}}$

Wherein N is the dimension in input image colors space.

Color gradient histogram is to obtain by color ratio rate gradient occurrence number discrete in computing block.Pass through histogram intersection

at known region, search out the piece mating most:

$D(H_{{\mathrm{\&psi;}}_p},H_{{\mathrm{\&psi;}}_s})=\frac{{\mathrm{\&Sigma;}}_{n=1}^{N_d}\min \{H_{{\mathrm{\&psi;}}_p}\left(n\right),H_{{\mathrm{\&psi;}}_s}\left(n\right)\}}{{\mathrm{\&Sigma;}}_{n=1}^{N_d}{H}_{{\mathrm{\&psi;}}_p}\left(n\right)}$

By normalized crosscorrelation, can obtain:

$D(H_{{\mathrm{\&psi;}}_p},H_{{\mathrm{\&psi;}}_s})=\frac{{\mathrm{\&Sigma;}}_{n=1}^{N_d}{H}_{{\mathrm{\&psi;}}_p}\left(n\right),H_{{\mathrm{\&psi;}}_s}\left(n\right)}{{\mathrm{\&Sigma;}}_{n=1}^{N_d}{\left(H_{{\mathrm{\&psi;}}_p}\left(n\right)\right)}^2}$

with

it is respectively the histogram of the piece of current patch to be repaired and known region search.N _dbe histogrammic group of number, n is the sequence number of group.

As shown in Figure 4, above-mentioned steps 4 comprises following two steps:

Step 4-1, copies to current patch to be repaired place by the match block searching, and in overlapping region, utilizes the figure technology of cutting to find optimum boundary.Particularly, definition p _i, p _jfor the pixel on overlapping region, I _p(p _i) and I _s(p _i) be respectively current patch to be repaired and match block at p _icolor value on point, I _p(p _j) and I _s(p _j) be respectively current patch to be repaired and match block at p _jcolor value on point, boundary function:

|| || represent the Euclidean distance between color value,

neighbours territories.The optimum function M (p that demarcates that demarcates _i, p _j) minimum situation.If minimum boundary function M (p _i, p _j) more than one, choose the point approaching with a upper separation as current separation.

Step 4-2, after piece has been repaired, the confidence level of putting in piece pixel in repairing area is:

C(p′)=C(p) $\&ForAll;p^{\&prime;}\&Element;{\mathrm{\&psi;}}_p\&cap;\mathrm{\&Omega;}$

The confidence level that is about to piece is given the pixel after filling up in piece.

Claims (9)

1. the image hole region method for repairing and mending based on variable-block, is characterized in that, comprising:

Step 1, the original dimension of setting piece is found the border for the treatment of repairing area on input picture;

Step 2, centered by each pixel that calculates respectively on zone boundary to be repaired not repair, size equals the priority of each piece of original dimension, chooses piece that priority is the highest as current patch to be repaired; The calculating of piece priority comprises: by the confidence level of the interior total proportion of pixel of known pixels point in piece and piece and the confidence level computing block of known pixels point, the structural condition of the structural information computing block of locating with zone boundary to be repaired, by the texture condition of the texture information computing block in piece, the confidence level of combined block, structural condition and texture condition are determined the priority of piece;

Step 3, the match block of searching for current patch to be repaired, comprising: according to the texture condition of current patch to be repaired, adjust the size of current patch to be repaired, then based on color gradient histogram, search for match block;

Step 4, repairs current patch to be repaired based on the figure technology of cutting by match block;

Repeated execution of steps 2-4, until the repairing area for the treatment of on input picture has been repaired.

2. image hole region method for repairing and mending according to claim 1, is characterized in that, the method for the structural condition of computing block is

$D\left(p\right)=\frac{|\&dtri;I_p^{\&perp;}\&CenterDot;n_p|}{\mathrm{\&alpha;}}$

The number of greyscale levels of input picture, n _pbe perpendicular to zone boundary to be repaired vector of unit length,

be the gradient of image, ⊥ is orthogonal operators,

intensity while being isophote arrival border.

3. image hole region method for repairing and mending according to claim 1, is characterized in that, the method for the texture condition of computing block is

ψ _pthe calibrated bolck of expression point centered by p point, β is normalized factor, p ' is piece ψ _pinterior other pixels of ordering except p, I (p ') is the color value of p ', (I (p ') μ) is I (p ') and piece ψ to f _paverage color between colour-difference, g (p, p ') is the Gaussian function based on p point and p ' space of points distance.

4. image hole region method for repairing and mending according to claim 1, is characterized in that, the confidence level C of combined block (p), structural condition D (p) and texture condition T (p) determine that the method for the priority P (p) of piece is

When the confidence level of piece is less than confidence level threshold value, the priority of putting piece is 0;

When the confidence level that the confidence level of piece is not less than confidence level threshold value or all is all less than confidence level threshold value, put priority P (the p)=C of piece ³(p)+D ³(p)-0.5 ³t ³(p).

5. image hole region method for repairing and mending according to claim 1, is characterized in that, if there is the priority of a plurality of the highest, from these a plurality of, selects piece that confidence level is the highest as current patch to be repaired.

6. image hole region method for repairing and mending according to claim 1, it is characterized in that, in step 3, the method of adjusting the size of current patch to be repaired according to the texture condition of current patch to be repaired comprises: when texture condition is less than corresponding texture threshold value, expand chunk sizes, contracting tile size when texture condition is greater than corresponding texture threshold value.

7. image hole region method for repairing and mending according to claim 6, it is characterized in that: described texture condition is the texture condition after normalization, described texture threshold value comprises 1/3 and 2/3, when texture condition is less than 1/3, the size of piece is extended to 9x9 pixel, when texture condition be greater than 1/3 and while being less than 2/3 piece size remain unchanged, when texture condition is greater than 2/3, piece size is contracted to 3x3 pixel.

8. image hole region method for repairing and mending according to claim 1, is characterized in that, searches for match block comprise based on color gradient histogram:

More current patch to be repaired and the last position relationship of having repaired piece, when the two approaches using last match block region around as region of search, when the two is kept off using whole known region as region of search;

By color gradient histogram, in corresponding region of search, search for match block.

9. image hole region method for repairing and mending according to claim 1, is characterized in that, step 4 also comprises: by the confidence level of inpainted pixels in this piece of belief update of piece, the confidence level of piece is less than the confidence level of pixel in known region.

Images