CN103150711A - Open computing language (OpenCL)-based image repair method - Google Patents

Abstract

The invention discloses an open computing language (OpenCL)-based image repair method, and belongs to the technical field of automatic image repair. An area to be repaired, a boundary and a source area are determined through an OpenCL platform by a computer, a target block and source blocks are defined, sum of square of deviations (SSDs) of color information of pixels of all the source blocks and the target block are calculated, and the source block with the minimum SSD is found by a central processing unit (CPU), and is used for repairing the target block, so that the repair of the whole image is finished. An OpenCL-based parallel acceleration method is provided, the characteristic of low cost of switching between a plurality of kernels and threads of a graphic processing unit (GPU) is utilized, and a part of simple and repeated work in an image repair algorithm is parallelized, so that the image processing time is greatly shortened, the image repair real-time performance is improved, and an application range is widened.

Description

A kind of image repair method based on OpenCL

Technical field

The present invention relates to a kind of image repair method, be specifically related to a kind of based on OpenCL(Open Computing Language, open computational language) GPU(Graphic Processing Unit, graphic process unit) accelerate image repair method, be used for impaired image repair or remove the shelter of image.

Technical background

Image repair refers to treat that according to image the known neighborhood information estimation of patch area treats the process that in patch area, defect information is repaired; its fundamental purpose is under human eye subjective system acceptable degree, the image of breakage to be repaired; be a study hotspot in computer vision field, remove the field and have a wide range of applications at historical relic's protection, video display special technology making, virtual reality, unnecessary object.And, growing along with the 3D technology, the virtual view synthetic technology needs fast the image cavity to repair, and greatly promotes for the rate request of algorithm.

Two class basic skills of present image reparation based on the texture reparation with based on non-texture reparation.Its algorithm idea of restorative procedure based on texture is first from waiting that repairing regional border chooses a pixel, simultaneously centered by this point, textural characteristics according to image, choose sizeable texture block, then wait to repair seek around the zone with it recently like the Texture Matching piece substitute this texture block.Its representative algorithm is the algorithm that the people such as Criminisi proposes, it has been used for reference, and the thought in the Texture Generating Approach is sought sample areas and coupling copies, take full advantage of simultaneously based on the diffusion way in the restorative procedure of structure and define the priority of repairing piece, make the reparation piece that is near edge (having more structural information) have higher reparation priority, thereby when repairing texture information, structural information is also had certain maintenance.At present, the workflow of Criminisi image repair algorithm is:

1, determine wait repairing zone, border, source region (namely removing the picture region after repairing zone and border).

2, centered by each point on zone boundary to be repaired, choose size and be m*m(such as 9*9) image block, calculate the priority value of each image block, priority value has two aspects to consist of: 1. in repairing area, the degree of confidence of pixel is big or small, and namely the known pixels number accounts for the ratio of pixel count in piece; 2. treat linear structure information that patch area comprises can with form a continuous linear structure on every side, treat namely whether the neighborhood of patch area is closed.

3, seek the highest reparation piece of priority along zone boundary to be repaired, result is object block.

4, at the image known region, all known regions of traversing graph picture, choose onesize image block (being the source piece), compare with object block, comparative approach is that the RGB(of reference source piece and each pixel of object block is redgreenblue) value, take turns doing the poor quadratic sum of asking R value, G value, B value difference value, obtain the summation of all these quadratic sums of pixel in piece, the piece of summation minimum is best matching blocks and is used for repairing.

5, upgrade and to treat the patch area border, and degree of confidence, priority value.

6, detect whether to remain repairing area, have and carried out for the 3rd step, if do not wait to repair the zone, algorithm finishes, and picture is repaired and completed.

Yet, traditional based on the CPU(central processing unit) Criminisi image repair algorithm often can not utilize to greatest extent existing calculation resources, because on the CPU platform, computing is serial, seek needs of work such as repairing piece and carry out calculating, coupling, the sequence of hundreds thousand of and even millions of repetitions according to the picture pixel count, often be wasted in operation time in the calculating and matching operation of waiting for a upper block of pixels.Although calculating is very fast each time, numerous operating delay is accumulated at together, will cause algorithm consuming time huge, has completely lost the ability of fast processing.Be 200510012165.1 as the patent No., denomination of invention namely belongs to these row for the patent of " a kind of image repair method ".

OpenCL full name Open Computing Language i.e. open computational language.OpenCL provides the framework standard of the opening of coding, especially a concurrent program for heterogeneous platform.The heterogeneous platform that OpenCL supports can be comprised of the processor of multi-core CPU, GPU or other types.OpenCL is comprised of two parts, and the one, be used to the language of writing kernel program (code that moves) on OpenCL equipment, the 2nd, define and control the API of platform.OpenCL provides based on task and two kinds of parallel computation mechanism of based on data, and it has greatly expanded the range of application of GPU, makes it no longer to be confined to the figure field.The present invention utilizes the subprogram in OpenCL optimization Criminisi image repair algorithm, utilizes the serial computing of the parallel computation replacement CPU of GPU, has produced huge acceleration effect.

Summary of the invention

The present invention is directed to present Criminisi image repair algorithm and calculate long drawback working time, the present invention proposes the parallel acceleration method based on OpenCL, utilize the characteristics that between GPU multinuclear, thread, switching is dirt cheap, with a large amount of simple work parallelizations that repeat of part in the image repair algorithm, greatly shorten the image processing time, improved real-time and the range of application of method.

Technical scheme of the present invention realizes in the following manner:

A kind of image repair method based on OpenCL, realized by computing machine, video card in computing machine is configured to support the video card of OpenCL agreement (for the AMD video card, from support the OpenCL standard more than HD5800 series comprehensively) or processor adopting AMD APU(Accelerated Processing Unit, OverDrive Processor ODP), the method step is as follows:

1, OpenCL platform initialization comprises and obtains the OpenCL platform, generates context, seek OpenCL equipment, create command queue, create OpenCL buffer(buffer memory) object, the creation procedure object, the program compiler object generates the program that kernel(namely can move in GPU);

2, determine wait repairing zone, border, source region (namely removing the picture region after repairing zone and border);

Wherein, the damaged place of repairing of maybe needing in region representation picture to be repaired, source region refers to the part of information completely in picture, the border refers to the boundary line between zone to be repaired and source region;

3, centered by each point on zone boundary to be repaired, choosing square image blocks (is sample block, size is decided according to picture texture, precision), calculate the priority value of each image block, priority value has two aspects to consist of: 1. in repairing area, the degree of confidence of pixel is big or small, and namely the known pixels number accounts for the ratio of pixel count in piece; 2. treat linear structure information that patch area comprises can with form a continuous linear structure on every side, treat namely whether the neighborhood of patch area is closed;

Image block ψ centered by 1 p on zone boundary to be repaired _pPriority P (p) computing formula as follows:

P(p)＝C(p)·D(p)

Wherein, C (p) is the degree of confidence item, and D (p) is data item, is defined as follows:

$C\left(p\right)=\frac{\underset{q\∈{\mathrm{\Ψ}}_p\∩\mathrm{\Φ}}{\mathrm{\Σ}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_p\right|}$

$D\left(p\right)=\frac{|\▿I_p^{\⊥}\·n_p|}{\mathrm{\α}}$

Minute subitem in C (p) expression formula represents the quantity of non-empty pixel in sample block, wherein: pixel, a Φ in q representative image piece (being sample block) represent picture source region (being the part of information completely in picture), (this expression comprises the process of an iteration to the confidence value that C (q) represents respectively each qualified q pixel, be that the confidence value of all qualified q pixels draws around it by calculating for the confidence value of p pixel), denominator | Ψ _p| total pixel quantity in the expression sample block, Ψ _p∩ Φ represents both to belong to the sample block centered by p, belongs to again the source region of image,

The ∈ symbol point q of described condition is afterwards satisfied in expression; In D (p) expression formula

The direction and the normal vector that represent respectively the isophote at p point on empty edge place,

Represent the direction intensity of isophote at p point on empty edge place in the drop shadow intensity that boundary method makes progress, α is normalized factor, and value is 255;

4, seek the highest image block of priority along zone boundary to be repaired, and the highest image block of definition priority is object block;

5, with former figure RGB(RGB) information, object block RGB(RGB) information and border, treat that patch area, source region zone bit matrix write the buffer of OpenCL equipment, and import the value in these buffer into kernel;

6 determine and distribute working cell in workitem(GPU) quantity, the size (be generally 16*16 workitem and form a workgroup) of the workgroup of working group that formed by workitem;

7, each workgroup is calculated the information of needs (as source figure rgb value, the object block rgb value) be mapped in the local internal memory of each workgroup, need this mappings work synchronous (mappings work of namely waiting for all workgroup is completed) after completing this work, then kernel is put into command queue's kernel program that brings into operation, the content of Kernel program is for to choose the image block onesize with object block at source region, be defined as the source piece, source piece and object block compare and calculate the error sum of squares SSD of chromatic information of the pixel of two image blocks (source piece and object block), comparative approach is: the RGB(of reference source piece and object block respective pixel is redgreenblue) value, it is poor to take turns doing, ask the R value, the G value, the quadratic sum of B value difference value, adopt this method, source region at whole picture, take a pixel as step-length, choose from left to right from top to bottom all qualified (condition is that in piece, all pixels are all the source region pixel) as the source piece, obtain above-mentioned active error sum of squares SSD with the chromatic information of object block pixel, like this by the source region on the whole picture of traversal, GPU finally can with active calculate with the image difference of same object block is parallel,

The formula of the error sum of squares SSD of the chromatic information of calculating source piece and object block pixel is as follows:

$\mathrm{SSD}=\mathrm{\Σ}\left\{\begin{array}{c}{(R_p\left(P\right)-R_q\left(P\right))}^2+\\ {(G_p\left(P\right)-G_q\left(P\right))}^2+\\ {(B_p\left(P\right)-B_q\left(P\right))}^2\end{array}\right\}$

Wherein, Rp, Gp, Bp represent respectively the intensity level (intensity value range is 0-255) of the RGB three primary color components of pixel in the piece of source; Rq, Gq, Bq represent respectively the intensity level of the RGB three primary color components of pixel in object block, add alphabetical P representative in formula in the bracket of above-mentioned alphabetical back and do two poor pixels at source piece and relative position identical (being all for example the third line in piece, the 4th that pixel that is listed as) in object block;

8, complete above-mentioned active computing with the error sum of squares SSD of the chromatic information of object block pixel after, the operation result information in kernel is write buffer, read by CPU;

9, CPU finds out the source piece of error sum of squares SSD minimum of the chromatic information of source piece and object block pixel, and this source piece is blocks and optimal matching blocks, gives object block with the rgb value of blocks and optimal matching blocks, namely completes the repairing of object block; Due to the repairing of object block image, its border will change, and border, degree of confidence, priority all should be upgraded, and treats the patch area border to upgrade frontier point degree of confidence, priority value therefore again upgrade;

10, CPU detects and whether to remain repairing area, has and carries out for the 3rd step, if finish in zone not repaired, picture is repaired and completed.

Method is comprised of CPU program and kernel program, and the CPU program is moved on CPU, and the kernel program is moved on GPU.Like this, we with image ratio in the criminisi algorithm to this huge heavy and work of repeating, transforming the kernel program as transfers to hundreds of cores of GPU and processes, greatly improve travelling speed, had extremely important and positive effect for the real-time that improves algorithm, the application of expansion algorithm.

Embodiment

The present invention will be further described below in conjunction with embodiment, but be not limited to this.

Embodiment:

A kind of image repair method based on OpenCL, realized by computing machine, video card in computing machine is configured to support the video card of OpenCL agreement (for the AMD video card, from support the OpenCL standard more than HD5800 series comprehensively) or processor adopting AMD APU(Accelerated Processing Unit, OverDrive Processor ODP), the method step is as follows:

1, OpenCL platform initialization comprises and obtains the OpenCL platform, generates context, seek OpenCL equipment, create command queue, create OpenCL buffer(buffer memory) object, the creation procedure object, the program compiler object generates the program that kernel(namely can move in GPU);

2, determine wait repairing zone, border, source region (namely removing the picture region after repairing zone and border);

Wherein, the damaged place of repairing of maybe needing in region representation picture to be repaired, source region refers to the part of information completely in picture, the border refers to the boundary line between zone to be repaired and source region;

3, centered by each point on zone boundary to be repaired, choosing square image blocks (is sample block, size is decided according to picture texture, precision), calculate the priority value of each image block, priority value has two aspects to consist of: 1. in repairing area, the degree of confidence of pixel is big or small, and namely the known pixels number accounts for the ratio of pixel count in piece; 2. treat linear structure information that patch area comprises can with form a continuous linear structure on every side, treat namely whether the neighborhood of patch area is closed;

Image block ψ centered by 1 p on zone boundary to be repaired _pPriority P (p) computing formula as follows:

P(p)＝C(p)·D(p)

Wherein, C (p) is the degree of confidence item, and D (p) is data item, is defined as follows:

$C\left(p\right)=\frac{\underset{q\∈{\mathrm{\Ψ}}_p\∩\mathrm{\Φ}}{\mathrm{\Σ}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_p\right|}$

$D\left(p\right)=\frac{|{\▿}_p^{\⊥}\·n_p|}{\mathrm{\α}}$

Minute subitem in C (p) expression formula represents the quantity of non-empty pixel in sample block, wherein: pixel, a Φ in q representative image piece (being sample block) represent picture source region (being the part of information completely in picture), (this expression comprises the process of an iteration to the confidence value that C (q) represents respectively each qualified q pixel, be that the confidence value of all qualified q pixels draws around it by calculating for the confidence value of p pixel), denominator | Ψ _p| total pixel quantity in the expression sample block, Ψ _p∩ Φ represents both to belong to the sample block centered by p, belongs to again the source region of image,

The ∈ symbol point q of described condition is afterwards satisfied in expression; In D (p) expression formula

The direction and the normal vector that represent respectively the isophote at p point on empty edge place, Represent the direction intensity of isophote at p point on empty edge place in the drop shadow intensity that boundary method makes progress, α is normalized factor, and value is 255;

4, seek the highest image block of priority along zone boundary to be repaired, and the highest image block of definition priority is object block;

5, with former figure RGB(RGB) information, object block RGB(RGB) information and border, treat that patch area, source region zone bit matrix write the buffer of OpenCL equipment, and import the value in these buffer into kernel;

6 determine and distribute working cell in workitem(GPU) quantity, the size (be generally 16*16 workitem and form a workgroup) of the workgroup of working group that formed by workitem;

7, each workgroup is calculated the information of needs (as source figure rgb value, the object block rgb value) be mapped in the local internal memory of each workgroup, need this mappings work synchronous (mappings work of namely waiting for all workgroup is completed) after completing this work, then kernel is put into command queue's kernel program that brings into operation, the content of Kernel program is for to choose the image block onesize with object block at source region, be defined as the source piece, source piece and object block compare and calculate the error sum of squares SSD of chromatic information of the pixel of two image blocks (source piece and object block), comparative approach is: the RGB(of reference source piece and object block respective pixel is redgreenblue) value, it is poor to take turns doing, ask the R value, the G value, the quadratic sum of B value difference value, adopt this method, source region at whole picture, take a pixel as step-length, choose from left to right from top to bottom all qualified (condition is that in piece, all pixels are all the source region pixel) as the source piece, obtain above-mentioned active error sum of squares SSD with the chromatic information of object block pixel, like this by the source region on the whole picture of traversal, GPU finally can with active calculate with the image difference of same object block is parallel,

The formula of the error sum of squares SSD of the chromatic information of calculating source piece and object block pixel is as follows:

$\mathrm{SSD}=\mathrm{\Σ}\left\{\begin{array}{c}{(R_p\left(P\right)-R_q\left(P\right))}^2+\\ {(G_p\left(P\right)-G_q\left(P\right))}^2+\\ {(B_p\left(P\right)-B_q\left(P\right))}^2\end{array}\right\}$

Wherein, Rp, Gp, Bp represent respectively the intensity level (intensity value range is 0-255) of the RGB three primary color components of pixel in the piece of source; Rq, Gq, Bq represent respectively the intensity level of the RGB three primary color components of pixel in object block, add alphabetical P representative in formula in the bracket of above-mentioned alphabetical back and do two poor pixels at source piece and relative position identical (being all for example the third line in piece, the 4th that pixel that is listed as) in object block;

8, complete above-mentioned active computing with the error sum of squares SSD of the chromatic information of object block pixel after, the operation result information in kernel is write buffer, read by CPU;

9, CPU finds out the source piece of error sum of squares SSD minimum of the chromatic information of source piece and object block pixel, and this source piece is blocks and optimal matching blocks, gives object block with the rgb value of blocks and optimal matching blocks, namely completes the repairing of object block; Due to the repairing of object block image, its border will change, and border, degree of confidence, priority all should be upgraded, and treats the patch area border to upgrade frontier point degree of confidence, priority value therefore again upgrade;

10, CPU detects and whether to remain repairing area, has and carries out for the 3rd step, if finish in zone not repaired, picture is repaired and completed.

Claims (1)

1. the image repair method based on OpenCL, realized by computing machine, and the video card in computing machine is configured to support video card or the processor adopting AMDAPU of OpenCL agreement, and the method step is as follows:

1) OpenCL platform initialization comprises and obtains the OpenCL platform, generates context, seeks OpenCL equipment, creates command queue, creates OpenCL buffer object, the creation procedure object, and the program compiler object generates kernel;

2) determine to wait to repair zone, border, source region, source region is namely removed the picture region after repairing zone and border;

Wherein, the damaged place of repairing of maybe needing in region representation picture to be repaired, source region refers to the part of information completely in picture, the border refers to the boundary line between zone to be repaired and source region;

3) centered by each point on zone boundary to be repaired, choose square image blocks as sample block, size is decided according to picture texture, precision, calculate the priority value of each image block, priority value has two aspects to consist of: 1. in repairing area, the degree of confidence of pixel is big or small, and namely the known pixels number accounts for the ratio of pixel count in piece; 2. treat linear structure information that patch area comprises can with form a continuous linear structure on every side, treat namely whether the neighborhood of patch area is closed;

Image block ψ centered by 1 p on zone boundary to be repaired _pPriority P (p) computing formula as follows:

P(p)＝C(p)·D(p)

Wherein, C (p) is the degree of confidence item, and D (p) is data item, is defined as follows:

$C\left(p\right)=\frac{\underset{q\∈{\mathrm{\Ψ}}_p\∩\mathrm{\Φ}}{\mathrm{\Σ}}C\left(q\right)}{\left|{\mathrm{\Ψ}}_p\right|}$

$D\left(p\right)=\frac{|\▿I_p^{\⊥}\·n_p|}{\mathrm{\α}}$

Minute subitem in C (p) expression formula represents the quantity of non-empty pixel in sample block, wherein: pixel, a Φ in q representative image piece represents the source region of picture, it is the part of information completely in picture, C (q) represents respectively the confidence value of each qualified q pixel, this value representation comprises the process of an iteration, be the confidence value of p pixel the confidence value of all qualified q pixels draws around it by calculating, denominator | Ψ _p| total pixel quantity in the expression sample block, Ψ _p∩ Φ represents both to belong to the sample block centered by p, belongs to again the source region of image, The ∈ symbol point q of described condition is afterwards satisfied in expression; In D (p) expression formula

The direction and the normal vector that represent respectively the isophote at p point on empty edge place,

Represent the direction intensity of isophote at p point on empty edge place in the drop shadow intensity that boundary method makes progress, α is normalized factor, and value is 255;

4) seek the highest image block of priority along zone boundary to be repaired, and the highest image block of definition priority is object block;

5) with former figure RGB information, object block RGB information and border, treat that patch area, source region zone bit matrix write the buffer of OpenCL equipment, and import the value in these buffer into kernel;

6) size of the workgroup of working group that determine and distribute the quantity of workitem, is formed by workitem;

7) each workgroup is calculated information such as the source figure rgb value that needs, the object block rgb value is mapped in the local internal memory of each workgroup, after completing this work, need are synchronous with this mappings work, wait for that namely the mappings work of all workgroup completes, then kernel is put into command queue's kernel program that brings into operation, the content of Kernel program is for to choose the image block onesize with object block at source region, be defined as the source piece, source piece and object block compare and calculate the error sum of squares SSD of the chromatic information of the pixel in source piece and object block, comparative approach is: the rgb value of reference source piece and object block respective pixel, it is poor to take turns doing, ask the R value, the G value, the quadratic sum of B value difference value, adopt this method, source region at whole picture, take a pixel as step-length, choose from left to right from top to bottom all qualified as the source piece, obtain above-mentioned active error sum of squares SSD with the chromatic information of object block pixel, like this by the source region on the whole picture of traversal, GPU finally can with active calculate with the image difference of same object block is parallel,

The formula of the error sum of squares SSD of the chromatic information of calculating source piece and object block pixel is as follows:

$\mathrm{SSD}=\mathrm{\Σ}\left\{\begin{array}{c}{(R_p\left(P\right)-R_q\left(P\right))}^2+\\ {(G_p\left(P\right)-G_q\left(P\right))}^2+\\ {(B_p\left(P\right)-B_q\left(P\right))}^2\end{array}\right\}$

Wherein, Rp, Gp, Bp represent respectively the intensity level of the RGB three primary color components of pixel in the piece of source, and intensity value range is 0-255; Rq, Gq, Bq represent respectively the intensity level of the RGB three primary color components of pixel in object block, add alphabetical P in formula in the bracket of above-mentioned alphabetical back and represent that to do two poor pixels identical with relative position in object block at the source piece;

8) complete above-mentioned active computing with the error sum of squares SSD of the chromatic information of object block pixel after, the operation result information in kernel is write buffer, read by CPU;

9) CPU finds out the source piece of error sum of squares SSD minimum of the chromatic information of source piece and object block pixel, and this source piece is blocks and optimal matching blocks, gives object block with the rgb value of blocks and optimal matching blocks, namely completes the repairing of object block; Due to the repairing of object block image, its border will change, and border, degree of confidence, priority all should be upgraded, and treats the patch area border to upgrade frontier point degree of confidence, priority value therefore again upgrade;

10) CPU detects and whether to remain repairing area, has and carries out for the 3rd step, if finish in zone not repaired, picture is repaired and completed.