AU2020104270A4 - Thangka image inpainting method combining with domain knowledge - Google Patents
Thangka image inpainting method combining with domain knowledge Download PDFInfo
- Publication number
- AU2020104270A4 AU2020104270A4 AU2020104270A AU2020104270A AU2020104270A4 AU 2020104270 A4 AU2020104270 A4 AU 2020104270A4 AU 2020104270 A AU2020104270 A AU 2020104270A AU 2020104270 A AU2020104270 A AU 2020104270A AU 2020104270 A4 AU2020104270 A4 AU 2020104270A4
- Authority
- AU
- Australia
- Prior art keywords
- image
- damaged
- block
- thangka
- inpainting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
- 238000000034 method Methods 0.000 title claims abstract description 104
- 230000001939 inductive effect Effects 0.000 claims description 6
- 238000010422 painting Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 4
- 239000003814 drug Substances 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 description 13
- 238000007781 pre-processing Methods 0.000 description 8
- 230000000007 visual effect Effects 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000003786 synthesis reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007591 painting process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 230000037303 wrinkles Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Image Analysis (AREA)
- Processing Or Creating Images (AREA)
Abstract
A Thangka image inpainting method combining with domain knowledge, comprising: judging
a shape of a damaged region and a type of a neighbourhood to the damaged region;
determining whether the damaged region is in a linear shape or block shape and whether the
block shape is a key block or a unique key block; and for the unique key block, performing, in
combination with relevant domain knowledge, providing of different damaged block types
suitable for a Thangka image, semantic annotating, acquiring of an exemplar image from an
image database based on similarity retrieval, and inpainting.
-1/1
Iae to be Annotating
hinted Annotation
Preprocessin
Retrieving |Acquiring
Performing knowledge-based detecting mn
and segmenting on a damaged region combinatio Thangka
.0 n with the
annotation
information
Acquiring a damaged block
Thangka
Image
Judging a shape of the damaged ' '| Knowledge
block and a type of its neighboring '- base --
Combining
Y with the p
Block Unique knowledge Retrieving
,shane? Tc the
annotation
N information
A
Algorith Ipainting
algorithm
in lirarydiscriminator
F Painting result
Figure 1
Description
-1/1
Iae to be Annotating hinted Annotation
Preprocessin
Retrieving |Acquiring Performing knowledge-based detecting mn and segmenting on a damaged region combinatio Thangka .0 n with the annotation information Acquiring a damaged block Thangka
Image Judging a shape of the damaged ' '| Knowledge block and a type of its neighboring '- base --
Combining Y with the p Block Unique knowledge Retrieving ,shane? Tc the annotation N information
Algorith Ipainting algorithm in lirarydiscriminator
F Painting result
Figure 1
The present invention belongs to the technical field of image inpainting, and in particular, to a Thangka image inpainting method combining with domain knowledge.
Digital image inpainting technology is presently a hot spot in the field of computer vision, and an important application is in virtual restoration of cultural heritages and artworks.
A Thangka image inpainting method combining with domain knowledge is mainly intended for solving the problem with inpainting of a "key block" in a damaged region of an image. In recent years, the inventors have solved some problems in the aspect of digital image inpainting for damaged Thangka.
For the classifying problem of Thangka images and non-Thangka images in a large number of images, an authorized patent is obtained: Method for Distinguishing Thangka Images and Non-Thangka Images, with a patent number of ZL2011201110320662.3.
For the problem of using different inpainting algorithms for different damaged regions according to shapes of damaged blocks of damaged Thangka images, information of neighborhoods to the damaged blocks, characteristics of existing inpainting algorithms, such a method has been granted of an authorized patent: Thangka Image Inpainting Method based on Damaged block Shape and Neighborhood Classification, with a patent number of ZL201110320658.7. For a damaged block in a damaged image, if the damaged block is a "unique block" in this image, for example, a Shakyamuni Buddha Thangka with a left ear damaged, a method of symmetric exemplar blocks may be used to inpaint the left ear, where the left ear is a unique block in this image, this image also has a symmetric exemplar to this block, and a right ear is a horizontal-direction symmetric exemplar for the Buddha's head image which is uniformly symmetrical. The inventors provide an image inpainting method based on symmetric exemplar blocks, which has been granted of an authorized patent for invention: Image Inpainting Method based on Symmetric exemplar blocks, with a patent number of ZL201410498115.8
In addition, existing image inpainting methods are all oriented to specific application objects, and all possess certain applicability and limitations. In the actual research on damage inpainting for Thangka images, if a damaged region is "unique" in a whole image and has no above-mentioned symmetric exemplar, solving the inpainting problem of this damaged image has certain practical value. At present, no special research and literatures have been reported.
In summary, the prior art fails to inpaint a unique "key block" damaged in a Thangka image.
Any discussion of the prior art throughout the specification should in no way be considered as an admission that such prior art is widely known or forms part of common general knowledge in the field.
[0001] The present invention discloses a Thangka image inpainting method combining with domain knowledge. The method includes: judging a shape of a damaged region and a type of a neighborhood to the damaged region; determining whether the damaged region is in a linear shape or block shape and whether the block shape is a key block or a unique key block; and for the unique key block, performing, in combination with relevant domain knowledge, providing of different damaged block types suitable for a Thangka image, semantic annotating, acquiring of an exemplar image from an image database based on similarity retrieval, and inpainting. In accordance with characteristics of existing algorithms and characteristics of a specific image, the method of the present invention adopts good points of the various algorithms and avoid their shortcomings. Selecting of an inpainting algorithm needs to be based on analysis and modifying of the various algorithms, shape classifying of damaged blocks, and classifying of neighborhoods to the damaged blocks. The problem with inpainting images in various damage forms is solved.
It is an object of the present invention to overcome or ameliorate at least one of the disadvantages of the prior art, or to provide a useful alternative.
1. A Thangka image inpainting method combining with domain knowledge, wherein the Thangka image inpainting method combining with the domain knowledge comprises: judging a shape of a damaged region and a type of a neighborhood to the damaged region; determining whether the damaged region is in a linear shape or block shape and whether the block shape is a key block or a unique key block; and for the unique key block, performing, in combination with relevant domain knowledge, providing of different damaged block types suitable for a Thangka image, semantic annotating, acquiring of a exemplar image from an image database based on similarity retrieval, and inpainting.
2. The Thangka image inpainting method combining with domain knowledge according to claim 1, wherein a method of inducing and acquiring Thangka domain knowledge comprises:
performing classifying and knowledge acquiring according to a Thangka painting content, namely, dividing into Buddhist icons, historical figures, traditional Tibetan medicine, architecture, Tibetan astronomical calendars, and graphic patterns according to the Thangka content, each type including a plurality of subtypes; and
acquiring Thangka domain knowledge through image annotating, wherein an image annotation method is a semiautomatic annotation method, and the semiautomatic annotation method comprises: mapping semantic concepts as independent types to image objects through classifying; and acquiring a complete concept vocabulary which is subjected to annotating in combination with a thesaurus compiling and classifying method and a thesaurus permuting method.
3. The Thangka image inpainting method combining with domain knowledge according to claim 1, wherein a method of analyzing a damaged region and representing knowledge of the damaged region comprises:
determining shape and dimension types of a damaged block through information of a neighborhood; and classifying and representing visually perceived crack, crease and patch damage forms in a Thangka image exemplar, wherein classifying and representing parameters comprise areas, shapes and surrounding information, and the surrounding information adopts gray averages, standard deviations of gray, average gradients, standard deviations of gradients, and a number of gradient magnitude points within a certain range.
4. The Thangka image inpainting method combining with domain knowledge according to claim 1, wherein a method of inpainting a damaged block of a Thangka image comprises: inpainting the damaged block in combination with domain knowledge, and if a damaged region is a unique key block, searching an image database to find similar images so as to inpaint the damaged block according to the similar images, the inpainting the damaged block in combination with domain knowledge
specifically comprising:
I, performing a method of finding a key block from the image database, comprising: performing semantic annotating on the damaged block, using a semantics-based image retrieval method to inquire similar images in the annotated image database, selecting exemplars of unique key blocks from the found similar images, performing similarity comparing on surrounding information of the object exemplars in the similar images and surrounding information of the damaged block of a damaged image, and sorting the exemplar images again according to similarity degrees; and
II, filling the unique key block of the damaged image with a found exemplar.
5. The Thangka image inpainting method combining with domain knowledge according to claim 4, wherein the filling the unique key block of the damaged image comprises the following steps of:
blackening a non-key block region of a retrieved exemplar image to obtain an image represented by A; blackening a damaged unique key block segmented from an image to be inpainted, so as to obtain an image represented by B;
zooming the image A according to a size of the damaged block in the image B, so that a size of a key block in the image A is equal to the size of the damaged block in the image B, wherein a zoomed image is represented by C;
creating a black image, represented by D, having a same size as the image to be inpainted;
locating an exemplar block of the image C to the image D according to a position of the damaged block in the image B, so as to obtain a location image, represented by E, of the exemplar block; and
adding together the image B and the image E to obtain an inpainted resulting image.
FIG. 1 is a flowchart of a Thangka image inpainting method combining domain knowledge provided by an embodiment of the present invention.
Preferred embodiments of the invention will now be described with reference to the accompanying drawings and non-limiting examples.
In order to solve the problems in the prior art, the present invention provides a Thangka image inpainting method combining with domain knowledge. The method includes: judging a shape of a damaged region and a type of a neighborhood to the damaged region; determining whether the damaged region is in a linear shape or block shape and whether the block shape is a key block or a unique key block; and for the unique key block, performing, in combination with relevant domain knowledge,providing of different damaged block types suitable for a Thangka image, semantic annotating, acquiring of a exemplar image from an image database based on similarity retrieval, and inpainting.
Further, a method of inducing and acquiring Thangka domain knowledge includes:
acquiring Thangka domain knowledge through image annotating, where an image annotation method is a semiautomatic annotation method, and the semiautomatic annotation method includes: mapping a semantic concept as an independent type to an image object through classifying; and acquiring a complete concept vocabulary which is subjected to annotating in combination with a thesaurus compiling and classifying method and a thesaurus permuting method.
Further, a method of analyzing a damaged region and representing knowledge of the damaged region includes:
determining shape and dimension types of a damaged block through information of a neighborhood; and classifying and representing visually perceived crack, crease and patch damage forms in a Thangka image exemplar, where classifying and representing parameters include areas, shapes and surrounding information, and the surrounding information adopts gray averages, standard deviations of gray, average gradients, standard deviations of gradients, and a number of gradient magnitude points within a certain range.
Further, a method of segmenting a damaged region of an image includes: detecting and segmenting the damaged region; during image segmenting, applying statistically obtained priori knowledge to the detecting and segmenting of the damaged region by integrating color characteristics, a spatial position and image edges; and performing detecting and segmenting on a damaged block of a Thangka image by comprehensively applying a Mean-Shift non-parametric multi-model segmenting method and a CamShift method based on color information.
Further, a method of inpainting a damaged block of a Thangka image in combination with Thangka domain knowledge includes: inpainting the damaged block in combination
-'7
with domain knowledge, and if a damaged region is a unique block, searching an image database to find similar images so as to inpaint the damaged block according to the similar images; and specifically includes:
I, performing a method of finding a key block from the image database, including: performing semantic annotating on the damaged block, using a semantics-based image retrieval method to inquire similar images in the annotated image database, using a "neighborhood characteristic"and "neighborhood classification"method mentioned in the authorized patent: Thangka Image Inpainting Method based on Damaged Block Shape and Neighborhood Classification to select exemplars ofunique "key blocks" from the front five found similar images, performing similarity comparing on surrounding information of the exemplars and surrounding information of the damaged block of a damaged image, and sorting the exemplars; and
II, performing a method of filling the unique block of the damaged image with a found exemplar. The filling the damaged unique "key block"includes the steps of:
blackening a non-key block region of each retrieved exemplar image to obtain an image represented by A; blackening a damaged unique "key block" segmented from an image to be inpainted, so as to obtain an image represented by B;
zooming the image A according to a size of the damaged block in the image B, so that a size of a unique key block in the image A is equal to the size of the damaged block in the image B, wherein a zoomed image is represented by C;
creating a black image, represented by D, having a same size as the image to be inpainted;
locating an exemplar block of the image C to the image D according to a position of the damaged block in the image B, so as to obtain a location image, represented by E, of the exemplar block; and
adding together the image B and the image E to obtain an inpainted resulting image.
The Thangka image inpainting method combining with domain knowledge provided by the present invention mainly includes: after a unique "key block" in an image is damaged, performing semantic annotating on the damaged block, performing database similarity retrieval of images to find similar exemplar images, performing similarity comparing on surrounding information of the unique "key block" of the damaged image and surrounding information of key blocks of the exemplar images to obtain a most similar exemplar image, and filling the unique "key block" of the damaged image with the key block of the most similar exemplar image to complete inpainting. According to the prior art, image inpainting based on variational partial differential equation (PDE) can well inpaint small-scale damages, such as scratches, in pictures, but cannot well inpaint a picture having particularly rich texture of the surrounding of a damaged region; a block-based texture synthesis inpainting method is a global search method of matched blocks, which easily causes mistaken matching; and basically all existing inpainting methods are intended for inpainting simple graphics, such as photographs and natural images. However, Thangka images are complicated and have no obvious foreground or background, with varying damage types. In accordance with characteristics of existing algorithms and characteristics of a specific image, the method of the present invention adopts good points of the various algorithms and avoid their shortcomings. Selecting of an inpainting algorithm is based on analysis and modifying of the various algorithms, shape classifying of damaged blocks, and classifying of neighborhoods to the damaged blocks. Inpainting damage with appropriate algorithms is a valuable issue to solve.
To make the objectives, technical solutions, and advantages of the present invention more comprehensible, the following provides further description with reference to embodiments. The specific embodiments described herein are merely used to explain the present invention but are not intended to limit the present invention.
The following describes in detail the application principle of the present invention with reference to accompanying drawings.
As shown in FIG. 1, the Thangka image inpainting method combining domain knowledge provided by an embodiment of the present invention includes: judging a shape of a damaged region and a type of a neighborhood to the damaged region; determining whether the damaged region is in a linear shape or block shape and whether the block shape is a key block or a unique key block; and for the unique key block, performing, in combination with relevant domain knowledge, providing of different damaged block types suitable for a Thangka image, semantic annotating, acquiring of an exemplar image from an image database based on similarity retrieval, and inpainting.
The Thangka image inpainting method combining domain knowledge according to an embodiment of the present invention provides positive analysis on basis of analysis, inducing and organizing of a Thangka domain knowledge system, establishing of a Thangka image database, relevant image processing, artificial intelligence, and other theories.
A method of inpainting a damaged block of a Thangka image in combination with Thangka domain knowledge, provided by an embodiment of the present invention, includes:
1) preprocessing a damaged block of an image to be inpainted:
in combination with a specific Thangka image, using a relevant image processing method and a denoising, deblurring and cleaning method for cultural heritage-involved visual images to obtain preprocessing algorithms suitable for a Thangka image; and analyzing a removed damaged portion and scattered residues in the removed damaged portion;
2) inpainting non-block shaped and "key block" damaged regions:
Selecting corresponding algorithms, namely, No. 1 to 5 algorithms or models in Table 1, to inpaint a damaged region of a Thangka image if the damage region is of a non-texture linear type; selecting corresponding algorithms, namely, No. 6 to 8 algorithms or models in Table 1 to inpaint the damaged region if the damaged region is of a "non-key block"block type; and selecting No. 9 to 10 algorithms to inpaint the damaged region if the damaged region is of a "key block"type, by judging image symmetry according to a method in the patent: Method of Distinguishing Thangka Images and Non-Thangka Images, and judging through human-computer interaction;
Table 1 Inpainting Algorithms
Inpai Image good for nting inpainting No. Algorithm/Model Algorithm Evaluation Spee Type of Damaged d Region
This model employs third-order partial Bertalmio- Sapiro-Caselles- differential equation, has complicated Low Linear Non-texture solving, only simulates a manual Bellester (BSCB) inpainting process, and has no rigorous Model theoretical foundation.
This model employs second-order partial Total Variation Ordin differential equation, is easy to 2 Linear Non-texture (TV) Model ary understand, has simple Solving, but fail to satisfy the visual connectivity principle.
Relat This model is the same as the TV model, 3 Fast TV Model ively Linear Non-texture but at ahigher inpainting speed. High
Relat This model employs third-order partial Curvature Driven differential equation, has complicated 4 Diffusions ively Linear Non-texture solving, and satisfies the visual connectivity principle.
This model has high inpainting speed, but Oliveira Model High Linear Non-texture easily causes edge blur.
This algorithm is capable of inpainting Exemplar-based Mixed damage. Block structured and texture damages and is 6 Image Inpainting High shape. suitable for inpainting large-scale Algorithm damages.
Modified This algorithm is the same as the
Exemplar-based Mixed damage. Block exemplar-based image inpainting 7 High Inpainting shape. algorithm, but images inpainted have Algorithm smoother edges.
Exemplar-based
Texture Very Texture damage. This algorithm has a higher speed for 8 Synthesis High Block shape. inpainting texture damage regions. Algorithm
Symmetry in at least This method is an extension of the one of eight directions exemplar-based image inpainting Eight Directions Relat globally or locally. algorithm, the core of which is eight based Symmetric directional symmetric exemplars, the 9 Exemplar ively Filling a missing part method is special for any-directional High via information of a symmetric exemplars, and the method has Painting .amad.n a higher speed than other methods of damagedregionimna inpainting any-directional symmetric symmetric direction. exemplars.
Symmetry in any direction globally or This method is an extension of the Any Direction Relat locally. Filling a exemplar-based image inpainting based Symmetric algorithm, the core of which is any 10 ively missing part via directional symmetric exemplars, and the High information of a method may solve the problems which Inpainting . . the exemplar-based image inpainting damagedregioninany algorithm fails to solve. symmetric direction.
and 3) inpainting a damaged block in combination with domain knowledge:
if a damaged region is a unique "key block", searching an image database to find similar images so as to inpaint the damaged block according to the similar images, the inpainting damaged block in combination with domain knowledge specifically including:
I, performing a method of finding a key block from the image database, including: performing semantic annotating on the damaged block, using a semantics-based image retrieval method to inquire similar images in the annotated image database, using a "neighborhood characteristic"and "neighborhood classification"method mentioned in the authorized patent: Thangka Image Inpainting Method based on Damaged Block Shape and Neighborhood Classification to select exemplars ofunique "key blocks" from the front five found similar images, performing similarity comparing on surrounding information of the exemplars and surrounding information of the damaged block of a damaged image, and sorting the exemplars again according to similarity degrees; and
II, performing a method of filling the unique "key block" of the damaged image with a found exemplar. The filling the damaged unique "key block" includes the steps of:
blackening a non-key block region of each retrieved exemplar image to obtain an image represented by A; blackening a damaged unique "key block" segmented from an image to be inpainted, so as to obtain an image represented by B;
zooming the image A according to a size of the damaged block in the image B, so that a size of a key block in the image A is equal to the size of the damaged block in the image B, wherein a zoomed image is represented by C;
creating a black image, represented by D, having a same size as the image to be inpainted;
locating an exemplar block of the image C to the image D according to a position of the damaged block in the image B, so as to obtain a location image, represented by E, of the exemplar block; and
adding together the image B and the image E to obtain an inpainted resulting image.
The following further describes the application principle of the present invention with reference to specific analysis.
(1) Thangka image domain knowledge and analysis for its acquisition
A. Method of inducing and acquiring Thangka domain knowledge
Inpainting of a real Thangka requires a painter and even a professional restorer, and the Thangka to be inpainted needs to be verified comprehensively to study its theme artistic conception, painting year, style and school of painting, painting process, material and color, etc. Therefore, an inpainting solution adjusted to the Thangka is formulated according to the relics restoration principle: restoring the old as the old. It indicates that inpainting is inseparable from domain knowledge. However, in image segmentation and inpainting, there are two aspects of knowledge utilization. One is damaged block neighborhood information, i.e. prior knowledge. The other aspect refers to image database retrieval and application of domain knowledge, for example, for a missing mouth of a Buddha image, obviously surrounding information is not available, there are no similar objects in this image, it is an only solution to find other like Buddha images with same painting style, similar age and similar color from a library, and image knowledge base retrieval according to a certain classification standard is an effective way to obtain domain knowledge.
B. Damaged Region Analysis and Knowledge Representation thereof
There are many reasons for Thangka damage, and the mixed damage of different properties includes mildew, wrinkle, stain, fracture, etc. This is a difference of damage of a Thangka from that of murals. Accordingly, it is not practical to completely solve digital inpainting of Thangka images. By analyzing and summarizing damaged regions of actual exemplar images, it is concluded that different segmenting and inpainting methods should be used for different damage forms. According to the present invention, damaged region analysis and related statistics are carried out on an actual damaged image exemplar so as to obtain statistical knowledge.
(2) Image Damaged Region Segmentation Combining with Thangka Domain Knowledge
It is not easy to determine whether an image to be inpainted has damaged blocks or not and find the damaged blocks. Due to diversity of images and complexity of modes, it is difficult to accurately locate a damaged region. People often combine global information of an image and priori knowledge when determining a damaged region of the image, so that it is possible to segment the damaged region by combining the domain knowledge under existing technical conditions. In addition, an image inpainting algorithm mainly includes inpainting based on a texture structure and a non-texture structure, and classifying of a damaged block includes classifying of its neighborhood, namely, the surrounding of the damaged block is texture, non-texture or a mixture of the two, so that the damaged block and its type may be determined. At present, a gray-based image segmentation technology is mature, where a damaged region is usually found in a gray space. The damaged region is segmented through a region growing and threshold segmentation algorithm.
(3) Inpainting Analysis for Damaged Block of Thangka Image
A. Preprocessing of Damaged Block of Image to be Inpainted
The actual damaged block is complicated. In order to inpaint a damaged real Thangka, it is necessary to carry out necessary pretreating, such as wiping off of dust and scattered residue on a damaged portion. Preprocessing of a damaged region of the image to be inpainted also includes similar contents, such as removing of noise in the damaged region, and cleaning of damaged blocks. It is necessary to judge the noise, to clean the interior of the damaged blocks and to perform related processing.
B. Inpainting of Damaged Block in Combination with Domain Knowledge
Inpainting of the present invention includes macro inpainting and micro inpainting. The micro inpainting refers to methods of variational partial differential equation. These methods only rely on propagating and diffusing of information of the surrounding of a damaged region, and therefore, can judge region types only through experiential knowledge. Application of domain knowledge is typically embodied in the macro inpainting. For example, when a unique object in an image is lost, e.g., a mouth of Shakyamuni Buddha in a Thangka is lost, a result of inpainting using only its surrounding information is easy to expect. Therefore, the present invention provides: annotating a mouth of Shakyamuni Buddha in an image exemplar library, correspondingly annotating an image to be inpainted, and acquiring a corresponding block in the annotated image database for filling so as to reduce a damage degree or to completely inpaint the image. The application of domain knowledge is based on the intra-class similarity of a same theme or similar images, so that a corresponding filling block can be accurately found and applied for inpainting only by combining surrounding information of a damaged block and the domain knowledge. If an image to be inpainted is an only image in a classified image database, the damaged image cannot be inpainted.
(4) Automatic Selecting of Inpainting Algorithms in Accordance with Damaged Block Shape and Neighborhood Information
Image inpainting based on variational PDE can well inpaint small-scale damages, such as scratches, in pictures, but cannot well inpaint a picture having particularly rich texture of the surrounding of a damaged region; a block-based texture synthesis inpainting method is a global search method of matched blocks, which easily causes mistaken matching; and basically all existing inpainting methods are intended for inpainting simple graphics, such as photographs and natural images. However, Thangka images are complicated and have no obvious foreground or background, with varying damage types. It is fully possible for the method of the present invention to adopt good points of the various algorithms and avoid their shortcomings, in accordance with characteristics of existing algorithms and characteristics of a specific image. Selecting of an inpainting algorithm needs to be based on analysis and modifying of the various algorithms, shape classifying of damaged blocks, classifying of neighborhoods to the damaged blocks, etc. Automatic selecting of appropriate algorithms is a valuable issue to solve.
The following further describes the application principle of the present invention with reference to specific embodiments.
(1) Thangka image domain knowledge and analysis for its acquisition
A. Method of inducing and acquiring Thangka domain knowledge
Thangka images may be divided into Buddhist icons, historical figures, traditional Tibetan medicine, architecture, Tibetan astronomical calendars, and graphic patterns according to the Thangka image content. Each type may also be divided into a plurality of subtypes, e. g. religious icon subtypes, including dozens of subtypes of Buddha, Bodhisattva, Patriarch, etc. The other types are similar. Taking Buddha as an example, there are dozens of subtypes, such as Shakyamuni Buddha, Dipamkara Buddha, and MedicineBuddha. After such a "Thangka tree" with clear layering is constructed, various exemplars of a same Buddha become "leaves"to this tree. The exemplars of the same Buddha are annotated with a same policy. The Shakyamuni Buddhais annotated with headwear, Mudra, Hand Figure, Seat, Light Pattern, etc. To annotate different Buddhas, it is necessary to mine their deep knowledge to reflect the differences of the Buddhas. Image annotation methods mainly include manual annotation, semiautomatic annotation and automatic annotation. An object to be annotated in the present invention is the "leaves" in a hierarchical classification library. For example, when annotated, ShakyamuniBuddha performs "Bhumisparsha Mudra" with the right hand and "Dhyana-Mudra" with the left hand, and holds a "Patra". Such an annotation method used includes: mapping semantic concepts (Bhumisparsha Mudra, Dhyana-Mudra and Patra) as an independent type to image objects, such as a right hand and a left hand, through classifying. A knowledge base is formed through semantic annotating for Thangkas, which is knowledge acquiring, and this base is an image database for retrieval later.
B. Damaged Region Analysis and Knowledge Representation thereof
Image inpainting algorithms may include texture structure-based and non-texture structure-based methods, and inpainting results are related to a size and shape of a block to be inpainted. Firstly, a damaged region may be a damaged block, or may include a plurality of damaged blocks. Whether a damaged block is a texture block or a non-texture block is closely related to a neighborhood type, and a type of the damaged block is determined through information of a neighborhood. Defining of a shape and size of the damaged block is an analytical content in shape classifying of the damaged block. Visually perceived crack, crease and plaque damage forms and other forms in a Thangka image exemplar are classified and represented, where main parameters include areas, shapes, surrounding information (such as gray averages, standard deviations of gray, average gradients, standard deviations of gradients, a number of gradient magnitude points within a certain range), etc., so that these priori knowledges can be utilized in detecting, segmenting and inpainting various damaged blocks.
(2) Image Damaged Region Segmentation Combining with Thangka Domain Knowledge
This part includes damaged region detecting and segmenting. In a case of image segmenting, integration of color characteristics, a spatial position and the like is required to represent damage form types and knowledge so as to perform damaged image annotating and counting. A key is to train a classifier which distinguishes damaged regions and non damaged regions, or different damage forms. Mean-Shift is a non-parametric multi-model segmenting method, which achieves a segmenting purpose by analyzing a characteristic space of an image and by a clustering method. This segmenting method acquires a density mode of an unknown type by directly estimating a local maximal value of a probability density function of the characteristic space, and determines a position of this mode so as to cluster the mode to a type related to the mode.
(3) Inpainting Analysis for Damaged Block of Thangka Image
A. Preprocessing of Damaged Block of Image to be Inpainted
In a preprocessing phase, by using a relevant image processing method and denoising, deblurring, cleaning, etc. For cultural heritage-involved visual images, preprocessing algorithms suitable for a Thangka image is obtained through an experimental effect; removing of scattered residues in a damaged block is an important content of the preprocessing, for example, which points may be useful information and which points need to be removed.
B. Inpainting of Damaged Block in Combination with Domain Knowledge
If a damaged block is in linear shape (within a certain threshold range, a threshold being determined through experimenting), a type of the damaged block is determined by using experiential knowledge, so that inpainting is achieved through algorithms 1 to 6 in an algorithm library. If the damaged block is in block shape and is not a key block, inpainting may be achieved through algorithms 6 to 8 in the algorithm library similarly. If the damaged block is a key block and an image is symmetrical, inpainting is achieved through algorithm 9 or 10 in the algorithm library. If the damaged block is a unique key block, inpainting is to be achieved through similarity retrieval of an image database.
The foregoing descriptions are merely preferred embodiments of the present invention, but are not intended to limit the present invention. Any modification, equivalent replacement, or improvement made without departing from the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Although the invention has been described with reference to specific examples, it will be appreciated by those skilled in the art that the invention may be embodied in many other forms, in keeping with the broad principles and the spirit of the invention described herein.
The present invention and the described preferred embodiments specifically include at least one feature that is industrial applicable.
Claims (5)
1. A Thangka image inpainting method combining with domain knowledge, wherein the Thangka image inpainting method combining with the domain knowledge comprises: judging a shape of a damaged region and a type of a neighborhood to the damaged region; determining whether the damaged region is in a linear shape or block shape and whether the block shape is a key block or a unique key block; and for the unique key block, performing, in combination with relevant domain knowledge, providing of different damaged block types suitable for a Thangka image, semantic annotating, acquiring of a exemplar image from an image database based on similarity retrieval, and inpainting.
2. The Thangka image inpainting method combining with domain knowledge according to claim 1, wherein a method of inducing and acquiring Thangka domain knowledge comprises:
performing classifying and knowledge acquiring according to a Thangka painting content, namely, dividing into Buddhist icons, historical figures, traditional Tibetan medicine, architecture, Tibetan astronomical calendars, and graphic patterns according to the Thangka content, each type including a plurality of subtypes; and
acquiring Thangka domain knowledge through image annotating, wherein an image annotation method is a semiautomatic annotation method, and the semiautomatic annotation method comprises: mapping semantic concepts as independent types to image objects through classifying; and acquiring a complete concept vocabulary which is subjected to annotating in combination with a thesaurus compiling and classifying method and a thesaurus permuting method.
3. The Thangka image inpainting method combining with domain knowledge according to claim 1, wherein a method of analyzing a damaged region and representing knowledge of the damaged region comprises:
determining shape and dimension types of a damaged block through information of a neighborhood; and classifying and representing visually perceived crack, crease and patch damage forms in a Thangka image exemplar, wherein classifying and representing parameters comprise areas, shapes and surrounding information, and the surrounding information adopts gray averages, standard deviations of gray, average gradients, standard deviations of gradients, and a number of gradient magnitude points within a certain range.
4. The Thangka image inpainting method combining with domain knowledge according to claim 1, wherein a method of inpainting a damaged block of a Thangka image comprises: inpainting the damaged block in combination with domain knowledge, and if a damaged region is a unique key block, searching an image database to find similar images so as to inpaint the damaged block according to the similar images, the inpainting the damaged block in combination with domain knowledge
specifically comprising:
I, performing a method of finding a key block from the image database, comprising: performing semantic annotating on the damaged block, using a semantics-based image retrieval method to inquire similar images in the annotated image database, selecting exemplars of unique key blocks from the found similar images, performing similarity comparing on surrounding information of the object exemplars in the similar images and surrounding information of the damaged block of a damaged image, and sorting the exemplar images again according to similarity degrees; and
II, filling the unique key block of the damaged image with a found exemplar.
5. The Thangka image inpainting method combining with domain knowledge according to claim 4, wherein the filling the unique key block of the damaged image comprises the following steps of:
blackening a non-key block region of a retrieved exemplar image to obtain an image represented by A; blackening a damaged unique key block segmented from an image to be inpainted, so as to obtain an image represented by B;
zooming the image A according to a size of the damaged block in the image B, so that a size of a key block in the image A is equal to the size of the damaged block in the image B, wherein a zoomed image is represented by C;
creating a black image, represented by D, having a same size as the image to be inpainted; locating an exemplar block of the image C to the image D according to a position of the damaged block in the image B, so as to obtain a location image, represented by E, of the exemplar block; and adding together the image B and the image E to obtain an inpainted resulting image.
-1/1- 23 Dec 2020
Image to be Annotating inpainted Annotation 2020104270
Preprocessin
Retrieving Acquiring Performing knowledge-based detecting in and segmenting on a damaged region combinatio n with the Thangka annotation
Annotating information Acquiring a damaged block Thangka
Image Judging a shape of the damaged d t b Knowledge block and a type of its neighboring base Y Combining with the Y P knowledge Block Unique Retrieving base and shape? the “Key N annotation N information
A Inpainting algorithm Algorith discriminator m library
Inpainting result
Figure 1
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2020104270A AU2020104270A4 (en) | 2020-12-23 | 2020-12-23 | Thangka image inpainting method combining with domain knowledge |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2020104270A AU2020104270A4 (en) | 2020-12-23 | 2020-12-23 | Thangka image inpainting method combining with domain knowledge |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| AU2020104270A4 true AU2020104270A4 (en) | 2021-03-11 |
Family
ID=74853824
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| AU2020104270A Ceased AU2020104270A4 (en) | 2020-12-23 | 2020-12-23 | Thangka image inpainting method combining with domain knowledge |
Country Status (1)
| Country | Link |
|---|---|
| AU (1) | AU2020104270A4 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116363660A (en) * | 2023-04-10 | 2023-06-30 | 湖南三湘银行股份有限公司 | OCR (optical character recognition) method and server based on deblurring |
-
2020
- 2020-12-23 AU AU2020104270A patent/AU2020104270A4/en not_active Ceased
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116363660A (en) * | 2023-04-10 | 2023-06-30 | 湖南三湘银行股份有限公司 | OCR (optical character recognition) method and server based on deblurring |
| CN116363660B (en) * | 2023-04-10 | 2023-12-19 | 湖南三湘银行股份有限公司 | OCR (optical character recognition) method and server based on deblurring |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110543837B (en) | Visible light airport airplane detection method based on potential target point | |
| CN109743879B (en) | Underground pipe gallery leakage detection method based on dynamic infrared thermography processing | |
| Li et al. | Reconstructing building mass models from UAV images | |
| Ye et al. | Color image segmentation using density-based clustering | |
| EP1969562B1 (en) | Edge-guided morphological closing in segmentation of video sequences | |
| EP1969559B1 (en) | Contour finding in segmentation of video sequences | |
| Recky et al. | Windows detection using k-means in cie-lab color space | |
| Zhang et al. | Detecting and extracting the photo composites using planar homography and graph cut | |
| CN110738676A (en) | GrabCT automatic segmentation algorithm combined with RGBD data | |
| JP2008097607A (en) | Method to automatically classify input image | |
| Yu et al. | Artificial intelligence for Dunhuang cultural heritage protection: the project and the dataset | |
| CN109961016B (en) | Multi-gesture accurate segmentation method for smart home scene | |
| CN108376232A (en) | A kind of method and apparatus of automatic interpretation for remote sensing image | |
| CN109448009A (en) | Infrared Image Processing Method and device for transmission line faultlocating | |
| Lim et al. | Text segmentation in color images using tensor voting | |
| Li et al. | Feature-convinced mesh denoising | |
| AU2020104270A4 (en) | Thangka image inpainting method combining with domain knowledge | |
| Lam et al. | Highly accurate texture-based vehicle segmentation method | |
| Sapirstein | Segmentation, reconstruction, and visualization of ancient inscriptions in 2.5 D | |
| Xu et al. | Based on improved edge detection algorithm for English text extraction and restoration from color images | |
| Xia et al. | A shadow detection method for remote sensing images using affinity propagation algorithm | |
| Dimiccoli et al. | Hierarchical region-based representation for segmentation and filtering with depth in single images | |
| Várkonyi-Kóczy | New advances in digital image processing | |
| Revathi et al. | An emerging trend of feature extraction method in video processing | |
| Yan et al. | A boosted decision tree approach to shadow detection in scanning electron microscope (SEM) images for machine vision applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FGI | Letters patent sealed or granted (innovation patent) | ||
| MK22 | Patent ceased section 143a(d), or expired - non payment of renewal fee or expiry |