CN106097268B - The remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure - Google Patents
The remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure Download PDFInfo
- Publication number
- CN106097268B CN106097268B CN201610409424.2A CN201610409424A CN106097268B CN 106097268 B CN106097268 B CN 106097268B CN 201610409424 A CN201610409424 A CN 201610409424A CN 106097268 B CN106097268 B CN 106097268B
- Authority
- CN
- China
- Prior art keywords
- picture
- silk
- tujia
- weaving
- texture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009941 weaving Methods 0.000 title claims abstract description 103
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000008439 repair process Effects 0.000 claims abstract description 43
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 20
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 20
- 238000013461 design Methods 0.000 claims abstract description 18
- 230000015556 catabolic process Effects 0.000 claims abstract description 11
- 238000012937 correction Methods 0.000 claims abstract description 7
- 238000001514 detection method Methods 0.000 claims abstract description 7
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 4
- 238000005259 measurement Methods 0.000 claims description 16
- 239000011159 matrix material Substances 0.000 claims description 15
- 238000009792 diffusion process Methods 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 239000000284 extract Substances 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 238000002156 mixing Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 14
- 238000005516 engineering process Methods 0.000 abstract description 13
- 238000013459 approach Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 11
- 238000011160 research Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 238000000354 decomposition reaction Methods 0.000 description 8
- 230000008859 change Effects 0.000 description 7
- 238000011478 gradient descent method Methods 0.000 description 7
- 239000002689 soil Substances 0.000 description 7
- 238000009826 distribution Methods 0.000 description 6
- 238000005457 optimization Methods 0.000 description 6
- 239000004753 textile Substances 0.000 description 6
- 239000013598 vector Substances 0.000 description 6
- 230000011218 segmentation Effects 0.000 description 5
- 230000009514 concussion Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000003709 image segmentation Methods 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 238000012913 prioritisation Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000001174 ascending effect Effects 0.000 description 2
- 238000004040 coloring Methods 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 150000001875 compounds Chemical class 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 235000013399 edible fruits Nutrition 0.000 description 2
- 239000004744 fabric Substances 0.000 description 2
- 239000004615 ingredient Substances 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000005381 potential energy Methods 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000005067 remediation Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 239000012141 concentrate Substances 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 210000003462 vein Anatomy 0.000 description 1
Classifications
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
Abstract
This application discloses the remaining patterns of " Tujia " picture weaving in silk tradition to digitize restorative procedure, including damaged area is detected automatically, and picture structure component and texture component are decomposed, and the textures synthesis reparation of the reparation of structure components Variational PDE and texture component specifically includes following steps：Establish automatic detection and the location model of the digital picture damaged area of " Tujia " picture weaving in silk；It is structure components and texture component that variation picture breakdown model, which is constructed, by " Tujia " picture weaving in silk picture breakdown；It designs Variational PDE model and repairs " Tujia " picture weaving in silk structure components, design the Texture Synthesis based on sample and repair " Tujia " picture weaving in silk texture component；The structure components and texture component being finally synthesizing after repairing, obtain the complete correction of " Tujia " picture weaving in silk digital picture.The application's repairs the remaining pattern of " Tujia " picture weaving in silk tradition by digital technology, can find most satisfied repairing effect by repeatedly attempting, while not needing to destroy original picture-weaving in silk, provide the approach of safe and convenient for the repair of " Tujia " picture weaving in silk.
Description
Technical field
The application belongs to digital picture and repairs field, specifically, being related to a kind of remaining pattern number of " Tujia " picture weaving in silk tradition
Change restorative procedure.
Background technique
" Tujia " picture weaving in silk is a wonderful work of Tujia's national handicraft, is referred to as " flower of Tujia "." Tujia " picture weaving in silk has long
History,《Later Han Dynasty's book southwest is quite smooth to be passed》In just have initial record to this picture-weaving in silk of Tujia." Tujia " picture weaving in silk is at it
In Emergence and Development and transition, the perfect combination of practicability and artistry is fully demonstrated, and become succession Tujia Culture and art
Important carrier.In June, 2006 " Tujia " picture weaving in silk be listed in national first batch of national non-material cultural heritage and lay special stress on protecting register.
But it now with the quickening of modernization, is influenced by many factors such as natural, society and humanistic environments, soil
The protection and succession of family's picture-weaving in silk are also faced with formidable challenges, and especially some traditional " Tujia " picture weaving in silk patterns just fade away.And it is proper
These proper tradition designs reflect the history culture of Tujia, the track of national origin are hidden, so being highly desirable to it
It is protected.
Local government has also specially set up the special class of protection, has formulated protective policy, and panel of expert is arranged to go deep into civil receipts
Collection arranges raw data, makes great efforts to protect this " Tujia " picture weaving in silk.Since the " Tujia " picture weaving in silk epoch in civil collection are remote, and passed through
It uses to perseverance, so usually will appear damaged and stained.It is carried out using complicated manual mode now with picture-weaving in silk artist
It repairs, but since repair is heavy, and local experienced picture-weaving in silk artist number is again limited, so this remediation efficiency
It is extremely low.Besides this direct repair is modified on original picture-weaving in silk, and this modification is usually irreversible, one
A slight fault may destroy precious original picture-weaving in silk.And Digital Image Inpainting then brings great freedom, I
Visually the most satisfied repairing effect can be searched out by multiple trial, without destroying original picture-weaving in silk.Institute
With in this case, Digital Image Inpainting can provide the approach of safe and convenient for the repair of " Tujia " picture weaving in silk.
Image repair, which refers to, carries out reconstruction to the image being damaged.Image repair person needs to take most appropriate side
Method restores the reset condition of image, while guaranteeing that image reaches optimal artistic effect.Early in the Renaissance, people are just
Start to repair some medieval arts work, its object is to make picture recovery original appearance by filling up some cracks, this work
Make just to be referred to as " Inpainting " (repairing, retouching) or " Retouching ".But repairing at that time is mainly that the art work is repaired
Multiple expert carries out by hand by professional knowledge, and this repair mode has the disadvantage that：(1) remediation efficiency is extremely low；(2) have
The art work of original preciousness may be destroyed.M.Bertalmio, which is put forward for the first time image repair, can be reduced to a mathematical expression
Formula is realized automatically using computer capacity.Image repair has been that one in computer graphics and computer vision grinds
Study carefully hot spot, (deletes groups of people in such as video image in historical relic's protection, ideo display stunt production, virtual reality, the rejecting of extra object
Object, text, subhead etc.) etc. have great application value.
Bertalmio etc. has been put forward for the first time digital picture and has repaired this art in the 27th SIGGRAPH meeting in 2000
Language, subsequent Digital Image Inpainting have obtained extensive research.Currently, the development of Digital Image Inpainting is concentrated mainly on
Two fields：(1) image repair based on non-grain structure is mainly used for repairing the digital picture of small scale breakage, grind at present
The person of studying carefully mostly uses to be repaired based on variation and partial differential equation of higher order (PDE) model；(2) based on the image mending of texture structure
Technology, is mainly used for filling the information of big lost block in image, and current research is concentrated mainly on based on Markov random field
(MRF) sample texture synthetic technology.
Digital Image Inpainting studies it and mainly still concentrates on theoretic in the development of this more than ten years.?
Application aspect also has a small amount of research achievement to be applied to the fields such as medicine, archaeology, but in the reparation side of fabric digital picture
Face, research achievement are seldom.Only a small amount of research is also to be directed to the very high fabric of some economic values, such as Tangka.Soil
Family's picture-weaving in silk is regional due to having part, and minority is compared in consumption, concerned degree be not it is very high, so being digitized to it
The research of reparation lies substantially in blank.But " Tujia " picture weaving in silk is as first batch of national non-material cultural heritage, in order to protect and
Picture-weaving in silk specific to this Tujia is passed on, is very important to its research in terms of carrying out Digital repair.
" Tujia " picture weaving in silk had not only included a large amount of contour structure, but also contained color and vein abundant, so single reparation skill
The repairing effect that art has been extremely difficult to, the restorative procedure based on picture breakdown can solve this problem.Based on picture breakdown
Repairing is exactly that complex pattern to be repaired is broken down into the sum of structure components and texture component using certain technology, then right respectively
Two kinds of components are individually repaired, and are finally synthesizing to obtain and are repaired result.Bertalmio etc. took the lead in 2003 by picture breakdown
It is applied to after image repair, some researchers study this recovery technique successively.But current this respect
Research achievement is in contrast still fewer, and few quantifier elimination is also the improvement aspect for concentrating on theoretical model and algorithm, rarely has
Certain digital picture is targetedly studied using this technology.
In conclusion having carried out a certain amount of research work to image repair both at home and abroad at present, but it is based on Variational Decomposition
Repairing research it is less, for " Tujia " picture weaving in silk Digital repair research lie substantially in blank.
Summary of the invention
In order to solve the above-mentioned technical problem, this application discloses a kind of " Tujia " picture weaving in silk tradition remaining pattern Digital repair sides
Method, including damaged area are detected automatically, and picture structure component and texture component are decomposed, the reparation of structure components Variational PDE and texture
The textures synthesis reparation of component, specifically includes following steps：
(1) automatic detection and the location model of the digital picture damaged area of " Tujia " picture weaving in silk are established；
(2) " Tujia " picture weaving in silk picture breakdown is structure components and texture component by building variation picture breakdown model；
(3) design Variational PDE model repairs " Tujia " picture weaving in silk structure components, designs the Texture Synthesis reparation based on sample
" Tujia " picture weaving in silk texture component；
(4) structure components and texture component being finally synthesizing after repairing, obtain the complete correction of " Tujia " picture weaving in silk digital picture.
Further, step (1) specific method is：" Tujia " picture weaving in silk image is analyzed, color appropriate is selected
Model extracts color characteristic and textural characteristics, and carries out effective integration to these characteristics, obtains Efficient image characteristic
According to；The Efficient image characteristic information that fusion obtains is dissolved into variation geometric active contour model, image Segmentation Technology is passed through
Realize that " Tujia " picture weaving in silk image breakage target area automatically extracts and positions.
It selects 2 color model to be combined into composite coloured channel, passes through integral image colouring information and line in this channel
Feature construction compound variation level set model is managed, minimum using energy functional realizes the accurate extraction to image damaged area
And positioning.Euler-Lagrange equation and gradient descent method in Theory of Variational Principles is used to solve functional in numerical value calculating minimum.
It is implemented as follows：
Choose HSI color space and CIE LAB color space.Rgb color space is transformed into HSI color space first,
It converts as follows:
Rgb color space is converted into Lab color space again, formula is as follows
Wherein
Since " Tujia " picture weaving in silk is textile, there is very strong consistency texture, the feature of texture is considered in cutting procedure, is broken
Damage region does not have texture, other complete areas have essentially identical consistency texture.Textural characteristicsIt is total using area grayscale
Raw matrix extracts.Gray level co-occurrence matrixes are a kind of effective ways for analyzing textural characteristics, and this method has studied grey in image texture
Spend the space dependence of grade.It to the distribution character of gray scale is indicated by the distribution of the pixel different to gray value,
These pixels also embody spatial relation and distribution character simultaneously.The main process of texture feature extraction is：(1)
Image " Tujia " picture weaving in silk image is subjected to re-quantization, changes to 16 grades by original 256 grades；(2) ash on four direction is constructed
Spend co-occurrence matrix, this four direction is horizontal, vertical, diagonal line, back-diagonal respectively, it is represented mathematically as 0 °, 45 °, 90 °,
135°；(3) statistic (energy, entropy, the moment of inertia, correlative) that can characterize picture material is extracted from this matrix as texture
Feature
The variation movable contour model based on region is established respectively in HSI color space and LAB color space, and is incorporated
The textural characteristics of imageThen the segmentation result in two different colours spaces is subjected to region merging technique, detects final soil
Family's picture-weaving in silk image damaged area.Variation movable contour model is：
Wherein U ∈ { X, Y, Z, L, a, b },For 4 statistics of area grayscale co-occurrence matrix, i.e. energy, entropy, inertia
Square, correlative.F (φ, c are calculated using alternative iteration method1,c2,c3,c4) minimum point：
Using in Theory of Variational Principles Euler-Lagrange equation and gradient descent method solve functional about φ minimum point：
Above equation is solved using finite difference.I.e.
The segmentation result of 6 Color Channels in two color model is respectively φi=0, i=1,2 ... 6；φi=0
The edge actually divided.Segmentation result in two different colours models is subjected to region merging technique, detects final soil
Family's picture-weaving in silk image damaged area is：
Ω={ (x, y):φi< 0, i=1,2 ... 6. }
Further, using priori knowledge, the structure components to " Tujia " picture weaving in silk image and texture divide the step (2) respectively
Amount is modeled, and Variation Model is obtained, and is obtained clean structural texture by functional minimization and is decomposed, specific method is：Structure
Component is modeled by non-convex biregular item, the non-convex sparse measurement comprising gradient and the non-convex sparse measurement of second dervative；Texture point
Amount is measured using rank of matrix, extracts consistency texture by the minimization of order；Variation Model is solved using alternative iteration method.
The Variational Decomposition model of foundation is as follows：
WhereinIt is non-convex biregular item, for measuring " Tujia " picture weaving in silk
The structure components of digital picture；||ρv||*It is concussion measurement, for extracting the texture component of blue Kapp digital picture.Due to soil
Family's picture-weaving in silk is textile, and texture has very strong consistency, so (being substantially order measurement rank (ρ v) using nuclear norm
Minimum Convex Closure network) measurement concussion.For potential-energy function, select non-convex Non-smooth surface function to preferably keep the side in structure components
Edge information, is selected as：
With
Variational Decomposition model is solved using alternative iteration method：
Fixed v, u2, u is solved by the following Variation Model of minimization1
This is very famous ROF model, is solved using single order predual algorithm；
Fixed v, u1, u is solved by the following Variation Model of minimization2
With in Theory of Variational Principles Euler-Lagrange equation and gradient descent method solve；
Fixed u1, u2, u is solved by the following Variation Model of minimization2
This optimization problem is solved using iteration soft-threshold algorithm, nuclear norm therein is using matrix singular value decomposition
Method.
Above-mentioned 3 optimization problems are iteratively solved, optimal solution u is obtained1, u2, v, then the structure of " Tujia " picture weaving in silk digital picture is divided
Amount is expressed as u=u1+u2；Texture component is expressed as v.
Further, step (3) specific method is：The " Tujia " picture weaving in silk structure components and texture that step (2) is obtained
Component is repaired respectively.Structure components reparation uses Variational PDE model；Texture component reparation uses Future Opportunities of Texture Synthesis.
The Variational PDE model of structure components reparation is to combine fractional order differential with tensor diffusion, general according to fractional order
The corresponding Euler-Lagrange equation of the mentioned Variation Model of letter theory deduction, and during Numerical Implementation, utilization is discrete
Fourier transform definition Fractional Derivative operator and its adjoint operator, derive the calculation formula of Fractional Derivative, design and mentioned
The numerical algorithm of repairing model.Specific Variational PDE repairing model design is as follows：
Wherein WithIt is α rank score of the u in the direction x and y respectively
Order derivative.u0It is the structure components of " Tujia " picture weaving in silk digital picture (second step is obtained using variation picture breakdown)；D is " Tujia " picture weaving in silk
The damaged area of digital picture (second step is divided to obtain using variation geometric active contour model).With in Theory of Variational Principles
Euler-Lagrange equation and gradient descent method solve this optimization problem：
In above formulaWithIt is respectivelyWithAdjoint operator.For the marginal information for further repairing image, with
Tensor diffusion is introduced in upper diffusion equation, i.e.,
T (x) is diffusion tensor, is calculated with the following method：The structure tensor of definition measurement Local Structure of Image
GρIt indicates using ρ as the Gaussian kernel of parameter.Definition
JρTwo characteristic values be
Their corresponding feature vectors are v1And v2, vi=(cos θi,sinθi), i=1,2.
Wherein
If μ1And μ2It is two characteristic values of diffusion tensor matrices T (x), if
v1And v2It is corresponding feature vector, there is v1=(cos θ, sin θ)；v2=(- sin θ, cos θ).
The matrix element of T (x) and the relationship of eigen vector are as follows：
Using edge enhanced diffustion tensor：μ1=g (λ1), μ2=1；Wherein g is edge function.
Above-mentioned PDE is solved using finite difference：
Fractional Derivative can be by being calculated using efficient Discrete Fourier Transform：
Integer order derivative is generalized to Fractional Derivative, obtains converting the fractional order difference under meaning based on discrete FouierWithThey are in the corresponding relationship of airspace and frequency domain：
Fractional order difference operatorWithAdjoint operatorWithAirspace and frequency domain corresponding relationship
For：
Texture component is repaired using sample texture synthetic technology, it is adaptive as standard using the complexity of image block
Change region of search with answering to improve reparation speed；It is determined using the complexity of image block and repairs order to obtain preferable reparation
Effect.Detailed process is as follows：Using the complexity of fractal dimension and comentropy measurement image block, the field of search is determined using complexity
Domain and reparation order.The Weighted Threshold of empirically determined entropy and fractal dimension, be greater than threshold value multiblock to be repaired at, selection compared with
Complete matching padding in big region of search；It is then opposite at the multiblock to be repaired for being less than threshold value.And the big image of complexity
Block is preferentially repaired.In numerical value calculating, figure is calculated in conjunction with difference box-covering method and fractal Brown motion self-similarity method
As the fractal dimension of block, utmostly to distinguish different roughness texture.
The complexity of block of pixels is measured using comentropy.Comentropy is a kind of statistical form of feature, it reflects figure
As in average information number.The comentropy of image indicates the information content that the aggregation characteristic of intensity profile in image is included,
Enable PiIndicate block of pixels ΨpMiddle gray value is ratio shared by the pixel of i, i.e.,
The unitary comentropy for then defining gray level image is：
Wherein b is normalized parameter, selects b=5 in an experiment.Above formula only defines a metamessage of gray level image
Entropy, for color image, using the mean value of the unitary comentropy under tri- Color Channels of RGB, i.e.,
In the image mending for carrying out the textures synthesis based on sample block, comentropy H (p) big repairing block will be repaired first
It is multiple.
Block of pixels Ψ is further measured using fractal boxpComplexity.The fractal box of image is a kind of spy
The statistical form of sign reflects the number of average information in block of pixels, and value is normally between the 2-3 of section.Fractal dimension
Closer to dimension 2, show more flat (under normal conditions, the constant value block of pixels Ψ of imagep2) fractal box of=C is；It is closer
Dimension 3 shows that grey scale change is more violent, and image is more complicated.The unitary fractal box for then defining gray level image is：
F (p)=aD (Ψp)-b
Wherein a, b are normalized parameter, select a=1 in an experiment；B=-2.It is apparent that F (p) ∈ [0,1].If picture
Intensity profile is more uniform in plain block, and fractal box F (p) is closer to 0；, whereas if grey scale change is more violent in block of pixels,
Containing many image informations, fractal box F (p) is closer to 1.For color image, using under tri- Color Channels of RGB
Unitary fractal box mean value, i.e.,
The fractal box of block of pixels and comentropy are integrated in prioritization functions, the texture based on sample block is being carried out
When the image repair of synthesis, fractal box and the big reparation block of comentropy will be repaired first.
Prioritization functions are defined as：
P (p)=α C (p) D (p)+β F (p)+γ H (p)
Wherein α > 0, β > 0 and γ > 0 is weight factor, and meets alpha+beta+γ=1.
It matches block search and uses following process：By match block ΨqBy complexity, (fractal box adds set with comentropy
Power, i.e. β F (p)+γ H (p)) size imparting ascending order structure, binary search is then used, sequence match block Ψ is being assignedqIt is right in set
Under complexity meaning with input multiblock Ψ to be repairedpImmediate match block ΨqK neighborhood in carry out matching search again, find
The smallest match block of SSD.I.e.
Ψq=argmin { d (Ψp,Ψq):Ψq∈K}
Wherein K is match block ΨqIn set under fractal box meaning with input multiblock Ψ to be repairedpImmediate
With block ΨqK neighborhood.The size of k is selected as k=32.
Further, the structure components after step (4) the synthesis reparation and the specific method of texture component are：To reparation
Structure components and texture component afterwards are weighted and averaged.
If the structure components and texture component after repairing are respectively u and v, then complete correction is expressed as
F=2 × (a × u+ (1-a) × v)
Wherein a ∈ (0,1) is weight factor.Adjust the available different visual effect of this parameter.Ordinary circumstance is divided into
Determine a=0.5.
Compared with prior art, the application can be obtained including following technical effect：
1) the application by digital technology repairs the remaining pattern of " Tujia " picture weaving in silk tradition, can be by repeatedly tasting
Examination, finds most satisfied repairing effect, while not needing to destroy original picture-weaving in silk, provides peace for the repair of " Tujia " picture weaving in silk
Complete convenient and fast approach.
2) the present processes have for the texture-rich such as " Tujia " picture weaving in silk, beautiful in colour and textile repairs effect well
Fruit can meet the vision requirement of people well.
3) " Tujia " picture weaving in silk digital picture damaged area it is automatic detection and positioning, by with the manually operated optimal knot of expert
Fruit compares, and accuracy is up to 98% or more.
4) the present processes have faster reparation speed, the complex pattern to be repaired of 1024 × 1024 or less sizes, repairing
Region is 50 × 50 hereinafter, can about complete entire repair process in 100 seconds to 600 seconds.
5) technical solution of the application, step is simple, easily operated, reproducible, and technical staff is easy to learn, easy grasp.
Certainly, any product for implementing the application must be not necessarily required to reach all the above technical effect simultaneously.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present application, constitutes part of this application, this Shen
Illustrative embodiments and their description please are not constituted an undue limitation on the present application for explaining the application.In the accompanying drawings：
Fig. 1 is damaged " Tujia " picture weaving in silk image in the embodiment of the present application；
Fig. 2 is damaged " Tujia " picture weaving in silk image segmentation result figure in the embodiment of the present application；
Fig. 3 is damaged " Tujia " picture weaving in silk image damaged area figure in the embodiment of the present application；
Fig. 4 is damaged " Tujia " picture weaving in silk image repair exposure mask figure in the embodiment of the present application；
Fig. 5 is damaged " Tujia " picture weaving in silk picture structure component map in the embodiment of the present application；
Fig. 6 is damaged " Tujia " picture weaving in silk image texture component map in the embodiment of the present application；
Fig. 7 is the reparation result figure of damaged " Tujia " picture weaving in silk image breakage structure components in the embodiment of the present application；
Fig. 8 is the reparation result figure of damaged " Tujia " picture weaving in silk image breakage texture component in the embodiment of the present application；
Fig. 9 is that damaged " Tujia " picture weaving in silk image finally repairs result figure in the embodiment of the present application.
Specific embodiment
Presently filed embodiment is described in detail below in conjunction with accompanying drawings and embodiments, thereby how the application is applied
Technological means solves technical problem and reaches the realization process of technical effect to fully understand and implement.
The remaining pattern of embodiment " Tujia " picture weaving in silk tradition digitizes restorative procedure
(1) automatic detection and the location model of the digital picture damaged area of " Tujia " picture weaving in silk are established
As shown in Figs 1-4, Fig. 1 is the local screenshot of a width breakage " Tujia " picture weaving in silk, and wherein white area is damaged area.Figure
2 be the damaged " Tujia " picture weaving in silk image segmentation result figure being split using variation geometric active contour to damaged area.Fig. 3
For damaged " Tujia " picture weaving in silk image damaged area figure, the interior zone of closed curve is damaged area.Damaged area gray value is determined
Justice is 1, other regions are defined as 0, obtains repairing exposure mask, as shown in Figure 4.
" Tujia " picture weaving in silk image is analyzed, color model appropriate is selected, extracts color characteristic and textural characteristics, and right
These characteristics carry out effective integration, obtain Efficient image characteristic；The Efficient image characteristic information that fusion obtains is melted
Enter into variation geometric active contour model, realizes that " Tujia " picture weaving in silk image breakage target area mentions automatically by image Segmentation Technology
It takes and positions.
It selects 2 color model to be combined into composite coloured channel, passes through integral image colouring information and line in this channel
Feature construction compound variation level set model is managed, minimum using energy functional realizes the accurate extraction to image damaged area
And positioning.
Choose HSI color space and CIE LAB color space.Rgb color space is transformed into HSI color space first,
It converts as follows:
Rgb color space is converted into Lab color space again, formula is as follows
Wherein
Since " Tujia " picture weaving in silk is textile, there is very strong consistency texture, the feature of texture is considered in cutting procedure, is broken
Damage region does not have texture, other complete areas have essentially identical consistency texture.Textural characteristicsIt is total using area grayscale
Raw matrix extracts.Gray level co-occurrence matrixes are a kind of effective ways for analyzing textural characteristics, and this method has studied grey in image texture
Spend the space dependence of grade.It to the distribution character of gray scale is indicated by the distribution of the pixel different to gray value,
These pixels also embody spatial relation and distribution character simultaneously.The main process of texture feature extraction is：(1)
Image " Tujia " picture weaving in silk image is subjected to re-quantization, changes to 16 grades by original 256 grades；(2) ash on four direction is constructed
Spend co-occurrence matrix, this four direction is horizontal, vertical, diagonal line, back-diagonal respectively, it is represented mathematically as 0 °, 45 °, 90 °,
135°；(3) statistic (energy, entropy, the moment of inertia, correlative) that can characterize picture material is extracted from this matrix as texture
Feature
The variation movable contour model based on region is established respectively in HSI color space and LAB color space
Incorporate the textural characteristics of imageWherein U ∈ { X, Y, Z, L, a, b },For 4 systems of area grayscale co-occurrence matrix
Metering, i.e. energy, entropy, the moment of inertia, correlative.F (φ, c are calculated using alternative iteration method1,c2,c3,c4) minimum point：
Using in Theory of Variational Principles Euler-Lagrange equation and gradient descent method solve functional about φ minimum point：
Above equation is solved using finite difference.I.e.
The segmentation result of 6 Color Channels in two color model is respectively φi=0, i=1,2 ... 6；φi=0
The edge actually divided.Segmentation result in two different colours models is subjected to region merging technique, detects final soil
Family's picture-weaving in silk image damaged area is：
Ω={ (x, y):φi< 0, i=1,2 ... 6. }
(2) " Tujia " picture weaving in silk digital picture is decomposed by structure components and texture component using Variational Decomposition model
As seen in figs. 5-6, structure components are modeled using non-convex biregular, Tujia is obtained by variation minimization and is knitted
Bright and beautiful picture structure component；Texture component is modeled using nuclear norm, " Tujia " picture weaving in silk image line is obtained by variation minimization
Manage component.
The structure components and texture component of " Tujia " picture weaving in silk image are modeled respectively using priori knowledge, obtain variation mould
Type obtains clean structural texture by functional minimization and decomposes, and specific method is：Structure components are built by non-convex biregular item
Mould, the non-convex sparse measurement comprising gradient and the non-convex sparse measurement of second dervative；Texture component is measured using rank of matrix, passes through order
Minimization extract consistency texture；Variation Model is solved using alternative iteration method.
The Variational Decomposition model of foundation is as follows：
WhereinIt is non-convex biregular item, for measuring " Tujia " picture weaving in silk
The structure components of digital picture；||ρv||*It is concussion measurement, for extracting the texture component of blue Kapp digital picture.Due to soil
Family's picture-weaving in silk is textile, and texture has very strong consistency, so (being substantially order measurement rank (ρ v) using nuclear norm
Minimum Convex Closure network) measurement concussion.For potential-energy function, select non-convex Non-smooth surface function to preferably keep the side in structure components
Edge information, is selected as：
With
Variational Decomposition model is solved using alternative iteration method：
Fixed v, u2, u is solved by the following Variation Model of minimization1
This is very famous ROF model, is solved using single order predual algorithm；
Fixed v, u1, u is solved by the following Variation Model of minimization2
With in Theory of Variational Principles Euler-Lagrange equation and gradient descent method solve；
Fixed u1, u2, u is solved by the following Variation Model of minimization2
This optimization problem is solved using iteration soft-threshold algorithm, nuclear norm therein is using matrix singular value decomposition
Method.
Above-mentioned 3 optimization problems are iteratively solved, optimal solution u is obtained1, u2, v, then the structure of " Tujia " picture weaving in silk digital picture is divided
Amount is expressed as u=u1+u2；Texture component is expressed as v.
(3) design Variational PDE model repairs " Tujia " picture weaving in silk structure components, designs the Texture Synthesis reparation based on sample
" Tujia " picture weaving in silk texture component
As Figure 7-8, structure components are repaired using Variational PDE model；Using the textures synthesis based on sample
Algorithm repairs texture component.
The " Tujia " picture weaving in silk structure components and texture component obtain to step (2) are repaired respectively.Structure components reparation is adopted
With Variational PDE model；Texture component reparation uses Future Opportunities of Texture Synthesis.
The Variational PDE model of structure components reparation is to combine fractional order differential with tensor diffusion, general according to fractional order
The corresponding Euler-Lagrange equation of the mentioned Variation Model of letter theory deduction, and during Numerical Implementation, utilization is discrete
Fourier transform definition Fractional Derivative operator and its adjoint operator, derive the calculation formula of Fractional Derivative, design and mentioned
The numerical algorithm of repairing model.Specific Variational PDE repairing model design is as follows：
Wherein WithIt is α rank score of the u in the direction x and y respectively
Order derivative.u0It is the structure components of " Tujia " picture weaving in silk digital picture (second step is obtained using variation picture breakdown)；D is " Tujia " picture weaving in silk
The damaged area of digital picture (second step is divided to obtain using variation geometric active contour model).With in Theory of Variational Principles
Euler-Lagrange equation and gradient descent method solve this optimization problem：
In above formulaWithIt is respectivelyWithAdjoint operator.For the marginal information for further repairing image, with
Tensor diffusion is introduced in upper diffusion equation, i.e.,
T (x) is diffusion tensor, is calculated with the following method：The structure tensor of definition measurement Local Structure of Image
GρIt indicates using ρ as the Gaussian kernel of parameter.Definition
JρTwo characteristic values be
Their corresponding feature vectors are v1And v2, vi=(cos θi,sinθi), i=1,2.
Wherein
If μ1And μ2It is two characteristic values of diffusion tensor matrices T (x), if
v1And v2It is corresponding feature vector, there is v1=(cos θ, sin θ)；v2=(- sin θ, cos θ).
The matrix element of T (x) and the relationship of eigen vector are as follows：
Using edge enhanced diffustion tensor：μ1=g (λ1), μ2=1；Wherein g is edge function.
Above-mentioned PDE is solved using finite difference：
Fractional Derivative can be by being calculated using efficient Discrete Fourier Transform：
Integer order derivative is generalized to Fractional Derivative, obtains converting the fractional order difference under meaning based on discrete FouierWithThey are in the corresponding relationship of airspace and frequency domain：
Fractional order difference operatorWithAdjoint operatorWithAirspace and frequency domain corresponding relationship
For：
Texture component is repaired using sample texture synthetic technology, it is adaptive as standard using the complexity of image block
Change region of search with answering to improve reparation speed；It is determined using the complexity of image block and repairs order to obtain preferable reparation
Effect.Detailed process is as follows：Using the complexity of fractal dimension and comentropy measurement image block, the field of search is determined using complexity
Domain and reparation order.The Weighted Threshold of empirically determined entropy and fractal dimension, be greater than threshold value multiblock to be repaired at, selection compared with
Complete matching padding in big region of search；It is then opposite at the multiblock to be repaired for being less than threshold value.And the big image of complexity
Block is preferentially repaired.In numerical value calculating, figure is calculated in conjunction with difference box-covering method and fractal Brown motion self-similarity method
As the fractal dimension of block, utmostly to distinguish different roughness texture.
The complexity of block of pixels is measured using comentropy.Comentropy is a kind of statistical form of feature, it reflects figure
As in average information number.The comentropy of image indicates the information content that the aggregation characteristic of intensity profile in image is included,
Enable PiIndicate block of pixels ΨpMiddle gray value is ratio shared by the pixel of i, i.e.,
The unitary comentropy for then defining gray level image is：
Wherein b is normalized parameter, selects b=5 in an experiment.Above formula only defines a metamessage of gray level image
Entropy, for color image, using the mean value of the unitary comentropy under tri- Color Channels of RGB, i.e.,
In the image mending for carrying out the textures synthesis based on sample block, comentropy H (p) big repairing block will be repaired first
It is multiple.
Block of pixels Ψ is further measured using fractal boxpComplexity.The fractal box of image is a kind of spy
The statistical form of sign reflects the number of average information in block of pixels, and value is normally between the 2-3 of section.Fractal dimension
Closer to dimension 2, show more flat (under normal conditions, the constant value block of pixels Ψ of imagep2) fractal box of=C is；It is closer
Dimension 3 shows that grey scale change is more violent, and image is more complicated.The unitary fractal box for then defining gray level image is：
F (p)=aD (Ψp)-b
Wherein a, b are normalized parameter, select a=1 in an experiment；B=-2.It is apparent that F (p) ∈ [0,1].If picture
Intensity profile is more uniform in plain block, and fractal box F (p) is closer to 0；, whereas if grey scale change is more violent in block of pixels,
Containing many image informations, fractal box F (p) is closer to 1.For color image, using under tri- Color Channels of RGB
Unitary fractal box mean value, i.e.,
The fractal box of block of pixels and comentropy are integrated in prioritization functions, the texture based on sample block is being carried out
When the image repair of synthesis, fractal box and the big reparation block of comentropy will be repaired first.
Prioritization functions are defined as：
P (p)=α C (p) D (p)+β F (p)+γ H (p)
Wherein α > 0, β > 0 and γ > 0 is weight factor, and meets alpha+beta+γ=1.
It matches block search and uses following process：By match block ΨqBy complexity, (fractal box adds set with comentropy
Power, i.e. β F (p)+γ H (p)) size imparting ascending order structure, binary search is then used, sequence match block Ψ is being assignedqIt is right in set
Under complexity meaning with input multiblock Ψ to be repairedpImmediate match block ΨqK neighborhood in carry out matching search again, find
The smallest match block of SSD.I.e.
Ψq=argmin { d (Ψp,Ψq):Ψq∈K}
Wherein K is match block ΨqIn set under fractal box meaning with input multiblock Ψ to be repairedpImmediate
With block ΨqK neighborhood.The size of k is selected as k=32.
(4) structure components and texture component after synthesis is repaired, obtain the complete correction of " Tujia " picture weaving in silk digital picture
As shown in figure 9, by after reparation structure components and texture component be weighted and averaged, synthesis repair after structure point
Amount and texture component, obtain the complete correction of final " Tujia " picture weaving in silk digital picture.
If the structure components and texture component after repairing are respectively u and v, then complete correction is expressed as
F=2 × (a × u+ (1-a) × v)
Wherein a ∈ 0,1) it is weight factor.Adjust the available different visual effect of this parameter.It sets under normal circumstances
A=0.5.
The application's repairs the remaining pattern of " Tujia " picture weaving in silk tradition by digital technology, can by repeatedly attempting,
Most satisfied repairing effect is found, while not needing to destroy original picture-weaving in silk, provides safety for the repair of " Tujia " picture weaving in silk
Convenient and fast approach.The present processes have for the texture-rich such as " Tujia " picture weaving in silk, beautiful in colour and textile repairs well
Effect can meet the vision requirement of people well.
" Tujia " picture weaving in silk digital picture damaged area it is automatic detection and positioning, by with the manually operated optimal result of expert
It compares, accuracy is up to 98% or more.The present processes have it is faster repair speed, 1024 × 1024 or less sizes to
Image is repaired, repairing area is 50 × 50 hereinafter, can about complete entire repair process in 100 seconds to 600 seconds.
The technical solution of the application, step is simple, easily operated, reproducible, and technical staff is easy to learn, easy grasp.
As used some vocabulary in the specification and claims to censure special component or method.Art technology
Personnel are, it is to be appreciated that different regions may call the same ingredient with different nouns.This specification and claims are not
In such a way that the difference of title is as ingredient is distinguished.As the "comprising" mentioned by throughout the specification and claims is
One open language, therefore should be construed to " including but not limited to "." substantially " refer within the acceptable error range, this field
Technical staff can solve the technical problem within a certain error range, basically reach the technical effect.Specification is subsequent
It is described as implementing the better embodiment of the application, so the description is for the purpose of the rule for illustrating the application, not
To limit scope of the present application.The protection scope of the application is as defined by the appended claims.
It should also be noted that, the terms "include", "comprise" or its any other variant are intended to nonexcludability
Include, so that commodity or system including a series of elements not only include those elements, but also including not clear
The other element listed, or further include for this commodity or the intrinsic element of system.In the feelings not limited more
Under condition, the element that is limited by sentence "including a ...", it is not excluded that in the commodity or system for including the element also
There are other identical elements.
Above description shows and describes several preferred embodiments of the present application, but as previously described, it should be understood that the application
Be not limited to forms disclosed herein, should not be regarded as an exclusion of other examples, and can be used for various other combinations,
Modification and environment, and the above teachings or related fields of technology or knowledge can be passed through in application contemplated scope described herein
It is modified.And changes and modifications made by those skilled in the art do not depart from spirit and scope, then it all should be in this Shen
It please be in the protection scope of appended claims.
Claims (3)
1. the remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure, which is characterized in that detected automatically including damaged area, image
Structure components and texture component are decomposed, the textures synthesis reparation of the reparation of structure components Variational PDE and texture component, specifically include with
Lower step：
(1) automatic detection and the location model of the digital picture damaged area of " Tujia " picture weaving in silk are established；
(2) " Tujia " picture weaving in silk picture breakdown is structure components and texture component by building variation picture breakdown model；
(3) design Variational PDE model repairs " Tujia " picture weaving in silk structure components, designs the Texture Synthesis based on sample and repairs Tujia
Picture-weaving in silk texture component；
(4) structure components and texture component being finally synthesizing after repairing, obtain the complete correction of " Tujia " picture weaving in silk digital picture；
The step (1) is to carry out the detection of damaged area using the variation movable contour model of blending image much information and determine
Position, specific method are：" Tujia " picture weaving in silk image is analyzed, 2 color model are selected, extracts color characteristic and textural characteristics,
And characteristic is merged, obtain Efficient image characteristic；The Efficient image characteristic information that fusion obtains is dissolved into
In Variation Model, image special objective extracted region and positioning are realized；
The step (2) is to be modeled respectively to the structure components and texture component of " Tujia " picture weaving in silk image using priori knowledge,
Variation Model is obtained, clean structural texture is obtained by functional minimization and is decomposed, specific method is：The structure components pass through
Non-convex biregular item modeling, the non-convex sparse measurement comprising gradient and the non-convex sparse measurement of second dervative；The texture component is adopted
It is measured with rank of matrix, consistency texture is extracted by the minimization of order；The Variation Model is solved using alternative iteration method.
2. the remaining pattern of " Tujia " picture weaving in silk tradition as described in claim 1 digitizes restorative procedure, which is characterized in that the step
(3) structure components are repaired for design Variational PDE model, design Texture Synthesis repairs texture component, has
Body method is：Fractional order differential is combined with tensor diffusion, proposes variation by the design for the Variational PDE repairing model
Model derives the corresponding Euler-Lagrange equation of mentioned Variation Model according to fractional order Functional Theory, and utilization is discrete
Fourier transform definition Fractional Derivative operator and its adjoint operator, derive the calculation formula of Fractional Derivative, design and mentioned
The numerical algorithm of repairing model；Design for the Texture Synthesis measures image using fractal box and comentropy
The complexity of block is repaired speed and is repaired using the region of search of complexity constraint match block and the definition of priority to improve
Multiple precision.
3. the remaining pattern of " Tujia " picture weaving in silk tradition as claimed in claim 2 digitizes restorative procedure, which is characterized in that the step
(4) structure components after synthesis reparation and the specific method of texture component are：To after reparation structure components and texture component into
Row weighted average.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610409424.2A CN106097268B (en) | 2016-06-12 | 2016-06-12 | The remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610409424.2A CN106097268B (en) | 2016-06-12 | 2016-06-12 | The remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106097268A CN106097268A (en) | 2016-11-09 |
| CN106097268B true CN106097268B (en) | 2018-11-23 |
Family
ID=57228684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610409424.2A Active CN106097268B (en) | 2016-06-12 | 2016-06-12 | The remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106097268B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108256163A (en) * | 2017-12-12 | 2018-07-06 | 吉首大学张家界学院 | " Tujia " picture weaving in silk product design method based on pel |
| CN108153973A (en) * | 2017-12-12 | 2018-06-12 | 吉首大学张家界学院 | " Tujia " picture weaving in silk pattern pel analysis method |
| CN109584178A (en) * | 2018-11-29 | 2019-04-05 | 腾讯科技（深圳）有限公司 | Image repair method, device and storage medium |
| CN110717550A (en) * | 2019-10-18 | 2020-01-21 | 山东大学 | Multi-modal image missing completion based classification method |
| CN111062126B (en) * | 2019-12-10 | 2022-04-12 | 湖北民族大学 | Tujia brocade design and appearance simulation method based on pattern example |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103093422A (en) * | 2011-11-04 | 2013-05-08 | 昆山云锦信息技术发展有限公司 | Repair of medical image of patella |
| CN104091332A (en) * | 2014-07-01 | 2014-10-08 | 黄河科技学院 | Method for optimizing multilayer image segmentation of multiclass color texture images based on variation model |
-
2016
- 2016-06-12 CN CN201610409424.2A patent/CN106097268B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103093422A (en) * | 2011-11-04 | 2013-05-08 | 昆山云锦信息技术发展有限公司 | Repair of medical image of patella |
| CN104091332A (en) * | 2014-07-01 | 2014-10-08 | 黄河科技学院 | Method for optimizing multilayer image segmentation of multiclass color texture images based on variation model |
Non-Patent Citations (6)
| Title |
|---|
| A Combined PDE and Texture Synthesis Approach to Inpainting;H Grossauer;《European Conference on Computer Vision-eccv》;20041231;第30卷(第22期);第1-11页 * |
| Image Restoration usingMultiresolution Texture Synthesis and Image Inpainting;H Yamauchi etal;《Proceedings of Cgi》;20031231(第1期);第1-6页 * |
| 变分和非凸正则在图像处理中的应用研究;韩雨;《中国博士学位论文全文数据库信息科技辑》;20130415(第4期);第10-100页 * |
| 基于偏微分方程理论的图像复原技术研究;卢兆林;《中国博士学位论文全文数据库信息科技辑》;20121015(第10期);第77页 * |
| 数字图像修补技术的研究进展与前景展望;万玮 等;《中国民族大学学报（自然科学版）》;20150531;第24卷(第2期);第1-4页 * |
| 融合颜色和纹理特征的彩色图像分割;贾士军 等;《测绘科学》;20141231;第39卷(第12期);第1-5页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106097268A (en) | 2016-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106097268B (en) | The remaining pattern of " Tujia " picture weaving in silk tradition digitizes restorative procedure | |
| CN106056155B (en) | Superpixel segmentation method based on boundary information fusion | |
| Taylor et al. | Authenticating Pollock paintings using fractal geometry | |
| CN103208001B (en) | In conjunction with shape-adaptive neighborhood and the remote sensing image processing method of texture feature extraction | |
| CN105279519B (en) | Remote sensing image Clean water withdraw method and system based on coorinated training semi-supervised learning | |
| CN107194872A (en) | Remote sensed image super-resolution reconstruction method based on perception of content deep learning network | |
| Smith et al. | Classification of archaeological ceramic fragments using texture and color descriptors | |
| CN104680492B (en) | Image repair method based on composition of sample uniformity | |
| CN108986132A (en) | A method of certificate photo Trimap figure is generated using full convolutional neural networks | |
| CN105844278A (en) | Multi-feature fused fabric scanning pattern recognition method | |
| CN106096542A (en) | Image/video scene recognition method based on range prediction information | |
| CN107909093A (en) | A kind of method and apparatus of Articles detecting | |
| CN106023098B (en) | Image mending method based on the more dictionary learnings of tensor structure and sparse coding | |
| CN104103076A (en) | Nuclear power plant planned restricted zone remote sensing inspecting method based on high-resolution remote sensing images | |
| CN103971367B (en) | Hydrologic data image segmenting method | |
| CN105809625A (en) | Fragment reconstruction method based on local texture pattern | |
| CN109448093A (en) | A kind of style image generation method and device | |
| Zhang et al. | Application and evaluation of object-oriented technology in high-resolution remote sensing image classification | |
| Sriram et al. | Classification of human epithelial type-2 cells using hierarchical segregation | |
| CN107220651A (en) | A kind of method and device for extracting characteristics of image | |
| Ledoux et al. | Toward a complete inclusion of the vector information in morphological computation of texture features for color images | |
| Ledoux et al. | Toward a full-band texture features for spectral images | |
| CN107368839B (en) | Automatic fault plane extraction method based on DEM | |
| Lyu et al. | Mural linear feature enhancement algorithm combining semi-supervised image segmentation and dimensionality reduction | |
| Huang | Research on classification of remote sensing image based on SVM including textural features |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |