CN106447610B - Image rebuilding method and device - Google Patents
Image rebuilding method and device Download PDFInfo
- Publication number
- CN106447610B CN106447610B CN201610789828.9A CN201610789828A CN106447610B CN 106447610 B CN106447610 B CN 106447610B CN 201610789828 A CN201610789828 A CN 201610789828A CN 106447610 B CN106447610 B CN 106447610B
- Authority
- CN
- China
- Prior art keywords
- image
- reconstructed
- image block
- dictionary
- grey scale
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012549 training Methods 0.000 claims abstract description 92
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 14
- 230000003321 amplification Effects 0.000 claims description 15
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 15
- 230000015572 biosynthetic process Effects 0.000 claims description 5
- 238000003786 synthesis reaction Methods 0.000 claims description 5
- 230000000052 comparative effect Effects 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 abstract description 22
- 230000000694 effects Effects 0.000 abstract description 17
- 238000010586 diagram Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- RKMGAJGJIURJSJ-UHFFFAOYSA-N 2,2,6,6-Tetramethylpiperidine Substances CC1(C)CCCC(C)(C)N1 RKMGAJGJIURJSJ-UHFFFAOYSA-N 0.000 description 6
- 239000000284 extract Substances 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- RINRSJBJOGCGBE-UHFFFAOYSA-N 3,3,5,6-tetramethyl-2h-pyrazine Chemical compound CC1=NCC(C)(C)N=C1C RINRSJBJOGCGBE-UHFFFAOYSA-N 0.000 description 3
- 230000004087 circulation Effects 0.000 description 3
- 238000001816 cooling Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000007850 degeneration Effects 0.000 description 1
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 1
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008929 regeneration Effects 0.000 description 1
- 238000011069 regeneration method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The present disclosure discloses a kind of image rebuilding method and devices, which comprises obtains image to be reconstructed;The image to be reconstructed is divided at least one image block to be reconstructed according to default size;Determine the grey scale change situation of the image block to be reconstructed;When the grey scale change situation of the image block to be reconstructed meets the first preset condition, the image block to be reconstructed is rebuild according to the dictionary of the double-deck dictionary training;When the grey scale change situation of the image block to be reconstructed meets the second preset condition, the image block to be reconstructed is rebuild based on the iterative algorithm of singular value decomposition.When the disclosure treats reconstruction image progress super-resolution reconstruction, the projection matrix that the to be reconstructed image block gentle for grey scale change is acquired using the double-deck dictionary of training, image block to be reconstructed more rich for edge details, corresponding high-definition picture block is rebuild using the iterative algorithm decomposed based on SVD, both preferable reconstruction effect can have been obtained, reconstruction time can be also saved.
Description
Technical field
This disclosure relates to image procossing, and in particular, to a kind of image rebuilding method and device.
Background technique
Image describes means as a kind of visual information, is one of the main source that people obtain information.And it measures
The key index of piece image is exactly image resolution ratio.Image resolution ratio refers to detailed information contained by piece image, details letter
Breath is more, and resolution ratio is higher, and the information obtained from image at this time is also more.But in the image actually obtained due to by
The limitation of imaging device and the influence of ambient enviroment can make to actually obtain image in the presence of degenerating, be unable to satisfy the need of practical application
It asks.
The super-resolution reconstruction of image is a kind of technology of resolution ratio for improving image.In the related technology, image high score
Resolution, which is rebuild, mainly uses the method based on frequency domain or the method based on airspace.However, carrying out image high score using these methods
When resolution is rebuild, it is bad not only to rebuild effect, but also reconstruction time is longer.
Therefore, the prior art is defective, needs to improve.
Summary of the invention
Purpose of this disclosure is to provide a kind of image rebuilding method and devices, efficiently to realize the high-resolution weight of image
It builds.
To achieve the goals above, in a first aspect, the disclosure provides a kind of image rebuilding method, comprising:
Obtain image to be reconstructed;
The image to be reconstructed is divided at least one image block to be reconstructed according to default size;
Determine the grey scale change situation of the image block to be reconstructed;
When the grey scale change situation of the image block to be reconstructed meets the first preset condition, according to the double-deck dictionary training
Dictionary rebuilds the image block to be reconstructed;
When the grey scale change situation of the image block to be reconstructed meets the second preset condition, changing based on singular value decomposition
The image block to be reconstructed is rebuild for algorithm.
Optionally, the step of grey scale change situation of the determination image block to be reconstructed includes:
Obtain the variance yields of the gray value of the image block to be reconstructed;
The variance yields is compared with preset threshold, with the determination grey scale change situation;
When the variance yields is less than the preset threshold, described in the grey scale change situation satisfaction of the image block to be reconstructed
First preset condition;
When the variance yields is greater than the preset threshold, described in the grey scale change situation satisfaction of the image block to be reconstructed
Second preset condition.
Optionally, the step of dictionary according to the double-deck dictionary training rebuilds the image block to be reconstructed packet
It includes:
It determines training library, obtains training data;The trained library include: preset quantity original high-resolution image and according to
The low resolution image that the original high-resolution image obtains;
According to the training data, training obtains main dictionary;
According to the main dictionary, the training data and the low-resolution image, residual dictionary is obtained;
According to the main dictionary, first stage reconstruction is carried out to the image block to be reconstructed;
According to the image that the residual dictionary and the first stage are rebuild, carries out second stage and rebuild to obtain
State the high-definition picture of image block to be reconstructed.
Optionally, the training data includes: to be formed using the high-frequency information that the original high-resolution image extracts
Image block characteristics vector sum by the low-resolution image carry out after interpolation amplification obtained low resolution interpolation amplification image into
The characteristics of image block eigenvector obtained after row filtering.
Optionally, the method also includes:
The high-definition picture obtained after each image block to be reconstructed is rebuild is synthesized to obtain the image to be reconstructed
High-definition picture.
Second aspect provides a kind of equipment for reconstructing image, comprising:
Module is obtained, for obtaining image to be reconstructed;
Image block module, for the image to be reconstructed to be divided at least one image to be reconstructed according to default size
Block;
Grey scale change determining module, for determining the grey scale change situation of the image block to be reconstructed;
First rebuild module, for when the image block to be reconstructed grey scale change situation meet the first preset condition when,
The image block to be reconstructed is rebuild according to the dictionary of the double-deck dictionary training;
Second rebuild module, for when the image block to be reconstructed grey scale change situation meet the second preset condition when,
The image block to be reconstructed is rebuild based on the iterative algorithm of singular value decomposition.
Optionally, the grey scale change determining module includes:
Variance acquisition submodule, the variance yields of the gray value for obtaining the image block to be reconstructed;
Comparative sub-module, for the variance yields to be compared with preset threshold, with the determination grey scale change situation;
When the variance yields is less than the preset threshold, described in the grey scale change situation satisfaction of the image block to be reconstructed
First preset condition;
When the variance yields is greater than the preset threshold, described in the grey scale change situation satisfaction of the image block to be reconstructed
Second preset condition.
Optionally, the first reconstruction module includes:
Training data acquisition submodule obtains training data for determining training library;The trained library includes: present count
The original high-resolution image of amount and the low resolution image obtained according to the original high-resolution image;
Main dictionary acquisition submodule, for according to the training data, training to obtain main dictionary;
Dictionary acquisition submodule is remained, is used for according to the main dictionary, the training data and the low-resolution image,
Obtain residual dictionary;
First stage rebuilds submodule, for carrying out the first stage to the image block to be reconstructed according to the main dictionary
It rebuilds;
Second stage rebuilds submodule, the image for being rebuild according to the residual dictionary and the first stage,
Second stage is carried out to rebuild to obtain the high-definition picture of the image block to be reconstructed.
Optionally, the training data includes: to be formed using the high-frequency information that the original high-resolution image extracts
Image block characteristics vector sum by the low-resolution image carry out after interpolation amplification obtained low resolution interpolation amplification image into
The characteristics of image block eigenvector obtained after row filtering.
Optionally, described device further include:
Synthesis module, the high-definition picture for obtaining after rebuilding each image block to be reconstructed are synthesized to obtain described
The high-definition picture of image to be reconstructed.
Through the above technical solutions, when treating reconstruction image progress super-resolution reconstruction, it is gentle for grey scale change
The projection matrix that image block to be reconstructed is acquired using the double-deck dictionary of training, to project into high-definition picture space, thus
Corresponding high-definition picture block is obtained, image reconstruction times are reduced;Image block to be reconstructed more rich for edge details, is adopted
Corresponding high-definition picture block is rebuild with the iterative algorithm decomposed based on SVD, reaches preferable and rebuilds effect.Both can obtain compared with
Good reconstruction effect, can also save reconstruction time.
Other feature and advantage of the disclosure will the following detailed description will be given in the detailed implementation section.
Detailed description of the invention
Attached drawing is and to constitute part of specification for providing further understanding of the disclosure, with following tool
Body embodiment is used to explain the disclosure together, but does not constitute the limitation to the disclosure.In the accompanying drawings:
Fig. 1 is the flow diagram of the image rebuilding method of one embodiment of the disclosure;
Fig. 2 is that the dictionary by the double-deck dictionary training of one embodiment of the disclosure obtains the high-definition picture of image block
Flow diagram;
Fig. 3 is the double-deck dictionary training flow diagram of one embodiment of the disclosure;
Fig. 4 is the double-deck dictionary reconstruction image flow diagram of one embodiment of the disclosure;
Fig. 5 is the flow diagram of the solution p norm minimum of one embodiment of the disclosure;
Fig. 6 is the flow diagram of the cooling algorithm of one embodiment of the disclosure;
Fig. 7 is the image rebuilding method flow diagram of one embodiment of the disclosure;
Fig. 8 a- Fig. 8 c is that the effect for carrying out full resolution pricture reconstruction using the image rebuilding method of the embodiment of the present disclosure is illustrated
Figure;
Fig. 9 is the structural schematic diagram for the equipment for reconstructing image that the disclosure one is implemented;
Figure 10 is the total block diagram for disclosing a kind of equipment for reconstructing image for terminal shown according to an exemplary embodiment.
Specific embodiment
It is described in detail below in conjunction with specific embodiment of the attached drawing to the disclosure.It should be understood that this place is retouched
The specific embodiment stated is only used for describing and explaining the disclosure, is not limited to the disclosure.
Referring to the flow diagram for the image rebuilding method that Fig. 1 is one embodiment of the disclosure.
In step slo, image to be reconstructed is obtained.
In embodiment of the disclosure, image to be reconstructed can for storage image (for example, the image that stores after shooting, or
The image etc. obtained by network), the image etc. shot in real time by capture apparatus.
In step s 11, image to be reconstructed is divided at least one image block to be reconstructed according to default size.
The default size of image block can be preset.In one embodiment, it is determined and is schemed according to the reconstruction effect of image
As the size of piecemeal.In practice, too small will lead to of image block needs image block excessive, and the reconstruction quality of image is caused to drop
It is low.Image block is excessive, then different method for reconstructing cannot be selected to carry out image reconstruction according to the gray scale situation of image block well, will
Accuracy can be reduced.As a result, in one embodiment, can by image block it is default be sized to 32 pixels × 32 pixels or
64 pixels × 64 pixels.
In step s 12, the grey scale change situation of image block to be reconstructed is determined.The grey scale change situation of image block to be reconstructed
It can be determined according to the variance yields of the gray scale of image block to be reconstructed.The side of the gray value of the pixel in statistical picture block can also be passed through
Formula determines.
In step s 13, when the grey scale change situation of image block to be reconstructed meets the first preset condition, according to the double-deck word
The dictionary of allusion quotation training is treated reconstruction image block and is rebuild;
When the grey scale change situation of image block to be reconstructed meets the second preset condition, the iteration based on singular value decomposition is calculated
Method is treated reconstruction image block and is rebuild.
The embodiment of the present disclosure obtains the high-resolution of image block according to the grey scale change situation of image block in different ways
Image.The grey scale change of image block is gentle, for example, grey scale change situation meets when the edge detail information of image block is less
First preset condition;The grey scale change of image block is fast, for example, then its gray scale becomes when the edge detail information of image block is relatively abundant
Change situation and meets the second preset condition.
In one embodiment, according to the variance yields of the gray value of image block and preset threshold σ2Determine the ash of the image block
Degree situation of change is to meet the first preset condition or the second preset condition.It is preset when the variance yields of the gray value of image block is less than
Threshold value σ2When, it is determined that the gray scale of the image block is smooth, variation is slow, obtains the image block according to the dictionary of the double-deck dictionary training
High-definition picture.When the variance yields of the gray value of image block is greater than preset threshold σ2When, it is determined that the details of the image block
Information is relatively abundanter, and the iterative algorithm based on singular value decomposition (Singular Value Decomposition, SVD) carries out weight
It builds, obtains the high-definition picture of the image block.
When obtaining the high-definition picture of the image block according to the dictionary of the double-deck dictionary training, by image block to be reconstructed multiplied by
Projection matrix projects image block to be reconstructed to high-definition picture space, obtains corresponding high-definition picture block.
In one embodiment, the image rebuilding method of the embodiment of the present disclosure further include: rebuild each image block to be reconstructed
The high-definition picture obtained afterwards is synthesized to obtain the high-definition picture of the image to be reconstructed.
According to the grey scale change situation of each image block to be reconstructed, the height of each image block to be reconstructed is obtained in different ways
Image in different resolution, each high-definition picture block, which is then carried out synthesis, can be obtained the high-definition picture of image to be reconstructed.
Here the piecemeal of synthesis and step S11 is corresponding.For example, carrying out piecemeal by the way of 32 pixels × 32 pixels, then
The full resolution pricture of these 32 pixels × 32 pixels image block is sequentially synthesized to the high-resolution that image to be reconstructed can be obtained
Image.
As a result, by the image rebuilding method of the embodiment of the present disclosure, when treating reconstruction image progress super-resolution reconstruction,
The projection matrix that the to be reconstructed image block gentle for grey scale change is acquired using the double-deck dictionary of training, with projection to high-resolution
In rate image space, to obtain corresponding high-definition picture block, reduce image reconstruction times;It is relatively abundant for edge details
Image block to be reconstructed, corresponding high-definition picture block is rebuild using the iterative algorithm that is decomposed based on SVD, reaches preferable weight
Build effect.Both preferable reconstruction effect can have been obtained, reconstruction time can be also saved.
In embodiment of the disclosure, in the embodiment of the present disclosure, when the grey scale change situation of the image block in image to be reconstructed
When meeting the second preset condition, the high-definition picture of image block is obtained by the dictionary of the double-deck dictionary training.Referring to fig. 2, it is
The dictionary by the double-deck dictionary training of one embodiment of the disclosure obtains the flow diagram of the high-definition picture of image block.
It include: training stage and reconstruction rank by the high-definition picture that the dictionary of the double-deck dictionary training obtains image block
Section.Following steps S21- step S23 belongs to the training stage, and step S24- step S25 belongs to phase of regeneration.
In the step s 21, it determines training library, obtains training data.
Referring to Fig. 3, training library includes: one or more image.These images include original high-resolution image HORGWith it is low
Image in different resolution LLF, for example, training library may include the original high-resolution image that 4 resolution ratio are 512*512 and a resolution
Rate is the low-resolution image of 256*256.
Wherein, low-resolution image LLFIt can be by high-definition picture HORGDown-sampling is carried out (for example, one point in interval
A pixel is taken to reduce the resolution ratio of image), fuzzy (for example, carrying out fuzzy operation using Gauss operator, change pixel
Gray value to be to reduce the resolution ratio of image) etc. degenerations obtain.
In one embodiment, in order to improve accuracy, the image in training library includes the more rich image of marginal information.
The more rich image of marginal information, the i.e. big image of the grey scale change of image.
The acquisition of training data:
Feature extraction is carried out respectively to the image in training library, and piecemeal handles to obtain training data
Wherein,It is the image block characteristics vector of the high-frequency information composition extracted using original high-resolution image.It is by low resolution
Rate image carries out the low resolution interpolation amplification image obtained after interpolation amplification, using the firstorder filter as shown in following formula (1)
The image block characteristics vector that feature obtains is extracted with second order filter.
f1=[0,0,1,0,0, -1], f2=f1 T
f3=[1,0,0, -2,0,0,1], f4=f3 T (1)
In one embodiment, can by original high-resolution image withSubtract each other to obtain
I.e. training data includes: the image block characteristics vector formed using the high-frequency information that high-definition picture extractsWith
Low-resolution image is carried out to the low resolution interpolation amplification image obtained after interpolation amplification, is filtered using single order shown in formula (1)
Wave device and second order filter extract the image block characteristics vector that feature obtainsMain dictionary training high-definition picture in Fig. 3
HHFAsMain dictionary training low-resolution image HLFAs
In step S22, according to training data, training obtains main dictionary.Main dictionary includes main low frequency dictionary LMDWith main height
Frequency dictionary HMD。
Wherein, for the image block characteristics vector in training dataDictionary training is carried out using K-SVD algorithm, is obtained
To main low frequency dictionary LMD, it may be assumed that
In formula (2), { qk}kFor rarefaction representation vector, | | | |0Indicate L0Norm.
Based on the image block characteristics vector in training dataDue toThen main high frequency dictionary can lead to
Optimization following formula (3) is crossed to obtain.
Wherein, matrix PhIt is respectively indicated with Q{ qk}kSet, i.e.,Q={ qk}k, using broad sense
The method of inverse matrix solves.After optimization, main high frequency dictionary HMDAs shown in formula (4).
HMD=PhQ+=PhQT(QQT)-1 (4)
In step S23, according to main dictionary, training data and low-resolution image, residual dictionary is obtained.
In embodiment of the disclosure, the reconstruction effect of high-definition picture can be reinforced by remaining the training of dictionary.
Utilize main dictionary { LMD,HMDTo main dictionary training high-definition picture HHF(i.e.) image reconstruction is carried out, obtain weight
Build main high frequency imaging HMHF, and the image H that will be obtainedMHFAs the input of second layer residual dictionary training, with main dictionary training
Journey is similar, obtains residual dictionary { LRD,HRD}.Referring to Fig. 3, by image HMHFWith main dictionary training high-definition picture HHF(i.e.)
Subtract each other to obtain residual dictionary training high-definition picture HRHF.By image HMHFWith low-resolution image LLFAddition obtains residual dictionary
Training low-resolution image HTMP。
As a result, according to residual dictionary training high-definition picture HRHFWith residual dictionary training low-resolution image HTMPIt carries out
Training obtains residual dictionary { LRD,HRD}.It is trained it should be understood that method shown in formula 2 and formula 3 can be used when training.
In step s 24, it according to main dictionary, treats reconstruction image and carries out first stage reconstruction.
Referring to fig. 4, image to be reconstructed is low-resolution image LINPUT.When carrying out first stage image reconstruction, to input
Low-resolution image to be reconstructed carry out interpolation amplification and be filtered using filter, and extract the feature of overlapping image block
Set(i.e. main dictionary training low-resolution image H in Fig. 6LF).Filter shown in formula (1) can be used in filter.It mentions
The characteristic set for taking overlapping image block, is the error of edge pixel in order to prevent.General overlaid pixel is The more the better, but examines
Consider operand, the pixel of overlapping can be arranged in a certain range, for example, setting 9*9 picture for the block of pixels of overlapping
Element.
Utilize main dictionary { LMD,HMD, according to formula (5), to main dictionary training low-resolution image HLFCollaboration expression is carried out,
To obtain rebuilding main high frequency imaging HMHF。
Wherein, DlFor low-resolution dictionary, the as L in the main dictionaryMD, parameter lambda for equation of equilibrium emulation item and
Sparse item, y, that is, main dictionary training low-resolution image HLF。
It can be obtained according to formula (5):
α=(Dl TDl+λI)-1Dl Ty (6)
In one embodiment, sparse coefficient and high-resolution dictionary directly can be mapped to height by high-definition picture block
In resolution space, it may be assumed that
X=Dhα (7)
Wherein, x is the high-definition picture of output, DhFor in the corresponding high-resolution dictionary of low-resolution dictionary, as
Corresponding H in main dictionaryMD。
Available according to formula (6) and (7), the high-definition picture x of output can be indicated by formula (8).
X=Dh(Dl TDl+λI)-1Dl Ty (8)
Referring to fig. 4, according to the main high frequency imaging H of reconstructionMHFWith main dictionary training low-resolution image HLFAddition obtains high-resolution
Rate image rebuilds the high-definition picture of output for the first stage.
According to formula (8) can be obtained low-resolution image to high-definition picture projection matrix PG, by formula (9) table
Show.
PG=Dh(Dl TDl+λI)-1Dl T (9)
Wherein, PGFor projection matrix.H in main dictionaryMDAnd LMDIt is that training obtains, I is unit matrix.It projects as a result,
Matrix can be with off-line calculation.As a result, directly by low resolution feature vector pl kMultiplied by the projection matrix, corresponding high-resolution is obtained
Rate image block, to greatly reduce the high resolution image reconstruction time.
In step s 25, according to residual dictionary and the full resolution pricture rebuild of first stage, treat reconstruction image into
Row second stage is rebuild.
Referring to fig. 4, according to the main high frequency imaging H of reconstructionMHFWith main dictionary training low-resolution image HLFAddition obtains residual word
Allusion quotation trains low-resolution image HTMP。
The input that the image that first stage is rebuild is rebuild as second stage.Second stage reconstruction process and first
Stage reconstruction process is similar, can be only that main dictionary is changed to residual dictionary, by main dictionary according to formula 5 to formula shown in formula 8
Training low-resolution image HLFReplace with residual dictionary training low-resolution image HTMP.Referring to fig. 4, according to residual dictionary { LRD,
HRDAnd residual dictionary training low-resolution image HTMPObtain residual dictionary training high-definition picture HRHF, then will remain word
Allusion quotation trains high-definition picture HRHFWith residual dictionary training low-resolution image HTMPAddition obtains full resolution pricture HEST, the height
Image in different resolution HESTThe full resolution pricture of image as to be reconstructed.
The dictionary (main dictionary and residual dictionary) obtained using the learning method training of the double-deck dictionary seeks projection matrix PG,
Low-resolution image to be reconstructed is projected directly into high resolution graphics image space according to projection matrix and obtains high-definition picture,
It is short with reconstruction time, the simple effect of reconstruction process.
In embodiment of the disclosure, in the embodiment of the present disclosure, when the grey scale change situation of the image block in image to be reconstructed
When meeting the second preset condition, image reconstruction is carried out based on the iterative algorithm that SVD is decomposed.
Singular value decomposition (SVD) can be compressed as a kind of important matrix disassembling method using the non-full rank of image
Mass data.Treat reconstruction image x is indicated with matrix X, and SVD is decomposed is defined as:
X=USXV′ (10)
Wherein, U and V is normal orthogonal basic matrix, SXFor a pair of of diagonal matrices, formula (10) is one effectively dilute to X
Representation method is dredged, it can be by giving up the relatively small singular value of some numerical value come approximate.Optimal problem, which can be exchanged into, to be made to weight
The order for building image reaches minimum.
Minimize:rank (X) s.t. | | y-Fx | |2< ε (11)
Wherein, X is the matrix form of image x to be reconstructed, and F is then the observing matrix of image x to be reconstructed.Observing matrix,
The operations such as fuzzy or down-sampling acquisition can be carried out by treating reconstruction image.Formula (11) may be expressed as:
min||X||*s.t.||y-Fx||2≤ε (12)
Wherein, | | X | |*For the nuclear norm of X, it is defined as the sum of singular value, due in general, lpNorm (0 < p < 1) can compare l1
Norm obtains preferable effect.Similar, the embodiment of the present disclosure uses lpNorm replaces l1Norm is estimated, it may be assumed that
At this point, the optimization problem of formula (13) is converted are as follows:
min||X||p, 0 < p < 1s.t. | | y-Fx | |2≤ε (14)
Objective function is set when solution as J (x):
J (x)=min | | y-Fx | |2+λ||X||p (15)
Wherein, parameter lambda is sparse item and fidelity term in balancing objective function.J (x) minimum can then be obtained into optimal solution:
Iterative threshold algorithm is a kind of simple and effective rarefaction representation image rebuilding method, which can directly eliminate
The artifact as brought by the down-sampling of the space K has reconstruction effect good, the few feature of free parameter.
Referring to the flow diagram for the solution p norm minimum that Fig. 5 is one embodiment of the disclosure.The embodiment of the present disclosure uses
Formula (14) are solved p norm minimum comprising following by a kind of First Order Iterative algorithm of Majumdar decomposed based on SVD
Step:
In step s 51, if xk=xk-1+FT(y-Fxk-1)。
In step S52, to matrix xkIt carries out conversion and is modelled as the matrix X to be rebuildk。
In step S53, to XkCarry out SVD decomposition, i.e. Xk=U ∑ VT。
In step S54, soft-threshold is carried out using the singular value that step S653 is decomposed and solves the resulting singular value of update
In step S55, according to the singular value that step S54 is obtained, X is solvedk+1, Xk+1=U ∑ VT, by Xk+1Vector turns to
xk+1。
In step S56, the number of iterations is updated, even k=k+1, and return step S51.
For the optimization problem for solving formula (14), the embodiment of the present disclosure uses Cooling algorithm.The algorithm is that parameter ε and λ are built
Connection is stood, which includes two circulations, and the target of interior circulation is to solve formula (15) minimum value when λ is fixed, terminate
Function are as follows:
When terminating function less than iteration ends value, interior loop termination.Outer circulation operation is then mainly according to decay factor
DecFac reduces the value of λ.When no longer meet | | y-Fx | |2> ε, outer loop end.
Referring to Fig. 6, the cooling algorithm of the embodiment of the present disclosure the following steps are included:
In step S61, initialized.Enabling sparse coefficient vector is x0=0, λ < max (FTX), the number of iterations t=1,
Iteration ends value Tol and ε and decay factor DecFac is set separately.
In step S62, the objective function of current iteration is calculated using formula (15)And
According to the above-mentioned First Order Iterative algorithmic minimizing J decomposed based on SVDk。
In step S63, new objective function is calculated for next iteration
In step S64, the number of iterations t=t+1 is updated.
In step S65, compare the size for terminating function and iteration ends value Tol, if function is more than or equal to Tol
When, it is back to step S63, is otherwise continued to execute.
In step S66, according to the decay factor DecFac of initialization, the size of λ is reduced.
In step S67, judgement | | y-Fx | |2With the size of ε, if | | y-Fx | |2> ε, return step S63, otherwise stops
Only iteration.
There is good image reconstruction effect for the image augmentation of small size based on the iterative algorithm that SVD is decomposed.
Referring to Fig. 7, the image rebuilding method of the embodiment of the present disclosure, to low-resolution image to be reconstructed according to the figure of setting
As the progress piecemeal processing of block piecemeal size, the Variance feature value of each image block is acquired.By the variance of each image block acquired
Value and the image variance threshold value of setting make a decision.If variance yields is less than threshold value, determine that the image block gray scale is smooth, variation is slow
Slowly, directly low-resolution image block to be reconstructed is projected multiplied by projection matrix to high-definition picture space and obtains corresponding high score
Resolution image block (i.e. using the double-deck dictionary method), conversely, variance yields is greater than threshold value, then it is assumed that the image block detailed information is richer
Richness, using the iterative algorithm reconstruction image for being based on singular value decomposition (Singular Value Decomposition, SVD).By
It is fast in the reconstruction time of the double-deck dictionary method, and the reconstruction of the method based on singular value decomposition effect is good, therefore, the embodiment of the present disclosure
Image rebuilding method can reach reduce algorithm the time required to purpose can the Equilibrium fitting time and rebuild effect, make image high-resolution
The result that the performance that rate is rebuild is optimal.
It is the effect that full resolution pricture reconstruction is carried out using the image rebuilding method of the embodiment of the present disclosure referring to Fig. 8 a- Fig. 8 c
Schematic diagram.Wherein, Fig. 8 a is original high-resolution image (as training library), and Fig. 8 b is low-resolution image, and Fig. 8 c is to rebuild
High-definition picture afterwards.
It is the structural schematic diagram for the equipment for reconstructing image that the disclosure is implemented referring to Fig. 9.The equipment for reconstructing image 900 includes:
Module 901 is obtained, for obtaining image to be reconstructed;
Image block module 902, it is to be reconstructed for the image to be reconstructed to be divided at least one according to default size
Image block;
Grey scale change determining module 903 is used to determine the grey scale change situation of the image block to be reconstructed;
First rebuilds module 904, meets the first preset condition for the grey scale change situation when the image block to be reconstructed
When, the image block to be reconstructed is rebuild according to the dictionary of the double-deck dictionary training;
Second rebuilds module 905, meets the second preset condition for the grey scale change situation when the image block to be reconstructed
When, the image block to be reconstructed is rebuild based on the iterative algorithm of singular value decomposition.
In one embodiment, grey scale change determining module 903 includes:
Variance acquisition submodule 9031, the variance yields of the gray value for obtaining the image block to be reconstructed;
Comparative sub-module 9032, for the variance yields to be compared with preset threshold, with the determination grey scale change
Situation;
When the variance yields is less than the preset threshold, described in the grey scale change situation satisfaction of the image block to be reconstructed
First preset condition;
When the variance yields is greater than the preset threshold, described in the grey scale change situation satisfaction of the image block to be reconstructed
Second preset condition.
In one embodiment, the first reconstruction module 904 includes:
Training data acquisition submodule 9041 obtains training data for determining training library;The trained library includes: pre-
If the original high-resolution image of quantity and the low resolution image obtained according to the original high-resolution image;
Main dictionary acquisition submodule 9042, for according to the training data, training to obtain main dictionary;
Dictionary acquisition submodule 9043 is remained, for according to the main dictionary, the training data and the low resolution
Image obtains residual dictionary;
First stage rebuilds submodule 9044, for carrying out first to the image block to be reconstructed according to the main dictionary
Stage rebuilds;
Second stage rebuilds submodule 9045, the figure for being rebuild according to the residual dictionary and the first stage
Picture carries out second stage and rebuilds to obtain the high-definition picture of the image block to be reconstructed.
In one embodiment, training data includes: the high-frequency information group extracted using the original high-resolution image
At image block characteristics vector sum the low-resolution image is subjected to obtained low resolution interpolation amplification figure after interpolation amplification
As the characteristics of image block eigenvector obtained after being filtered.
In one embodiment, device 900 further include:
Synthesis module 906, the high-definition picture for obtaining after rebuilding each image block to be reconstructed are synthesized to obtain
The high-definition picture of the image to be reconstructed.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
Figure 10 is a kind of block diagram of equipment for reconstructing image 100 for terminal shown according to an exemplary embodiment, should
Device 100 can be mobile terminal.As shown in Figure 10, which may include: processor 1001, memory 1002, more matchmakers
Body component 1003, input/output (I/O) interface 1004, communication component 1005 and video capture component 1006.
Wherein, processor 1001 is used to control the integrated operation of the device 100, to complete the above-mentioned image for terminal
All or part of the steps in method for reconstructing.Memory 1002 is for storing various types of data to support in the device 100
Operation, these data for example may include any application or method for being operated on the device 100 instruction,
And the relevant data of application program, such as contact data, the message of transmitting-receiving, picture, audio, video etc..The memory
1002 can be realized by any kind of volatibility or non-volatile memory device or their combination, such as static random is deposited
Access to memory (Static Random Access Memory, abbreviation SRAM), electrically erasable programmable read-only memory
(Electrically Erasable Programmable Read-Only Memory, abbreviation EEPROM), erasable programmable
Read-only memory (Erasable Programmable Read-Only Memory, abbreviation EPROM), programmable read only memory
(Programmable Read-Only Memory, abbreviation PROM), and read-only memory (Read-Only Memory, referred to as
ROM), magnetic memory, flash memory, disk or CD.
Multimedia component 1003 may include screen and audio component.Wherein screen for example can be touch screen, audio group
Part is used for output and/or input audio signal.For example, audio component may include a microphone, microphone is outer for receiving
Portion's audio signal.The received audio signal can be further stored in memory 1002 or be sent out by communication component 1005
It send.Audio component further includes at least one loudspeaker, is used for output audio signal.I/O interface 1004 is processor 1001 and its
Interface is provided between his interface module, other above-mentioned interface modules can be keyboard, mouse, button etc..These buttons can be
Virtual push button or entity button.
Communication component 1005 is for carrying out wired or wireless communication between the device 100 and other equipment.Wireless communication, example
Such as Wi-Fi, bluetooth, near-field communication (Near Field Communication, abbreviation NFC), 2G, 3G or 4G or they in
One or more of combinations, therefore the corresponding communication component 1005 may include: Wi-Fi module, bluetooth module, NFC module.
Video capture component 1006 may include the modules such as camera, signal processing, for acquiring image.
In embodiment of the disclosure, image to be reconstructed can be the image by 1006 shooting, collecting of video capture component,
It can also be the image obtained from network server or other terminal devices by communication component 1005.
In one exemplary embodiment, device 100 can be by one or more application specific integrated circuit
(Application Specific Integrated Circuit, abbreviation ASIC), digital signal processor (Digital
Signal Processor, abbreviation DSP), digital signal processing appts (Digital Signal Processing Device,
Abbreviation DSPD), programmable logic device (Programmable Logic Device, abbreviation PLD), field programmable gate array
(Field Programmable Gate Array, abbreviation FPGA), controller, microcontroller, microprocessor or other electronics member
Part is realized, for executing the above-mentioned image rebuilding method for terminal.
In a further exemplary embodiment, a kind of non-transitory computer-readable storage medium including instruction is additionally provided
Matter, the memory 1002 for example including instruction, above-metioned instruction can be executed above-mentioned to complete by the processor 10101 of device 100
Image rebuilding method for terminal.Illustratively, which can be ROM, arbitrary access
Memory (Random Access Memory, abbreviation RAM), CD-ROM, tape, floppy disk and optical data storage devices etc..
The image rebuilding method and device of the embodiment of the present disclosure, according to the variance yields of image block, different grey scale changes are not
With different image rebuilding methods, the reconstruction time of synthetic image and the performance indicator of image reconstruction effect is selected, weight is being ensured
In the case where building high-definition picture quality, the time needed for capable of largely reducing reconstruction image.
Any process described otherwise above or method description can be by flow chart or in embodiment of the disclosure
It is interpreted as, expression includes the steps that one or more codes for realizing specific logical function or the executable instruction of process
Module, segment or part, and the range of disclosure embodiment includes other realization, wherein can not by shown or
The sequence of discussion, including according to related function by it is basic simultaneously in the way of or in the opposite order, Lai Zhihang function, this is answered
By embodiment of the disclosure, the technical personnel in the technical field understand.
The preferred embodiment of the disclosure is described in detail in conjunction with attached drawing above, still, the disclosure is not limited to above-mentioned reality
The detail in mode is applied, in the range of the technology design of the disclosure, a variety of letters can be carried out to the technical solution of the disclosure
Monotropic type, these simple variants belong to the protection scope of the disclosure.
It is further to note that specific technical features described in the above specific embodiments, in not lance
In the case where shield, can be combined in any appropriate way, in order to avoid unnecessary repetition, the disclosure to it is various can
No further explanation will be given for the combination of energy.
In addition, any combination can also be carried out between a variety of different embodiments of the disclosure, as long as it is without prejudice to originally
Disclosed thought equally should be considered as disclosure disclosure of that.
Claims (8)
1. a kind of image rebuilding method characterized by comprising
Obtain image to be reconstructed;
The image to be reconstructed is divided at least one image block to be reconstructed according to default size;
Determine the grey scale change situation of the image block to be reconstructed;
When the grey scale change situation of the image block to be reconstructed meets the first preset condition, according to the dictionary of the double-deck dictionary training
The image block to be reconstructed is rebuild, wherein the dictionary according to the double-deck dictionary training is to the image block to be reconstructed
Rebuild and comprise determining that trained library, obtain training data, the trained library include preset quantity original high-resolution image and
The low resolution image obtained according to the original high-resolution image;According to the training data, training obtains main dictionary;According to
The main dictionary, the training data and the low-resolution image obtain residual dictionary;According to the main dictionary, to described
Image block to be reconstructed carries out first stage reconstruction;According to the residual dictionary and the image rebuild of the first stage, into
Row second stage is rebuild to obtain the high-definition picture of the image block to be reconstructed;
When the grey scale change situation of the image block to be reconstructed meets the second preset condition, the iteration based on singular value decomposition is calculated
Method rebuilds the image block to be reconstructed.
2. the method according to claim 1, wherein the grey scale change feelings of the determination image block to be reconstructed
The step of condition includes:
Obtain the variance yields of the gray value of the image block to be reconstructed;
The variance yields is compared with preset threshold, with the determination grey scale change situation;
When the variance yields is less than the preset threshold, the grey scale change situation of the image block to be reconstructed meets described first
Preset condition;
When the variance yields is greater than the preset threshold, the grey scale change situation of the image block to be reconstructed meets described second
Preset condition.
3. the method according to claim 1, wherein the training data includes: to utilize the original high-resolution
The image block characteristics vector sum of the high-frequency information composition of rate image zooming-out obtains after the low-resolution image is carried out interpolation amplification
To low resolution interpolation amplification image be filtered after obtained characteristics of image block eigenvector.
4. the method according to claim 1, wherein the method also includes:
The high-definition picture obtained after each image block to be reconstructed is rebuild is synthesized to obtain the high score of the image to be reconstructed
Resolution image.
5. a kind of equipment for reconstructing image characterized by comprising
Module is obtained, for obtaining image to be reconstructed;
Image block module, for the image to be reconstructed to be divided at least one image block to be reconstructed according to default size;
Grey scale change determining module, for determining the grey scale change situation of the image block to be reconstructed;
First rebuild module, for when the image block to be reconstructed grey scale change situation meet the first preset condition when, according to
The dictionary of the double-deck dictionary training rebuilds the image block to be reconstructed, wherein the first reconstruction module includes: trained number
Training data is obtained, the trained library includes the original high-resolution figure of preset quantity for determining training library according to acquisition submodule
Picture and the low resolution image obtained according to the original high-resolution image;Main dictionary acquisition submodule, for according to the instruction
Practice data, training obtains main dictionary;Dictionary acquisition submodule is remained, for according to the main dictionary, the training data and institute
Low-resolution image is stated, residual dictionary is obtained;First stage rebuilds submodule, is used for according to the main dictionary, to described to weight
It builds image block and carries out first stage reconstruction;Second stage rebuilds submodule, for according to the residual dictionary and first rank
The image that Duan Chongjian is obtained carries out second stage and rebuilds to obtain the high-definition picture of the image block to be reconstructed;
Second rebuilds module, for being based on when the grey scale change situation of the image block to be reconstructed meets the second preset condition
The iterative algorithm of singular value decomposition rebuilds the image block to be reconstructed.
6. device according to claim 5, which is characterized in that the grey scale change determining module includes:
Variance acquisition submodule, the variance yields of the gray value for obtaining the image block to be reconstructed;
Comparative sub-module, for the variance yields to be compared with preset threshold, with the determination grey scale change situation;
When the variance yields is less than the preset threshold, the grey scale change situation of the image block to be reconstructed meets described first
Preset condition;
When the variance yields is greater than the preset threshold, the grey scale change situation of the image block to be reconstructed meets described second
Preset condition.
7. device according to claim 5, which is characterized in that the training data includes: to utilize the original high-resolution
The image block characteristics vector sum of the high-frequency information composition of rate image zooming-out obtains after the low-resolution image is carried out interpolation amplification
To low resolution interpolation amplification image be filtered after obtained characteristics of image block eigenvector.
8. device according to claim 5, which is characterized in that described device further include:
Synthesis module, the high-definition picture for obtaining after rebuilding each image block to be reconstructed are synthesized to obtain described to weight
Build the high-definition picture of image.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610789828.9A CN106447610B (en) | 2016-08-31 | 2016-08-31 | Image rebuilding method and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610789828.9A CN106447610B (en) | 2016-08-31 | 2016-08-31 | Image rebuilding method and device |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106447610A CN106447610A (en) | 2017-02-22 |
CN106447610B true CN106447610B (en) | 2019-07-23 |
Family
ID=58164042
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610789828.9A Expired - Fee Related CN106447610B (en) | 2016-08-31 | 2016-08-31 | Image rebuilding method and device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106447610B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107123097B (en) * | 2017-04-26 | 2019-08-16 | 东北大学 | A kind of imaging method of the calculation matrix based on optimization |
CN109712209B (en) * | 2018-12-14 | 2022-09-20 | 深圳先进技术研究院 | PET image reconstruction method, computer storage medium, and computer device |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6304605B1 (en) * | 1994-09-13 | 2001-10-16 | Nokia Mobile Phones Ltd. | Video compressing method wherein the direction and location of contours within image blocks are defined using a binary picture of the block |
CN101980291A (en) * | 2010-11-03 | 2011-02-23 | 天津大学 | Random micro-displacement-based super-resolution image reconstruction method |
CN102222357A (en) * | 2010-04-15 | 2011-10-19 | 温州大学 | Foot-shaped three-dimensional surface reconstruction method based on image segmentation and grid subdivision |
CN103366340A (en) * | 2012-04-06 | 2013-10-23 | 索尼公司 | Image processing device and method and electronic device using image processing device and method |
CN105844590A (en) * | 2016-03-23 | 2016-08-10 | 武汉理工大学 | Image super-resolution reconstruction method and system based on sparse representation |
-
2016
- 2016-08-31 CN CN201610789828.9A patent/CN106447610B/en not_active Expired - Fee Related
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6304605B1 (en) * | 1994-09-13 | 2001-10-16 | Nokia Mobile Phones Ltd. | Video compressing method wherein the direction and location of contours within image blocks are defined using a binary picture of the block |
CN102222357A (en) * | 2010-04-15 | 2011-10-19 | 温州大学 | Foot-shaped three-dimensional surface reconstruction method based on image segmentation and grid subdivision |
CN101980291A (en) * | 2010-11-03 | 2011-02-23 | 天津大学 | Random micro-displacement-based super-resolution image reconstruction method |
CN103366340A (en) * | 2012-04-06 | 2013-10-23 | 索尼公司 | Image processing device and method and electronic device using image processing device and method |
CN105844590A (en) * | 2016-03-23 | 2016-08-10 | 武汉理工大学 | Image super-resolution reconstruction method and system based on sparse representation |
Non-Patent Citations (4)
Title |
---|
An Improved Super-resolution Image Reconstruction Algorithm;Yong Yin,et al;《International Journal of Signal Processing,Image Processing and Pattern Recognition》;20160331;第9卷(第3期);第103-112页 |
图像插值的改进自适应核回归方法;印勇等;《光电工程》;20110131;第38卷(第1期);第112-116页 |
采用CDD模型的自适应图像修复算法;印勇等;《重庆大学学报》;20130430;第36卷(第4期);第80-86页 |
采用曲率驱动的自适应图像修复算法;印勇等;《光电子·激光》;20121231;第23卷(第12期);第2426-2431页 |
Also Published As
Publication number | Publication date |
---|---|
CN106447610A (en) | 2017-02-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7392227B2 (en) | Feature pyramid warping for video frame interpolation | |
CN108022212B (en) | High-resolution picture generation method, generation device and storage medium | |
CN109166130B (en) | Image processing method and image processing device | |
Yu et al. | A unified learning framework for single image super-resolution | |
WO2018161775A1 (en) | Neural network model training method, device and storage medium for image processing | |
US8594464B2 (en) | Adaptive super resolution for video enhancement | |
CN110322416A (en) | Image processing method, device and computer readable storage medium | |
WO2020015330A1 (en) | Enhanced neural network-based image restoration method, storage medium, and system | |
CN109361934B (en) | Image processing method, device, equipment and storage medium | |
CN109409503A (en) | Training method, image conversion method, device, equipment and the medium of neural network | |
CN109525859A (en) | Model training, image transmission, image processing method and relevant apparatus equipment | |
JP2003018398A (en) | Method for generating a super-resolution image from pixel image | |
CN106663314A (en) | Real time skin smoothing image enhancement filter | |
CN105631807A (en) | Single-frame image super resolution reconstruction method based on sparse domain selection | |
CN110222758A (en) | A kind of image processing method, device, equipment and storage medium | |
WO2020119581A1 (en) | Magnetic resonance parameter imaging method and apparatus, device and storage medium | |
RU2697928C1 (en) | Superresolution of an image imitating high detail based on an optical system, performed on a mobile device having limited resources, and a mobile device which implements | |
WO2022166298A1 (en) | Image processing method and apparatus, and electronic device and readable storage medium | |
CN109949217B (en) | Video super-resolution reconstruction method based on residual learning and implicit motion compensation | |
CN110958469A (en) | Video processing method and device, electronic equipment and storage medium | |
KR20200132682A (en) | Image optimization method, apparatus, device and storage medium | |
EP4075373A1 (en) | Image processing method and apparatus | |
CN106447610B (en) | Image rebuilding method and device | |
CN110443764A (en) | Video repairing method, device and server | |
Dai et al. | Image super-resolution via residual block attention networks |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190723 Termination date: 20200831 |
|
CF01 | Termination of patent right due to non-payment of annual fee |