CN107845065B - Super-resolution image reconstruction method and device - Google Patents
Super-resolution image reconstruction method and device Download PDFInfo
- Publication number
- CN107845065B CN107845065B CN201710835137.2A CN201710835137A CN107845065B CN 107845065 B CN107845065 B CN 107845065B CN 201710835137 A CN201710835137 A CN 201710835137A CN 107845065 B CN107845065 B CN 107845065B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution image
- resolution
- dictionary
- detail layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 30
- 238000010276 construction Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 31
- 238000012937 correction Methods 0.000 claims description 15
- 238000001914 filtration Methods 0.000 claims description 8
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 8
- 230000006870 function Effects 0.000 description 13
- 238000005457 optimization Methods 0.000 description 11
- 238000002595 magnetic resonance imaging Methods 0.000 description 6
- 210000004556 brain Anatomy 0.000 description 4
- 238000009499 grossing Methods 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 2
- 230000008602 contraction Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 238000013019 agitation Methods 0.000 description 1
- 238000003759 clinical diagnosis Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007850 degeneration Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000004884 grey matter Anatomy 0.000 description 1
- 125000001475 halogen functional group Chemical group 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 210000004885 white matter Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
- G06T3/4076—Super resolution, i.e. output image resolution higher than sensor resolution by iteratively correcting the provisional high resolution image using the original low-resolution image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/28—Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2211/00—Image generation
- G06T2211/40—Computed tomography
- G06T2211/416—Exact reconstruction
Abstract
The present invention provides a super-resolution image reconstruction apparatus, including: a training sample acquisition unit for acquiring a training sample; the dictionary construction unit is used for constructing a dictionary; the low-resolution image input unit is used for converting the low-resolution image into a high-resolution image initial estimation; the sparse coding unit is used for carrying out sparse coding on each image slice in the detail layer of the currently estimated high-resolution image according to the dictionary; an image updating unit that updates a detail layer of a currently estimated high-resolution image; and the super-resolution image reconstruction unit stores the updated high-resolution image estimation when the iterative solution is in a convergence state, and otherwise, circularly and iteratively executes the sparse coding of each image slice in the high-resolution image estimation and the detail layer of the high-resolution image estimation. The invention also provides a corresponding super-resolution image reconstruction method. By adopting the technical scheme of the invention, the resolution ratio of the magnetic resonance image can be obviously improved, the image noises, the blurring and other distortions can be effectively removed, the complex fine structure can be recovered, and the magnetic resonance image has better subjective and objective effects.
Description
Technical Field
The invention belongs to the field of image processing, and particularly relates to a super-resolution image reconstruction method and device.
Background
With the rapid development of the magnetic resonance imaging technology, the resolution, the signal-to-noise ratio and the scanning speed of the magnetic resonance image are greatly improved, but the magnetic resonance imaging equipment special for the neonate is few, and the difficult problem that the magnetic resonance imaging effect of the neonate is not ideal is still not well solved. In magnetic resonance imaging of neonates, a single voxel may contain several different types of tissue, and external disturbances tend to cause agitation of the neonate in the magnetic resonance imaging environment, which factors further reduce the resolution of the captured magnetic resonance image of the neonate.
In order to improve the resolution of magnetic resonance images of neonates and the diagnosis efficiency of doctors, it is common in clinic to perform super-resolution reconstruction on neonate images by using an interpolation method. Image interpolation typically uses known data points on a low-resolution image grid to estimate unknown data points on a high-resolution image grid, and common methods are: nearest neighbor interpolation, polynomial interpolation, spline interpolation, and edge-based interpolation. Although these methods have the advantage of low computational cost, they are prone to distortion phenomena such as halo, ringing, edge smoothing, blurring, and aliasing effects in the reconstructed high-resolution images.
In the prior art, an image super-resolution method based on sparse expression is provided, which mainly adopts a convex optimization regular term to establish an objective function, neglects contrast distribution change of white matter and gray matter brightness of a newborn brain, and does not fully consider distortion phenomena such as large noise, low resolution, partial volume effect and the like of a newborn brain magnetic resonance image, so that the reconstructed high-resolution newborn magnetic resonance image has smooth edges and insufficient resolution. Furthermore, super resolution methods based on sparse representations typically train a super complete dictionary for sparse representation from an internal image or an external image set. But the internal images are from input degraded low resolution images and the external images are from a general large scale image set, and the global and local patterns in these internal and external images may not be similarly compatible with the target high resolution, resulting in target high resolution image reconstruction errors. The existing super-resolution method based on deep learning needs a large number of image sets and corresponding labels to train a convolutional neural network model or generate a confrontation network model, but in reality, magnetic resonance images of newborns are not easy to acquire and are small in number, and modeling of longitudinal image priori knowledge of the same individual or different individuals is not considered, so that the super-resolution performance of the training model is restricted. Therefore, the anatomical accuracy and resolution of the magnetic resonance images of the neonate reconstructed by the existing super-resolution method are still not ideal.
Disclosure of Invention
The invention aims to solve the defects of the prior art, a dictionary is trained by utilizing high-frequency detail layers of longitudinal magnetic resonance images (such as infant images) of different individuals, a super-resolution model of the magnetic resonance images of the newborn based on residual structured sparse expression is established for the input magnetic resonance images of the newborn with low resolution, so that image noise is inhibited, the super-resolution reconstruction performance of the magnetic resonance images of the newborn is improved, and the clinical diagnosis and treatment application of the magnetic resonance imaging technology in the field of imaging of the newborn are further promoted.
Super-resolution image reconstruction apparatus comprising:
the training sample acquisition unit is used for acquiring a training sample, and the training sample comprises high-resolution infant magnetic resonance images of a plurality of different individuals;
the dictionary construction unit is used for constructing a dictionary, and the dictionary comprises K sub-dictionaries;
the obtaining of the sub-dictionary comprises the following steps:
clustering a detail layer of a training sample to obtain K class groups, performing eigenvalue decomposition on a covariance matrix of each class group to obtain a unitary matrix formed by eigenvectors of each class group, wherein the unitary matrix formed by the eigenvectors of each class group is a sub-dictionary;
the low-resolution image input unit is used for inputting any magnetic resonance image of the newborn as a low-resolution image, carrying out image interpolation on the low-resolution image to initialize a high-resolution image and obtaining a currently estimated high-resolution image;
the sparse coding unit is used for dividing the currently estimated high-resolution image into a basic layer and a detail layer by adopting image double-layer representation, extracting all image slices of the currently estimated high-resolution image detail layer, and carrying out sparse coding on each image slice in the detail layer of the currently estimated high-resolution image according to a dictionary;
the image updating unit is used for updating the detail layer of the estimated high-resolution image, namely updating the estimated high-resolution image, for each image slice in the detail layer of the current estimated high-resolution image;
the super-resolution image reconstruction unit is used for storing the updated high-resolution image estimation when the updated high-resolution image estimation and the sparse coding of each image slice in the current estimated high-resolution image detail layer are in a convergence state; if not, taking the updated high-resolution image estimation as the current estimated high-resolution image to enter the next iteration so as to continuously carry out updating calculation on the high-resolution image estimation and the sparse coefficient of each image slice in the current estimated high-resolution image detail layer;
and estimating the updated high-resolution image to be the super-resolution image.
Further, the sparse coding unit divides the current estimated high resolution image into a base layer and a detail layer, and comprises:
and filtering the current estimated high-resolution image to obtain a base layer, wherein the difference value between the current estimated high-resolution image and the base layer is a detail layer.
Further, the sparse coding unit sparsely codes each image slice in the detail layer of the currently estimated high-resolution image according to the dictionary, and the method comprises the following steps:
step 41, selecting one image slice from all image slices of the currently estimated high resolution image detail layer as the target image slice eiWhere I1, 2,, I is the number of all image slices of the current estimated high resolution image detail layer;
step 42, selecting a sub-dictionary D from the K sub-dictionarieskAccording to sub-dictionary DkFor the target image eiCarrying out sparse coding and obtaining a sparse coefficient alphaiWherein K is 1, 2.., K; namely, performing sparse coding on each image slice of the detail layer of the currently estimated high-resolution image according to the dictionary to obtain a sparse coefficient;
selecting one sub-dictionary D from K sub-dictionarieskSaid sub-dictionary DkShould satisfy the corresponding family center and target image eiHas the smallest euclidean distance.
Further, when the updated high-resolution image estimation in the super-resolution image reconstruction unit and the sparse coding of each image slice in the estimated high-resolution image detail layer are in a convergence state, the following formula (1) is satisfied:
in the formula (1), I is 1,2, and, I, eta are auxiliary parameters, c is more than or equal to 0, c represents parameters of a parameterized non-convex penalty term,showing the target image e after the t-th iteration correctioniThe coefficient of sparsity of (a) is,is alphai (t)Non-local mean of, alphai (t)Showing the target image e after the t-th iteration correctioniThe sparse coefficient of (d);
γi (t)=αi (t)-μi (t)+ηDk THT(Y-H(Z(t)+Dkαi (t))),Dkk is the kth sub-dictionary, K is 1,2(t)A base layer representing the high resolution neonatal image estimate updated at the t-th iteration;
for the target image e after the t iteration correctioniThe regularization parameter of (a) is,δ(t)is the noise variance of the high-resolution image estimate after the t-th iteration update;for the t-th iteration sparse coefficientStandard deviation of (2).
Further, the updated high resolution image estimation in the super resolution image reconstruction unitWherein alpha is(t+1)Sparse coefficient matrix, alpha, of detail layer estimated for the t-th iteratively corrected high resolution image(t+1)={α1 (t+1),α2 (t+1),...αi (t+1),...,αI (t+1)D is a dictionary, D ═ D1,D2,...,Dk,..,DK}。
The invention also provides a super-resolution image reconstruction method, which comprises the following steps:
step 1, obtaining a training sample, wherein the training sample comprises high-resolution infant magnetic resonance images of a plurality of different individuals;
step 2, constructing a dictionary, wherein the dictionary comprises K sub-dictionaries;
the obtaining of the sub-dictionary comprises the following steps:
clustering a detail layer of a training sample to obtain K class groups, performing eigenvalue decomposition on a covariance matrix of each class group to obtain a unitary matrix formed by eigenvectors of each class group, wherein the unitary matrix formed by the eigenvectors of each class group is a sub-dictionary;
step 3, inputting any low-resolution magnetic resonance image of the newborn as a low-resolution image, and initializing a high-resolution image for the low-resolution image by utilizing image interpolation to obtain a currently estimated high-resolution image;
step 4, dividing the currently estimated high-resolution image into a basic layer and a detail layer by adopting image double-layer representation, extracting all image slices of the currently estimated high-resolution image detail layer, and carrying out sparse coding on each image slice in the detail layer of the currently estimated high-resolution image according to a dictionary to obtain a corresponding sparse coefficient;
step 5, updating the detail layer of the currently estimated high-resolution image according to the sparse coefficient of each image slice in the detail layer of the currently estimated high-resolution image, namely updating the currently estimated high-resolution image;
step 6, when the updated current estimated high-resolution image and the sparse code of each image slice in the detail layer of the high-resolution image estimation are in a convergence state, storing the updated high-resolution image estimation; if not, taking the updated high-resolution image estimation as the current estimated high-resolution image to enter the next iteration so as to continuously update and calculate the current estimated high-resolution image and the sparse coefficient of each image slice in the current estimated high-resolution image detail layer;
and estimating the updated high-resolution image to be the super-resolution image.
Further, the step 4 of dividing the current estimated high resolution image into a base layer and a detail layer includes:
and filtering the current estimated high-resolution image to obtain a base layer, wherein the difference value between the current estimated high-resolution image and the base layer is a detail layer.
Further, in step 4, sparsely encoding each image slice in the detail layer of the currently estimated high resolution image according to the dictionary and obtaining a corresponding sparse coefficient, including:
step 41, selecting one image slice from all image slices of the currently estimated high resolution image detail layer as the target image slice eiWhere I1, 2,, I is the number of all image slices of the current estimated high resolution image detail layer;
step 42, selecting a sub-dictionary D from the K sub-dictionarieskAccording to sub-dictionary DkFor the target image eiCarrying out sparse coding and obtaining a sparse coefficient alphaiWherein K is 1, 2.., K; namely, the sparse coding of each image slice of the detail layer of the current estimated high-resolution image according to the dictionary is completed.
Further, when the high resolution image estimation updated in step 6 and the sparse coding of each image slice in the detail layer of the high resolution image estimation are in a convergence state, formula (1) is satisfied:
in the formula (1), I is 1,2, and, I, eta are auxiliary parameters, c is more than or equal to 0, c represents parameters of a parameterized non-convex penalty term,showing the target image e after the t iteration correctioniThe coefficient of sparsity of (a) is,is alphai (t)Non-local mean of, alphai (t)Showing the target image e after the t iteration correctioniThe sparse coefficient of (d);
γi (t)=αi (t)-μi (t)+ηDk THT(Y-H(Z(t)+Dkαi (t))),Dkk is the kth sub-dictionary, K is 1,2(t)A base layer representing the high resolution neonatal image estimate updated at the t-th iteration;
for the target image e after the t iteration correctioniThe regularization parameter of (a) is,δ(t)is the noise variance of the high-resolution image estimate after the t-th iteration update;for the t-th iteration sparse coefficientStandard deviation of (2).
Further, the updated high resolution image estimate in step 6Wherein alpha is(t+1)Sparse coefficient matrix, alpha, of detail layer estimated for the t-th iteratively corrected high resolution image(t+1)={α1 (t+1),α2 (t+1),...αi (t+1),...,αI (t+1)D is a dictionary, D ═ D1,D2,...,Dk,..,DK}。
Compared with the prior art, the invention has the following technical characteristics:
the method can reconstruct the high-resolution magnetic resonance image of the newborn, effectively remove image noise, blur and other distortions, recover a complex fine structure, have better subjective and objective effects and generally exceed the most advanced image super-resolution method in the prior art.
Drawings
FIG. 1 is a schematic diagram of a super-resolution image reconstruction apparatus;
FIG. 2 is a flow chart of the present invention;
FIG. 3(a) is a comparison graph (one) of the SNR experimental results; FIG. 3(b) is a graph comparing the results of SSIM; FIG. 3(c) is a comparison graph of SNR experimental results; fig. 3(d) is a graph comparing the results of SSIM experiments.
Detailed Description
In order that the invention may be more clearly understood, the invention will now be described in further detail with reference to the accompanying drawings and examples.
Example 1
The present embodiment provides a super-resolution image reconstruction apparatus, as shown in fig. 1, including:
the training sample acquisition unit is used for acquiring a training sample, and the training sample comprises high-resolution infant magnetic resonance images of a plurality of different individuals;
the dictionary construction unit is used for constructing a dictionary, and the dictionary comprises K sub-dictionaries;
the obtaining of the sub-dictionary comprises the following steps:
clustering a detail layer of a training sample to obtain K class groups, performing eigenvalue decomposition on a covariance matrix of each class group to obtain a unitary matrix formed by eigenvectors of each class group, wherein the unitary matrix formed by the eigenvectors of each class group is a sub-dictionary;
for a low-resolution newborn magnetic resonance image of a certain individual, a group of infant magnetic resonance images of other individuals are selected to train a sparse modeling dictionary for image super-resolution reconstruction in an off-line mode.
The invention provides a method for classifying children images in a training sample into a detail layer and a base layer, filtering and smoothing are carried out on each child image, for example, a Gaussian filter is adopted, the filtered smooth image is recorded as the base layer, the difference value between the image before filtering and the smooth image is recorded as the detail layer, then the method divides the detail layers of all the child images in the training sample into K families through K mean value clustering, and obtains the family center of each family.
Specifically, the objective function of K-means clustering is defined as follows:
wherein K is the number of clusters, mkIs the kth cluster center or class SkMean value of the middle image slice, r represents the average value from a set of baby images L ═ (L)1,L2,…,LN) Set of image slices in extracted detail layer R, S ═ { S ═ S1,S2,…,SKDenotes a set of class families. The present invention solves the above optimization problem using an iterative refinement method with fast convergence local optima. For each family S of detail image sliceskThe invention firstly adopts the eigenvalue to decompose the covariance matrix Sk Then, the feature vectors are formed into corresponding sub-dictionary DkSatisfy the formula
Where d is the eigenvector corresponding to the eigenvalue ρ and I is the identity matrix. By collecting the trained sub-dictionaries, the invention constructs the whole overcomplete dictionary D ═ D1,D2,…,DK}。
A low-resolution image input unit for inputting any low-resolution magnetic resonance image of the newborn as a low-resolution image, and initializing the low-resolution image with image interpolation to obtain a high-resolution image estimation;
due to the fact that the neonatal brain magnetic resonance image has distortion phenomena such as large noise, low resolution, partial volume effect and the like, the high-resolution neonatal magnetic resonance image estimated by the interpolation method is smooth in edge and insufficient in resolution, and therefore sparse modeling needs to be conducted on the currently estimated high-resolution image to obtain the super-resolution image with high precision and ideal resolution.
The sparse coding unit is used for dividing the currently estimated high-resolution image into a base layer and a detail layer, extracting all image slices of the currently estimated high-resolution image detail layer, and performing sparse coding on each image slice in the currently estimated high-resolution image detail layer according to a dictionary to obtain a corresponding sparse coefficient;
in the invention, the current estimated high-resolution image is filtered to obtain a base layer, and the difference value between the current estimated high-resolution image and the base layer is a detail layer.
The method improves the performance of similarity measurement by replacing the image with the detail layer, recovers the high-frequency components of the high-resolution image by using the sparse coding of the detail layer, can effectively remove image noise, blur and other distortions, and particularly can achieve a good recovery effect on complex fine structures.
In the invention, the sparse coding unit sparsely codes each image slice in the detail layer of the current estimated high-resolution image according to the dictionary and obtains the corresponding sparse coefficient, and the method comprises the following steps:
step 41, selecting one image slice from all image slices of the currently estimated high resolution image detail layer as the target image slice eiWhere I1, 2,, I is the number of all image slices of the current estimated high resolution image detail layer;
step 42, selecting a sub-dictionary D from the K sub-dictionarieskAccording to sub-dictionary DkFor the target image eiCarrying out sparse coding and obtaining a sparse coefficient alphaiWherein K is 1, 2.., K; namely, performing sparse coding on each image slice of the detail layer of the currently estimated high-resolution image according to the dictionary to obtain a sparse coefficient;
selecting one sub-dictionary D from K sub-dictionarieskSaid sub-dictionary DkShould satisfy the corresponding family center and target image eiHas the smallest euclidean distance.
In this embodiment, the image slices are all 5 × 5 × 5 voxel image slices.
The image updating unit is used for updating the detail layer of the currently estimated high-resolution image according to the sparse coefficient of each image slice in the detail layer of the currently estimated high-resolution image, namely updating the high-resolution image estimation;
the super-resolution image reconstruction unit is used for storing the updated high-resolution image estimation when the updated high-resolution image estimation and the sparse coding of each image slice in the detail layer of the high-resolution image estimation are in a convergence state; if not, taking the updated high-resolution image estimation as the current estimated high-resolution image and entering the next iteration to continuously update and calculate the sparse coefficient of each image slice in the detail layer of the high-resolution image estimation and the high-resolution image estimation; and finally, estimating the updated high-resolution image to be the super-resolution image.
In order to obtain accurate estimation of the sparse coefficient of an original pure image X from an observed low-resolution magnetic resonance image Y of a newborn, the invention provides a reconstruction model of a magnetic resonance super-resolution image of the newborn with residual structured sparse expression, which comprises the following steps:
s.t.X=Z+E,E=Dα,
wherein alpha isYRepresenting sparse coefficients, D, estimated from an observed low resolution neonatal magnetic resonance image YLIs an overcomplete dictionary which is learnt from a detail layer set of a baby magnetic resonance image set L, Z and E are respectively a basic layer and a detail layer of which X is decomposed, and alphaiIs an image slice E centered at coordinate i within detail layer EiY is the observed low resolution neonatal magnetic resonance image, X is the original high resolution image, H is the degradation matrix used to describe blurring and down-sampling, and λ is a penalty parameter. c is a parameter for controlling the non-convex penalty term phi, and is not less than 0. Subsequently, the invention will select the appropriate parameter c to ensure that the optimization problem is a convex function, and derive a closed solution to the image super-resolution problem. The invention adopts partial secondary penalty function asFor non-convex regularization terms containing parameterization:
wherein, muiIs an image slice E in a slice E of a magnetic resonance image of a newborniOf a group of similar image slices egOf (a) sparse coefficient alphagIs calculated as a non-local weighted average of (a). In particular, muiIs defined as follows:
wherein the weight functionh is a parameter for controlling the rate of decay, W is a normalization parameter and satisfies
Training dictionary D ═ D off-line1,D2,…,DKWith the known premise, the optimization problem about the image super-resolution can be simplified into another form:
theorem: if λ > 0 and 0 ≦ c < 1/λ, then the objective function is a strictly convex function.
The invention uses an iterative contraction algorithm to solve the sparse coding optimization problem:
when the high-resolution image estimation after updating in the super-resolution image reconstruction unit and the sparse coding of each image slice in the detail layer of the high-resolution image estimation are in a convergence state, the method satisfies the following formula (1):
in the formula (1), I is 1,2, wherein, I and eta are auxiliary parameters, c is more than or equal to 0, c represents parameters of a parameterized non-convex penalty term,representing the target image e after the t iteration updateiThe coefficient of sparsity of (a) is,is alphai (t)Non-local mean of, alphai (t)Representing the target image e after the t iteration updateiThe sparse coefficient of (d);
γi (t)=αi (t)-μi (t)+ηDk THT(Y-H(Z(t)+Dkαi (t))),Dkk is the kth sub-dictionary, K is 1,2,.., K, H is the degeneration matrix, Y is the input low resolution neonatal magnetic resonance image, Z(t)A base layer representing the high resolution neonatal image estimate updated at the t-th iteration;
for the target image e after the t iteration correctioniThe regularization parameter of (a) is,δ(t)is the noise variance of the currently estimated high resolution image after the t-th iteration update;for the t-th iteration sparse coefficientStandard deviation of (2).
As can be seen from the formula (1), the invention adopts the non-convex regular term containing parameters to construct the sparse model of the image super-resolution problem, and adopts the strict convex function solution in the prior art to provide an optimization method.
Updated currently estimated high resolution image in super resolution image reconstruction unitWherein alpha is(t+1)Sparse coefficient matrix, alpha, of detail layer estimated for the t +1 th iteratively corrected high resolution image(t+1)={α1 (t+1),α2 (t+1),...αi (t+1),...,αI (t+1)D is a dictionary, D ═ D1,D2,...,Dk,..,DK}。
Example 2
The embodiment also provides a super-resolution image reconstruction method, as shown in fig. 1, including:
step 1, obtaining a training sample, wherein the training sample comprises high-resolution infant magnetic resonance images of a plurality of different individuals;
step 2, constructing a dictionary, wherein the dictionary comprises K sub-dictionaries;
the obtaining of the sub-dictionary comprises the following steps:
clustering a detail layer of a training sample to obtain K class groups, performing eigenvalue decomposition on a covariance matrix of each class group to obtain a unitary matrix formed by eigenvectors of each class group, wherein the unitary matrix formed by the eigenvectors of each class group is a sub-dictionary;
for a low-resolution newborn magnetic resonance image of a certain individual, the method selects a group of infant magnetic resonance images of other individuals to train a dictionary of an image super-resolution reconstruction sparse model.
The invention provides a method for classifying the images of children in a training sample into a detail layer and a basic layer, filtering and smoothing are carried out on each image of children, for example, a Gaussian filter is adopted, the filtered smoothed image is taken as the basic layer, the difference value between the image before filtering and the smoothed image is taken as the detail layer, then the invention divides the detail layers of all the images in the training sample into K families through K mean value clustering, and the family center of each family is obtained.
Specifically, the objective function of K-means clustering is defined as follows:
wherein K is the number of clusters, mkIs the kth cluster center or class SkMean value of the middle image slice, r represents the average value from a set of baby images L ═ (L)1,L2,…,LN) Set of image slices in extracted detail layer R, S ═ { S ═ S1,S2,…,SKDenotes a set of class families. The present invention solves the above optimization problem using an iterative refinement method with fast convergence local optima. For each family S of detail image sliceskThe invention firstly adopts the eigenvalue to decompose the covariance matrix Sk Then, the feature vectors are formed into corresponding sub-dictionary DkTo make it satisfy
Where d is the eigenvector corresponding to the eigenvalue ρ and I is the identity matrix. By collecting these trained sub-dictionaries, the present invention constructs a complete overcomplete dictionary, D ═ D1,D2,…,DK}。
Step 3, inputting any low-resolution magnetic resonance image of the newborn as a low-resolution image, and initializing high-resolution image estimation on the low-resolution image by utilizing image interpolation;
because the neonatal brain magnetic resonance image has distortion phenomena such as large noise, low resolution, partial volume effect and the like, the edge of the high-resolution neonatal magnetic resonance image estimated by using an interpolation method is smooth and the resolution is insufficient, and therefore the current estimated high-resolution image needs to be subjected to sparse coding to obtain a super-resolution image with high precision and ideal resolution.
Step 4, dividing the currently estimated high-resolution image into a base layer and a detail layer, extracting all image slices of the currently estimated high-resolution image detail layer, and performing sparse coding on each image slice in the currently estimated high-resolution image detail layer according to a dictionary to obtain a sparse coefficient;
in the invention, the current estimated high-resolution image is filtered to obtain a base layer, and the difference value between the current estimated high-resolution image and the base layer is a detail layer.
The method improves the performance of similarity measurement by replacing the images of the infants by the detail layer, recovers the high-frequency components of the high-resolution images by the sparse coding of the detail layer, can effectively remove the distortion such as noise, blur and the like of the images of the newborns, and particularly can well recover the complex fine structure.
In the invention, in step 4, each image slice in the detail layer of the current estimated high-resolution image is sparsely encoded according to the dictionary and a sparse coefficient is obtained, which comprises the following steps:
step 41, selecting one image slice from all image slices of the currently estimated high resolution image detail layer as the target image slice eiWhere I1, 2,, I is the number of all image slices of the current estimated high resolution image detail layer;
step 42, selecting a sub-dictionary D from the K sub-dictionarieskAccording to sub-dictionary DkFor the target image eiCarrying out sparse coding and obtaining corresponding sparse coefficient alphaiWherein K is 1, 2.., K; namely, finishing the sparse coding of each image slice of the detail layer of the currently estimated high-resolution image according to the dictionary;
selecting one sub-dictionary D from K sub-dictionarieskSaid sub-dictionary DkShould satisfy the corresponding family center and target image eiHas the smallest euclidean distance.
In this embodiment, the image slices are all 5 × 5 × 5 voxel image slices.
Step 5, updating the detail layer of the currently estimated high-resolution image according to the obtained sparse coefficient of each image slice in the detail layer of the currently estimated high-resolution image, namely updating the high-resolution image estimation;
step 6, when the updated high-resolution image estimation and the sparse coding of each image slice in the detail layer of the high-resolution image estimation are in a convergence state, storing the updated high-resolution image estimation; otherwise, taking the updated high-resolution image estimation as the current estimation high-resolution image and entering the next iteration to continuously carry out sparse coding on the current estimation high-resolution image and each image slice in the current estimation high-resolution image detail layer; and finally, estimating the updated high-resolution image to be the super-resolution image.
In order to obtain accurate estimation of the sparse coefficient of an original pure image X from an observed low-resolution magnetic resonance image Y of a newborn, the invention provides a reconstruction model of a magnetic resonance super-resolution image of the newborn with residual structured sparse expression, which comprises the following steps:
s.t.X=Z+E,E=Dα,
wherein alpha isYRepresenting sparse coefficients, D, estimated from an observed low resolution neonatal magnetic resonance image YLIs an overcomplete dictionary which is learnt from a detail layer set of a baby magnetic resonance image set L, Z and E are respectively a basic layer and a detail layer of which X is decomposed, and alphaiIs an image slice E centered at coordinate i within detail layer EiY is the observed low resolution neonatal magnetic resonance image, X is the original high resolution image, H is the degradation matrix used to describe blurring and down-sampling, and λ is a penalty parameter. c is a parameter for controlling the non-convex penalty term phi, and is not less than 0. Subsequently, the invention will select the appropriate parameter c to ensure that the optimization problem is a convex function, and derive a closed solution to the image super-resolution problem. The invention adopts partial secondary punishment function as non-convex regular term:
wherein, muiIs an image slice E in a slice E of a magnetic resonance image of a newborniOf a group of similar image slices egOf (a) sparse coefficient alphagIs calculated as a non-local weighted average of (a). In particular, muiIs defined as follows:
wherein the weight functionh is a parameter for controlling the rate of decay, W is a normalization parameter and satisfies
Training dictionary D ═ D off-line1,D2,…,DKWith the known premise, the optimization problem about the image super-resolution can be simplified into another form:
theorem: if λ > 0 and 0 ≦ c < 1/λ, then the objective function is a strictly convex function.
The invention uses an iterative contraction algorithm to solve the sparse coding optimization problem:
in the present invention, when the high resolution image estimation updated in step 6 and the sparse coding of each image slice in the detail layer of the high resolution image estimation are in a convergence state, formula (1) is satisfied:
in the formula (1), I is 1,2, and, I, eta are auxiliary parameters, c is more than or equal to 0, and c represents parameterized non-convex penalty termThe parameters are set to be in a predetermined range,showing the target image e after the t-th iteration correctioniThe coefficient of sparsity of (a) is,is alphai (t)Non-local mean of, alphai (t)Showing the target image e after the t iteration correctioniThe sparse coefficient of (d);
γi (t)=αi (t)-μi (t)+ηDk THT(Y-H(Z(t)+Dkαi (t))),Dkk is the kth sub-dictionary, K is 1,2(t)A base layer representing the high resolution neonatal image estimate updated at the t-th iteration;
δ(t)is the noise variance of the currently estimated high resolution image after the t-th iteration update;for the t-th iteration sparse coefficientStandard deviation of (2).
As can be seen from the formula (1), the invention adopts the non-convex regular term containing parameters to construct the sparse model of the image super-resolution problem, and adopts the strict convex function solution in the prior art to provide an optimization method.
Updated high resolution image estimation in step 6Wherein alpha is(t+1)For the high-resolution image corrected for the t-th iterationSparse coefficient matrix, alpha, of the estimated detail layer(t+1)={α1 (t+1),α2 (t+1),...αi (t+1),...,αI (t+1)D is a dictionary, D ═ D1,D2,...,Dk,..,DK}。
The experimental results are as follows:
fig. 3(a) and 3(b) are comparison graphs of SNR and SSIM experimental results obtained by performing super-resolution reconstruction on 24 magnetic resonance T1 images of a newborn through a Spline interpolation method (Spline), a non-local mean upsampling method (NLMU), a low rank total variation method (LRTV) and the RSSR method provided by the present invention, respectively.
Fig. 3(c) and 3(d) are graphs comparing experimental results of SNR and SSIM obtained by performing super-resolution reconstruction on 24 magnetic resonance T2 images of the newborn respectively by Spline interpolation (Spline), non-local mean upsampling (NLMU), Low Rank Total Variation (LRTV) and RSSR method provided by the present invention.
Claims (2)
1. The super-resolution image reconstruction device is characterized by comprising:
the training sample acquisition unit is used for acquiring a training sample, and the training sample comprises high-resolution infant magnetic resonance images of a plurality of different individuals;
the dictionary construction unit is used for constructing a dictionary, and the dictionary comprises K sub-dictionaries;
the obtaining of the sub-dictionary comprises the following steps:
clustering a detail layer of a training sample to obtain K class groups, performing eigenvalue decomposition on a covariance matrix of each class group to obtain a unitary matrix formed by eigenvectors of each class group, wherein the unitary matrix formed by the eigenvectors of each class group is a sub-dictionary;
the low-resolution image input unit is used for inputting any magnetic resonance image of the newborn as a low-resolution image, carrying out image interpolation on the low-resolution image to initialize a high-resolution image and obtaining a current estimated high-resolution image;
the sparse coding unit is used for dividing the current estimated high-resolution image into a basic layer and a detail layer, extracting all image slices of the detail layer of the current estimated high-resolution image, and carrying out sparse coding on each image slice in the detail layer of the current estimated high-resolution image according to a dictionary;
the image updating unit is used for updating the detail layer of the estimated high-resolution image, namely updating the estimated high-resolution image, for each image slice in the detail layer of the current estimated high-resolution image;
the super-resolution image reconstruction unit is used for storing the updated high-resolution image estimation when the updated high-resolution image estimation and the sparse coding of each image slice in the current estimated high-resolution image detail layer are in a convergence state; if not, taking the updated high-resolution image estimation as the current estimated high-resolution image to enter the next iteration so as to continuously carry out updating calculation on the high-resolution image estimation and the sparse coefficient of each image slice in the current estimated high-resolution image detail layer;
estimating the updated high-resolution image to be a super-resolution image;
the sparse coding unit divides the current estimated high-resolution image into a base layer and a detail layer, and comprises the following steps:
filtering the current estimated high-resolution image to obtain a base layer, wherein the difference value between the current estimated high-resolution image and the base layer is a detail layer;
the sparse coding unit sparsely codes each image slice in the detail layer of the current estimated high-resolution image according to the dictionary, and comprises the following steps:
step 41, selecting one image slice from all image slices of the currently estimated high resolution image detail layer as the target image slice eiWhere I1, 2,, I is the number of all image slices of the current estimated high resolution image detail layer;
step 42, selecting a sub-dictionary D from the K sub-dictionarieskAccording to sub-dictionary DkFor the target image eiCarrying out sparse coding and obtaining a sparse coefficient alphaiWherein K is 1, 2.., K; namely, finishing the sparse coding of each image slice of the detail layer of the current estimated high-resolution image according to the dictionary and obtaining the sparse codingA thinning coefficient;
selecting one sub-dictionary D from K sub-dictionarieskSaid sub-dictionary DkShould satisfy the corresponding family center and target image eiHas the minimum Euclidean distance;
when the updated high-resolution image estimation in the super-resolution image reconstruction unit and the sparse coding of each image slice in the estimated high-resolution image detail layer are in a convergence state, the method satisfies the following formula (1):
in the formula (1), I is 1,2, and, I, eta are auxiliary parameters, c is more than or equal to 0, c represents parameters of a parameterized non-convex penalty term,showing the target image e after the t-th iteration correctioniThe coefficient of sparsity of (a) is,is alphai (t)Non-local mean of, alphai (t)Showing the target image e after the t-th iteration correctioniThe sparse coefficient of (d);
γi (t)=αi (t)-μi (t)+ηDk THT(Y-H(Z(t)+Dkαi (t))),Dkk is the kth sub-dictionary, K is 1,2(t)A base layer representing the high resolution neonatal image estimate updated at the t-th iteration;
for the target image e after the t iteration correctioniThe regularization parameter of (a) is,δ(t)is the noise variance of the high-resolution image estimate after the t-th iteration update;for the t-th iteration sparse coefficientA standard deviation of (d);
updated high resolution image estimation in super resolution image reconstruction unitWherein alpha is(t+1)Sparse coefficient matrix, alpha, of detail layer estimated for the t-th iteratively corrected high resolution image(t+1)={α1 (t+1),α2 (t+1),...αi (t+1),...,αI (t+1)D is a dictionary, D ═ D1,D2,...,Dk,..,DK}。
2. The super-resolution image reconstruction method is characterized by comprising the following steps:
step 1, obtaining a training sample, wherein the training sample comprises high-resolution infant magnetic resonance images of a plurality of different individuals;
step 2, constructing a dictionary, wherein the dictionary comprises K sub-dictionaries;
the obtaining of the sub-dictionary comprises the following steps:
clustering a detail layer of a training sample to obtain K class groups, performing eigenvalue decomposition on a covariance matrix of each class group to obtain a unitary matrix formed by eigenvectors of each class group, wherein the unitary matrix formed by the eigenvectors of each class group is a sub-dictionary;
step 3, inputting any low-resolution magnetic resonance image of the newborn as a low-resolution image, and initializing a high-resolution image for the low-resolution image by utilizing image interpolation to obtain a currently estimated high-resolution image;
step 4, dividing the currently estimated high-resolution image into a basic layer and a detail layer by adopting image double-layer representation, extracting all image slices of the currently estimated high-resolution image detail layer, and carrying out sparse coding on each image slice in the detail layer of the currently estimated high-resolution image according to a dictionary to obtain a corresponding sparse coefficient;
step 5, updating the detail layer of the currently estimated high-resolution image according to the sparse coefficient of each image slice in the detail layer of the currently estimated high-resolution image, namely updating the currently estimated high-resolution image;
step 6, when the updated current estimated high-resolution image and the sparse code of each image slice in the detail layer of the high-resolution image estimation are in a convergence state, storing the updated high-resolution image estimation; if not, taking the updated high-resolution image estimation as the current estimated high-resolution image to enter the next iteration so as to continuously update and calculate the current estimated high-resolution image and the sparse coefficient of each image slice in the current estimated high-resolution image detail layer;
estimating the updated high-resolution image to be a super-resolution image;
in step 4, the current estimated high resolution image is divided into a base layer and a detail layer, and the method comprises the following steps:
filtering the current estimated high-resolution image to obtain a base layer, wherein the difference value between the current estimated high-resolution image and the base layer is a detail layer;
in step 4, according to the dictionary, performing sparse coding on each image slice in the detail layer of the currently estimated high-resolution image to obtain a corresponding sparse coefficient, and the method comprises the following steps:
step 41, selecting one image slice from all image slices of the currently estimated high resolution image detail layer as the target image slice eiWhere I1, 2,, I is the number of all image slices of the current estimated high resolution image detail layer;
step 42, selecting a sub-dictionary D from the K sub-dictionarieskAccording to sub-dictionary DkFor the target imageSheet eiCarrying out sparse coding and obtaining a sparse coefficient alphaiWherein K is 1, 2.., K; namely, finishing sparse coding of each image slice of the detail layer of the currently estimated high-resolution image according to the dictionary;
when the high-resolution image estimation updated in the step 6 and the sparse coding of each image slice in the detail layer of the high-resolution image estimation are in a convergence state, the following formula (1) is satisfied:
in the formula (1), I is 1,2, and, I, eta are auxiliary parameters, c is more than or equal to 0, c represents parameters of a parameterized non-convex penalty term,showing the target image e after the t iteration correctioniThe coefficient of sparsity of (a) is,is alphai (t)Non-local mean of, alphai (t)Showing the target image e after the t iteration correctioniThe sparse coefficient of (d);
γi (t)=αi (t)-μi (t)+ηDk THT(Y-H(Z(t)+Dkαi (t))),Dkk is the kth sub-dictionary, K is 1,2(t)A base layer representing the high resolution neonatal image estimate updated at the t-th iteration;
for the target image e after the t iteration correctioniThe regularization parameter of (a) is,δ(t)is the noise variance of the high-resolution image estimate after the t-th iteration update;for the t-th iteration sparse coefficientA standard deviation of (d);
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710835137.2A CN107845065B (en) | 2017-09-15 | 2017-09-15 | Super-resolution image reconstruction method and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710835137.2A CN107845065B (en) | 2017-09-15 | 2017-09-15 | Super-resolution image reconstruction method and device |
Publications (2)
Publication Number | Publication Date |
---|---|
CN107845065A CN107845065A (en) | 2018-03-27 |
CN107845065B true CN107845065B (en) | 2021-06-15 |
Family
ID=61683290
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201710835137.2A Expired - Fee Related CN107845065B (en) | 2017-09-15 | 2017-09-15 | Super-resolution image reconstruction method and device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107845065B (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108532700B (en) * | 2018-03-02 | 2019-06-21 | 广州医谷生物医疗科技有限公司 | The hospital's self-stripping system avoided infection |
CN108764368B (en) * | 2018-06-07 | 2021-11-30 | 西安邮电大学 | Image super-resolution reconstruction method based on matrix mapping |
CN109166093B (en) * | 2018-07-09 | 2020-09-01 | 西北大学 | Image salient region detection method |
CN109145738B (en) * | 2018-07-18 | 2021-07-27 | 浙江工业大学 | Dynamic video segmentation method based on weighted non-convex regularization and iterative re-constrained low-rank representation |
CN109447905B (en) * | 2018-11-06 | 2022-11-18 | 大连海事大学 | Maritime image super-resolution reconstruction method based on discrimination dictionary |
CN110619603B (en) * | 2019-08-29 | 2023-11-10 | 浙江师范大学 | Single image super-resolution method for optimizing sparse coefficient |
CN113344829A (en) * | 2021-04-21 | 2021-09-03 | 复旦大学 | Portable ultrasonic video optimization reconstruction method for multi-channel generation countermeasure network |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105550988A (en) * | 2015-12-07 | 2016-05-04 | 天津大学 | Super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity |
CN106934766A (en) * | 2017-03-15 | 2017-07-07 | 西安理工大学 | A kind of infrared image super resolution ratio reconstruction method based on rarefaction representation |
CN107067367A (en) * | 2016-09-08 | 2017-08-18 | 南京工程学院 | A kind of Image Super-resolution Reconstruction processing method |
-
2017
- 2017-09-15 CN CN201710835137.2A patent/CN107845065B/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105550988A (en) * | 2015-12-07 | 2016-05-04 | 天津大学 | Super-resolution reconstruction algorithm based on improved neighborhood embedding and structure self-similarity |
CN107067367A (en) * | 2016-09-08 | 2017-08-18 | 南京工程学院 | A kind of Image Super-resolution Reconstruction processing method |
CN106934766A (en) * | 2017-03-15 | 2017-07-07 | 西安理工大学 | A kind of infrared image super resolution ratio reconstruction method based on rarefaction representation |
Non-Patent Citations (3)
Title |
---|
基于MG-GMRES算法的图像超分辨率重建;韩玉兵 等;《计算机学报》;20070630;第30卷(第6期);全文 * |
基于稀疏表示的图像超分辨率重建方法研究;徐国明;《中国博士学位论文全文数据库 信息科技辑》;20170515;摘要,正文第1.2.1.4、2.3.3、5.1-5.4节 * |
基于稀疏高频双字典的磁共振成像超分辨率重建法;陈佳文;《电子技术与软件工程》;20170131;全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN107845065A (en) | 2018-03-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107845065B (en) | Super-resolution image reconstruction method and device | |
CN107610194B (en) | Magnetic resonance image super-resolution reconstruction method based on multi-scale fusion CNN | |
Armanious et al. | Unsupervised medical image translation using cycle-MedGAN | |
Wu et al. | Stabilizing deep tomographic reconstruction: Part A. Hybrid framework and experimental results | |
Ravishankar et al. | Data-driven learning of a union of sparsifying transforms model for blind compressed sensing | |
CN108090871B (en) | Multi-contrast magnetic resonance image reconstruction method based on convolutional neural network | |
Trinh et al. | Novel example-based method for super-resolution and denoising of medical images | |
CN106056647B (en) | A kind of magnetic resonance fast imaging method based on the sparse double-deck iterative learning of convolution | |
Yang et al. | Single-image super-resolution reconstruction via learned geometric dictionaries and clustered sparse coding | |
Yu et al. | Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity | |
CN107038730B (en) | Sparse representation image reconstruction method based on Gaussian scale structure block grouping | |
CN107330953B (en) | Non-convex low-rank based dynamic MRI reconstruction method | |
CN105279740A (en) | Image denoising method based on sparse regularization | |
Poddar et al. | Manifold recovery using kernel low-rank regularization: Application to dynamic imaging | |
CN104574456B (en) | A kind of super lack sampling K data imaging method of magnetic resonance based on figure regularization sparse coding | |
CN108765313B (en) | Hyperspectral image denoising method based on intra-class low-rank structure representation | |
Li et al. | MRI reconstruction with interpretable pixel-wise operations using reinforcement learning | |
CN111161184B (en) | Rapid MR image denoising method based on MCP sparse constraint | |
Ongie et al. | A fast algorithm for structured low-rank matrix recovery with applications to undersampled MRI reconstruction | |
CN108898568B (en) | Image synthesis method and device | |
CN114998458A (en) | Undersampled magnetic resonance image reconstruction method based on reference image and data correction | |
CN114581550B (en) | Magnetic resonance imaging down-sampling and reconstruction method based on cross-domain network | |
CN106296583B (en) | Based on image block group sparse coding and the noisy high spectrum image ultra-resolution ratio reconstructing method that in pairs maps | |
CN108573512B (en) | Complex visual image reconstruction method based on depth coding and decoding dual model | |
Hou et al. | PNCS: Pixel-level non-local method based compressed sensing undersampled MRI image reconstruction |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210615 Termination date: 20210915 |
|
CF01 | Termination of patent right due to non-payment of annual fee |