CN109389556B - Multi-scale cavity convolutional neural network super-resolution reconstruction method and device - Google Patents
Multi-scale cavity convolutional neural network super-resolution reconstruction method and device Download PDFInfo
- Publication number
- CN109389556B CN109389556B CN201811113028.0A CN201811113028A CN109389556B CN 109389556 B CN109389556 B CN 109389556B CN 201811113028 A CN201811113028 A CN 201811113028A CN 109389556 B CN109389556 B CN 109389556B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- convolution
- super
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013527 convolutional neural network Methods 0.000 title claims description 25
- 238000013528 artificial neural network Methods 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims description 59
- 239000010410 layer Substances 0.000 claims description 52
- 239000013598 vector Substances 0.000 claims description 19
- 238000011176 pooling Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 15
- 238000007781 pre-processing Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 15
- 238000005070 sampling Methods 0.000 claims description 7
- 230000002776 aggregation Effects 0.000 claims description 6
- 238000004220 aggregation Methods 0.000 claims description 6
- 239000002356 single layer Substances 0.000 claims description 6
- 239000011800 void material Substances 0.000 claims description 6
- 238000011478 gradient descent method Methods 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method and a device for reconstructing a super-resolution of a multi-scale cavity convolution neural network, which can effectively solve the problem of small reception field in the prior art by constructing a cavity convolution super-resolution network, increase the network reception field, and meanwhile, in order to enable the cavity convolution super-resolution network to be suitable for reconstructing super-resolution of different scales, the original image is subjected to data enhancement of different scales, so that the cavity convolution super-resolution network can process images of different multiples, and the cavity convolution super-resolution network can generalize the super-resolution of the multi-scale image.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a method and a device for reconstructing super-resolution of a multi-scale cavity convolutional neural network.
Background
At present, a deep convolutional neural network is widely used in computer vision problems and shows excellent performance in image classification, target detection and other problems. The single-image super-resolution is a typical computer vision problem, and for the reconstruction problem of low-resolution images, dong and the like propose an algorithm for reconstructing the super-resolution of the images by using a convolutional neural network, and the idea is to use the strong learning capability of a network model to learn the end-to-end mapping relationship from low resolution to high resolution so as to reconstruct the super-resolution images. Later, dong et al improved on the basis of the SRCNN method, and reduced the feature map dimensions and convolution kernel size with low resolution images as network input, and proposed a fast super-resolution method that reduced network training parameters. Sh i et al propose a super-resolution method using sub-pixel convolution layers to rearrange the finally obtained feature maps to obtain a high-resolution image, and greatly increase the computation speed. However, the above 3 deep learning methods all belong to a shallow convolutional neural network structure, and all have the problems of small receptive field and very local extracted features, so that the super-resolution effect has certain limitations.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a method and a device for reconstructing the super-resolution of the multi-scale cavity convolutional neural network, which can increase the network receptive field, extract more global characteristics and are suitable for reconstructing the super-resolution of multi-scale images.
The technical scheme adopted by the invention for solving the problems is as follows:
a multi-scale cavity convolutional neural network super-resolution reconstruction method comprises the following steps:
carrying out multi-scale data enhancement on the original image to obtain an enhanced image;
carrying out data preprocessing on the enhanced image to obtain a preprocessed image;
constructing a cavity convolution super-resolution network, and performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network, wherein the parameter needing to be continuously updated in the cavity convolution super-resolution network is theta = { W = 1 ,W 2 ,W 3 ,B 1 ,B 2 ,B 3 In which W is 1 And B 1 Weight and bias variables, W, representing the void convolution kernel 2 Comprises n 2 Each size is n 1 ×f 2 ×f 2 Convolution kernel of (A), B 2 Is a size n 2 Vector of dimensions, W 3 Corresponding to c pieces of size n 2 ×f 3 ×f 3 Convolution kernel of (A), B 3 Is a c-dimensional vector; the reconstructed image is F (Y; theta), and the corresponding original image is X; using the mean square error as a loss function:
wherein n is the number of training samples;
the weight value of the loss function is updated by a random gradient descent method, and the formula is as follows:
where l ∈ {1,2,3}, i is the number of layers and iterations, and η is the learning rate.
Further, performing multi-scale data enhancement on the original image to obtain an enhanced image, including:
taking an original image as an initial training set;
scaling each image in the initial training set in different scales respectively, and forming an intermediate training set by combining the original images;
and respectively rotating each image in the intermediate training set by different angles, and forming a processing training set by combining each image in the intermediate training set, wherein the images in the processing training set are the enhanced images.
Further, the data preprocessing is performed on the enhanced image to obtain a preprocessed image, and the data preprocessing comprises: extracting a brightness component of the enhanced image, cutting the brightness component, and downsampling the cut image according to different scale factors;
up-sampling the down-sampled image according to the corresponding scale factor to generate an initial high-resolution image with the resolution higher than that of the original image;
and cutting the initial high-resolution image and the original image into a high-resolution image block and an original image block with the size of N x N according to the step length M, wherein both M and N are constants.
Further, constructing a cavity convolution super-resolution network comprises the following steps:
replacing the pooling layer in the convolutional neural network and the convolutional layer behind the pooling layer with a hollow convolutional layer.
Further, performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network, wherein the super-resolution reconstruction comprises the following steps:
extracting image characteristics of the original image block and the corresponding high-resolution image block;
establishing a single-layer neural network and learning a nonlinear mapping relation between two image features by using the mapping characteristics of a convolution kernel for the obtained image features of the original image block and the image features of the high-resolution image block;
and carrying out aggregation reconstruction on the mapped high-resolution image blocks to generate a high-resolution image.
The device for storing the super-resolution reconstruction method of the multi-scale cavity convolutional neural network comprises a control module and a storage medium for storing control instructions, wherein the control module reads the control instructions in the storage medium and executes the following operations:
carrying out multi-scale data enhancement on the original image to obtain an enhanced image;
carrying out data preprocessing on the enhanced image to obtain a preprocessed image;
constructing a cavity convolution super-resolution network, and performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network, wherein a parameter needing to be continuously updated in the cavity convolution super-resolution network is theta = { W = 1 ,W 2 ,W 3 ,B 1 ,B 2 ,B 3 In which W is 1 And B 1 Weight and bias variables, W, representing the hole convolution kernel 2 Comprises n 2 Each size is n 1 ×f 2 ×f 2 Convolution kernel of (A), B 2 Is a size n 2 Vector of dimensions, W 3 Corresponding to c pieces of size n 2 ×f 3 ×f 3 Of a convolution kernel of, B 3 Is a c-dimensional vector; the reconstructed image is F (Y; theta), and the corresponding original image is X; using the mean square error as a loss function:
wherein n is the number of training samples;
the weight value of the loss function is updated by a random gradient descent method, and the formula is as follows:
where l ∈ {1,2,3}, i is the number of layers and iterations, and η is the learning rate.
Further, the control module executes operation to perform multi-scale data enhancement on the original image, and when the enhanced image is obtained, the control module comprises the following operations:
taking an original image as an initial training set;
scaling each image in the initial training set in different scales respectively, and forming an intermediate training set by combining the original images;
and respectively rotating each image in the intermediate training set by different angles, and forming a processing training set by combining each image in the intermediate training set, wherein the images in the processing training set are the enhanced images.
Further, the control module executes operation to perform data preprocessing on the enhanced image, and when the preprocessed image is obtained, the control module comprises the following operations:
extracting a brightness component of the enhanced image, cutting the brightness component, and downsampling the cut image according to different scale factors;
up-sampling the down-sampled image according to the corresponding scale factor to generate an initial high-resolution image with the resolution higher than that of the original image;
and cutting the initial high-resolution image and the original image into a high-resolution image block and an original image block with the size of N x N according to the step length M, wherein both M and N are constants.
Further, when the control module executes the operation to construct the cavity convolution super-resolution network, the method comprises the following operations:
replacing the pooling layer in the convolutional neural network and the convolutional layer behind the pooling layer with a hollow convolutional layer.
Further, when the control module executes the operation to perform the super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network, the method comprises the following operations:
extracting image characteristics of the original image block and the corresponding high-resolution image block;
establishing a single-layer neural network and learning a nonlinear mapping relation between two image features by using the mapping characteristics of a convolution kernel for the obtained image features of the original image block and the image features of the high-resolution image block;
and performing aggregation reconstruction on the mapped high-resolution image blocks to generate a high-resolution image.
The invention has the beneficial effects that: a method and a device for reconstructing a super-resolution of a multi-scale cavity convolution neural network are provided, wherein a cavity convolution super-resolution network is constructed, so that the problem of small reception field in the prior art can be effectively solved, the network reception field is increased, and meanwhile, in order to enable the cavity convolution super-resolution network to be suitable for reconstructing super-resolution of different scales, different-scale data enhancement is carried out on an original image, so that the cavity convolution super-resolution network can process images of different size multiples, and the cavity convolution super-resolution network can generalize the multi-scale image super-resolution.
Drawings
The invention is further illustrated with reference to the following figures and examples.
FIG. 1 is a flow chart of a super-resolution reconstruction method of the present invention;
FIG. 2 is a further detailed flow chart of the multiscale data enhancement step of FIG. 1;
FIG. 3 is a flow chart further detailing the steps of data preprocessing of FIG. 1;
FIG. 4 is a flowchart further detailing the step of performing super resolution reconstruction in FIG. 1.
Detailed Description
Referring to fig. 1, an embodiment of the present invention provides a multi-scale cavity convolutional neural network super-resolution reconstruction method, including but not limited to the following steps:
s100, performing multi-scale data enhancement on an original image to obtain an enhanced image;
step S200, carrying out data preprocessing on the enhanced image to obtain a preprocessed image;
and S300, constructing a cavity convolution super-resolution network, and performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network.
In this embodiment, by constructing the cavity convolution super-resolution network, the problem of small reception field in the prior art can be effectively solved, the network reception field is increased, and meanwhile, in order to enable the cavity convolution super-resolution network to be suitable for reconstruction of super-resolution images of different scales, data enhancement of different scales is performed on an original image, so that the cavity convolution super-resolution network can process images of different multiples, and thus the cavity convolution super-resolution network can generalize multi-scale image super-resolution.
Further, referring to fig. 2, in step S100 of this embodiment, performing multi-scale data enhancement on the original image to obtain an enhanced image, including but not limited to the following steps:
step S110, taking the original image as an initial training set;
step S120, zooming each image in the initial training set in different scales respectively, and combining the original images to form an intermediate training set;
step S130, respectively rotating each image in the intermediate training set by different angles, and forming a processing training set by combining each image in the intermediate training set, where the images in the processing training set are the enhanced images.
In this embodiment, 91 images in the Timofte data set in the prior art are used as original images, and the 91 original images are used as an initial training set, and then the initial training set is respectively processed by adopting two ways of scaling and rotating. For example, 91 original images in the initial training set are first down-sampled and scaled to 0.6 times, 0.7 times, 0.8 times and 0.9 times of the original size, and 91 original images are combined to form an intermediate training set, where the number of images in the intermediate training set is 5 times that in the initial training set; and then, respectively rotating each image in the intermediate training set by 90 degrees, 180 degrees and 270 degrees, and combining the images in the original intermediate training set to form a processing training set, wherein the images in the processing training set are the enhanced images. At this time, the number of enhanced images is increased to 20 times the number of images in the initial training set.
Further, referring to fig. 3, in step S200 of this embodiment, the data preprocessing is performed on the enhanced image to obtain a preprocessed image, which includes, but is not limited to, the following steps:
step S210, extracting the brightness component of the enhanced image, cutting the brightness component, and down-sampling the cut image according to different scale factors;
step S220, up-sampling the down-sampled image according to the corresponding scale factor to generate an initial high-resolution image with the resolution higher than that of the original image;
and step S230, cutting the initial high-resolution image and the original image into a high-resolution image block and an original image block with the size of N × N according to the step length M, wherein both M and N are constants.
In this embodiment, after obtaining an enhanced image, first, performing spatial conversion on the enhanced image to obtain a YCbCr spatial image, then extracting a Y nonlinear luminance component of the enhanced image, then clipping the luminance component, and downsampling the clipped image by scale factors of 2,3, and 4, respectively, to obtain a corresponding low-resolution image, then, upsampling the low-resolution image by the same scale factor, respectively, to generate an initial high-resolution image with a resolution higher than that of the original image, and in addition, for matching network input, clipping the initial high-resolution image and the original image into a high-resolution image block and an original image block with a size of 41 × 41 according to a step size of 14, and using the high-resolution image block and the original image block as input data of a hole convolution super-resolution network.
Further, in step S300 of this embodiment, a hollow convolution super-resolution network is constructed, which is mainly formed by replacing the pooling layer in the convolutional neural network and the convolutional layer behind the pooling layer with a hollow convolutional layer. In order to solve the problem that the downsampling operation of the pooling layer in the conventional convolutional neural network causes information loss, the pooling layer and the convolutional layer behind the pooling layer are replaced by a hole convolutional layer, and the hole convolutional layer has the main functions of: how to remove the operation of pooling downsampling without reducing the receptive field of the network, thereby reconstructing the super-resolution image. Therefore, for the operation of reconstructing the super-resolution image, the cavity convolution super-resolution network with the cavity convolution layer has better performance than the traditional convolution neural network.
Further, referring to fig. 4, in step S300 of this embodiment, performing super-resolution reconstruction on the preprocessed image by using a hole convolution super-resolution network, including, but not limited to, the following steps:
step S310, extracting image features of the original image block and the corresponding high-resolution image block.
For applying a convolutional neural network to super-resolution reconstruction, the extraction and representation of image blocks is a convolution process. The convolution operation in the hole convolution super-resolution network is the same as that of the conventional convolution neural network, but the convolution kernel is different from the original one. In order to enlarge the network receptive field and enable the network to learn more global characteristics, the invention adopts the hole convolution kernel to carry out convolution operation, thereby achieving the purpose of extracting the network characteristics. In the step, 9 × 9 convolution kernels with a cavity factor of 2 are used to extract image features of the original image block and the corresponding high-resolution image block, and a feature diagram output by the network is set to 64, that is, an output channel is 64. Specifically, the convolution operation formula is as follows:
F 1 (Y)＝max(0,W 1 * l Y+B 1 )
wherein, W 1 And B 1 Weight and bias variables representing the kernel of the hole convolution l Indicating a hole convolution. W 1 Corresponding to a size of c × f 1 ×f 1 N of (A) to (B) 1 A filter, c being the dimension of the input image, f 1 Is the size of the filter. B is 1 Is n 1 Dimension vectors, and each element corresponds to a convolution kernel. Therefore, the output of the first layer of the hole convolution super-resolution network has n in total according to the principle of convolution 1 A characteristic diagram.
In particular, the hole convolution operation is specifically as follows:
defining a discrete function: f is Z 2 → R, suppose Ω R = [ -R, R] 2 ∩Z 2 K is omega → R is (2r + 1) 2 The discrete convolution operation is defined as:
(F*k)(p)＝∑ s+t＝p
the hole convolution is generally expressed in terms of:
(F* l k)(p)＝∑ s+lt＝p
wherein l is a void factor l Representing the hole convolution, when l =1, the method is a common discrete convolution operation, the network based on the hole convolution supports the exponential increase of the receptive field without losing resolution information, and F is recorded 0 ,F 1 ,...,F n-1 :Z 2 R is a discrete function, k 0 ,k 1 ,...,k n-1 Omega → R is a discrete 3x3 filter, with exponentially growing convolution kernels,
F i+1 ＝F i * 2i k i when i =0, 1.,. N-2
Definition F i+1 The element P receptive field in (A) is as follows: f 0 Can change F i+1 (p) a set of elements of the value. F i+ The size of the P receptive field in 1 is the number of these element sets. Is readily available, F i+1 Large receptive field of each elementIs small as (2) i+2 -1)×(2 i+2 -1)。
And step S320, establishing a single-layer neural network and learning a nonlinear mapping relation between the two image characteristics by using the mapping characteristics of the convolution kernel for the obtained image characteristics of the original image block and the image characteristics of the high-resolution image block.
The operation of the step is mainly realized in the second layer of the network, and the step is mainly to nonlinearly map the high-dimensional vector generated in the first layer of the network to another high-dimensional vector, and the newly generated high-dimensional vectors form another vector set. In this step, the feature maps generated by the 64 channels generated in step S310 are subjected to nonlinear mapping using a 1 × 1 convolution kernel, high-dimensional vectors with low resolution are mapped to a high-dimensional vector space having high-resolution features, and 32 feature maps are generated. Next, the vector generated in the first layer of the network is taken as n 1 Feature maps of dimensions, n in the second layer of the network 1 Vector mapping of dimensions to n 2 And (5) maintaining. Such an operation is equivalent to using n 2 Convolution is performed by convolution kernels with the size of 1 × 1, and the second layer of the network is represented as:
F 2 ＝max(0,W 2 *F 1 (Y)+B 2 )
wherein, W 2 Comprises n 2 Each size is n 1 ×f 2 ×f 2 Of a convolution kernel of, B 2 Is one with the size of n 2 A vector of dimensions. Each output of the second layer of the network will be used to reconstruct a high resolution image block of the image. And finally, reconstructing the 32 channel characteristic graphs into a high-resolution image block through a 5-by-5 convolution kernel, and realizing image super-resolution reconstruction.
And step S330, carrying out aggregation reconstruction on the mapped high-resolution image blocks to generate a high-resolution image.
The operation of the step is mainly realized in the third layer of the network, the step generates the high-resolution image from the high-resolution image set of the second layer of the network, and the generated image is similar to a real original image. In the conventional method, the reconstruction operation generates the final complete high resolution image by averaging the predicted high resolution blocks. In fact, the averaging process can also be regarded as a pre-trained filter convolving on a feature map set. The operation of this step in the third layer of the network is inspired by the convolution operation to obtain the final high-resolution image. The third layer of the network is represented as:
F(Y)＝W 3 *F 2 (Y)+B 3
wherein, W 3 Corresponding to c pieces of size n 2 ×f 3 ×f 3 Of a convolution kernel of, B 3 Is a c-dimensional vector. If the high resolution block is in the image domain, then treating the filter as an average filter; if the high resolution block is other domain, the convolution kernel W 3 The coefficients are projected onto the image domain and averaged.
Then, if a high resolution image is obtained from the above result, the parameter to be continuously updated in the hollow convolution super-resolution network is Θ = { W = { (W) 1 ,W 2 ,W 3 ,B 1 ,B 2 ,B 3 And the reconstructed image is F (Y; theta) and the corresponding original image is X. Using the mean square error as a loss function:
where n is the number of training samples. The weight value of the loss function is updated by a random gradient descent (SGD) method, and the formula is as follows:
where l ∈ {1,2,3}, i is the number of layers and iterations, and η is the learning rate.
Under the same calculation conditions, the hole convolution provides a larger receptive field. When the network layer needs a larger receptive field, but the computational resources are limited and the number or size of the convolution kernels cannot be increased, the hole convolution can be utilized. The hole convolution can integrate multi-scale content information, does not lose resolution and supports exponential increase of receptive field. Meanwhile, content information with different scales is integrated, so that any resolution is added into the existing network structure, and network reusability is facilitated. In the traditional convolutional neural network, a pooling layer is adopted to achieve the purpose of reducing the dimension, but the method can generate certain side effect in the image super-resolution, the pixel size of the pooled feature layer is low, and the feature map can lose certain precision even through upsampling, so that the sensing field of the network is increased by adopting a cavity convolution mode instead of the pooling layer.
In addition, in order to implement the method for reconstructing the super-resolution of the multi-scale cavity convolutional neural network in the above embodiment, an embodiment of the present invention further provides an apparatus for storing the method for reconstructing the super-resolution of the multi-scale cavity convolutional neural network, where the apparatus includes a control module and a storage medium for storing a control instruction, and the control module reads the control instruction in the storage medium and performs the following operations:
carrying out multi-scale data enhancement on the original image to obtain an enhanced image;
carrying out data preprocessing on the enhanced image to obtain a preprocessed image;
and constructing a cavity convolution super-resolution network, and performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network.
Further, the control module executes operation to perform multi-scale data enhancement on the original image, and when an enhanced image is obtained, the method comprises the following operations:
taking an original image as an initial training set;
scaling each image in the initial training set in different scales respectively, and forming an intermediate training set by combining the original images;
and respectively rotating each image in the intermediate training set by different angles, and forming a processing training set by combining each image in the intermediate training set, wherein the images in the processing training set are the enhanced images.
Further, the control module executes operations to perform data preprocessing on the enhanced image, and when a preprocessed image is obtained, the operations include:
extracting a brightness component of the enhanced image, cutting the brightness component, and downsampling the cut image according to different scale factors;
up-sampling the down-sampled image according to the corresponding scale factor to generate an initial high-resolution image with the resolution higher than that of the original image;
and cutting the initial high-resolution image and the original image into a high-resolution image block and an original image block with the size of N x N according to the step length M, wherein both M and N are constants.
Further, when the control module executes the operation to construct the hole convolution super-resolution network, the method comprises the following operations: replacing the pooling layer in the convolutional neural network and the convolutional layer behind the pooling layer with a hollow convolutional layer.
Further, when the control module executes the operation to perform the super-resolution reconstruction on the preprocessed image by using the hole convolution super-resolution network, the method comprises the following operations:
extracting image characteristics of the original image block and the corresponding high-resolution image block;
establishing a single-layer neural network and learning a nonlinear mapping relation between two image features by using the mapping characteristics of a convolution kernel for the obtained image features of the original image block and the image features of the high-resolution image block;
and performing aggregation reconstruction on the mapped high-resolution image blocks to generate a high-resolution image.
While the preferred embodiments of the present invention have been described in detail, it will be understood by those skilled in the art that the foregoing and various other changes, omissions and deviations in the form and detail thereof may be made without departing from the scope of this invention.
Claims (10)
1. A super-resolution reconstruction method of a multi-scale cavity convolution neural network is characterized by comprising the following steps: the method comprises the following steps:
carrying out multi-scale data enhancement on the original image to obtain an enhanced image;
carrying out data preprocessing on the enhanced image to obtain a preprocessed image;
constructing a cavity convolution super-resolution network, and performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network, wherein a convolution operation formula is as follows:
F 1 (Y)＝max(0,W 1 * l Y+B 1 )
wherein, W 1 And B 1 Weight and bias variables representing the kernel of the hole convolution l Representing a hole convolution; w 1 Corresponding to a size of c × f 1 ×f 1 N of (A) to (B) 1 A filter, c being the dimension of the input image, f 1 Is the size of the filter;
defining a discrete function: f is Z 2 → R, assuming Ω R = [ -R, R] 2 ∩Z 2 K is omega → R is (2r + 1) 2 The discrete convolution operation is defined as:
(F*k)(p)＝∑ s+t＝p
the hole convolution is generally expressed in terms of:
(F* l k)(p)＝∑ s+lt＝p
wherein l is a void factor l Representing the hole convolution, when l =1, the method is a common discrete convolution operation, the network based on the hole convolution supports the exponential increase of the receptive field without losing resolution information, and F is recorded 0 ,F 1 ,...,F n-1 :Z 2 K is a discrete function → R 0 ,k 1 ,...,k n-1 Omega → R is a discrete 3x3 filter, with exponentially growing convolution kernels,
F i+1 ＝F i * 2i k i when i =0, 1.,. N-2
Definition F i+1 The element P receptive field in (A) is as follows: f 0 Can change F i+1 (p) a set of elements of a value; f i+1 The size of the receptor field is the number of the element sets to obtain F i+1 The receptive field of each element is (2) i+2 -1)×(2 i+2 -1)；
The parameter needing to be continuously updated in the cavity convolution super-resolution network is theta = { W = 1 ,W 2 ,W 3 ,B 1 ,B 2 ,B 3 In which W is 1 And B 1 Weight and bias variables, W, representing the void convolution kernel 2 Comprises n 2 Each size is n 1 ×f 2 ×f 2 Of a convolution kernel of, B 2 Is a size n 2 Vector of dimensions, W 3 Corresponding to c pieces of size n 2 ×f 3 ×f 3 Of a convolution kernel of, B 3 Is a c-dimensional vector; the reconstructed image is F (Y; theta), and the corresponding original image is X; using the mean square error as a loss function:
wherein n is the number of training samples;
the weight value of the loss function is updated by a random gradient descent method, and the formula is as follows:
where l ∈ {1,2,3}, i is the number of layers and iterations, and η is the learning rate.
2. The method for reconstructing the super-resolution of the multi-scale hole convolutional neural network of claim 1, wherein: the multi-scale data enhancement of the original image to obtain an enhanced image comprises the following steps:
taking an original image as an initial training set;
scaling each image in the initial training set in different scales respectively, and forming an intermediate training set by combining the original images;
and respectively rotating each image in the intermediate training set by different angles, and forming a processing training set by combining each image in the intermediate training set, wherein the images in the processing training set are the enhanced images.
3. The method for reconstructing the super-resolution of the multi-scale hole convolutional neural network of claim 1, wherein: the data preprocessing is performed on the enhanced image to obtain a preprocessed image, and the data preprocessing comprises the following steps:
extracting a brightness component of the enhanced image, cutting the brightness component, and downsampling the cut image according to different scale factors;
up-sampling the down-sampled image according to the corresponding scale factor to generate an initial high-resolution image with the resolution higher than that of the original image;
and cutting the initial high-resolution image and the original image into a high-resolution image block and an original image block with the size of N x N according to the step length M, wherein both M and N are constants.
4. The method for reconstructing the super-resolution of the multi-scale hole convolutional neural network of claim 1, wherein: the method for constructing the cavity convolution super-resolution network comprises the following steps:
replacing the pooling layer in the convolutional neural network and the convolutional layer behind the pooling layer with a hollow convolutional layer.
5. The multi-scale hole convolutional neural network super-resolution reconstruction method of claim 3, wherein: the super-resolution reconstruction of the preprocessed image by using the cavity convolution super-resolution network comprises the following steps:
extracting image characteristics of the original image block and the corresponding high-resolution image block;
establishing a single-layer neural network and learning a nonlinear mapping relation between two image features by using the mapping characteristics of a convolution kernel for the obtained image features of the original image block and the image features of the high-resolution image block;
and carrying out aggregation reconstruction on the mapped high-resolution image blocks to generate a high-resolution image.
6. A device for storing a multi-scale cavity convolution neural network super-resolution reconstruction method is characterized in that: the control module reads the control instructions in the storage medium and executes the following operations:
carrying out multi-scale data enhancement on the original image to obtain an enhanced image;
carrying out data preprocessing on the enhanced image to obtain a preprocessed image;
constructing a cavity convolution super-resolution network, and performing super-resolution reconstruction on the preprocessed image by using the cavity convolution super-resolution network, wherein a convolution operation formula is as follows:
F 1 (Y)＝max(0,W 1 * l Y+B 1 )
wherein, W 1 And B 1 Weight and bias variables representing the kernel of the hole convolution l Representing a hole convolution; w 1 Corresponding to a size of c × f 1 ×f 1 N of (A) to (B) 1 A filter, c being the dimension of the input image, f 1 Is the size of the filter;
defining a discrete function: f is Z 2 → R, suppose Ω R = [ -R, R] 2 ∩Z 2 K is omega → R is (2r + 1) 2 The discrete convolution operation defined as:
(F*k)(p)＝∑ s+t＝p
the hole convolution is generally expressed in terms of:
(F* l k)(p)＝∑ s+lt＝p
wherein l is a void factor l Representing the hole convolution, when l =1, the method is a common discrete convolution operation, the network based on the hole convolution supports the exponential increase of the receptive field without losing resolution information, and F is recorded 0 ,F 1 ,...,F n-1 :Z 2 K is a discrete function → R 0 ,k 1 ,...,k n-1 Omega → R is a discrete 3x3 filter, with exponentially growing convolution kernels,
F i+1 ＝F i * 2i k i when i =0, 1.,. N-2
Definition F i+1 The element P receptive field in (A) is as follows: f 0 Can change F i+1 (p) a set of elements of a value; f i+1 The size of the receptor field is the number of the element sets to obtain F i+1 The receptive field of each element is (2) i+2 -1)×(2 i+2 -1)；
The parameter needing to be continuously updated in the cavity convolution super-resolution network is theta = { W = 1 ,W 2 ,W 3 ,B 1 ,B 2 ,B 3 In which W is 1 And B 1 Weight and bias variables, W, representing the void convolution kernel 2 Comprises n 2 Each size is n 1 ×f 2 ×f 2 Convolution kernel of (A), B 2 Is a size n 2 Vector of dimensions, W 3 Corresponding to c pieces of size n 2 ×f 3 ×f 3 Of a convolution kernel of, B 3 Is a c-dimensional vector; the reconstructed image is F (Y; theta), and the corresponding original image is X; using the mean square error as a loss function:
wherein n is the number of training samples;
the weight value of the loss function is updated by a random gradient descent method, and the formula is as follows:
where l ∈ {1,2,3}, i is the number of layers and iterations, and η is the learning rate.
7. The apparatus of claim 6, wherein: the control module executes operation to perform multi-scale data enhancement on the original image, and when the enhanced image is obtained, the control module comprises the following operations: taking an original image as an initial training set;
zooming each image in the initial training set in different scales respectively, and combining the original images to form an intermediate training set;
and respectively rotating each image in the intermediate training set by different angles, and forming a processing training set by combining each image in the intermediate training set, wherein the images in the processing training set are the enhanced images.
8. The apparatus of claim 6, wherein: the control module executes operation to perform data preprocessing on the enhanced image, and when a preprocessed image is obtained, the control module comprises the following operations: extracting a brightness component of the enhanced image, cutting the brightness component, and downsampling the cut image according to different scale factors;
up-sampling the down-sampled image according to the corresponding scale factor to generate an initial high-resolution image with the resolution higher than that of the original image;
and cutting the initial high-resolution image and the original image into a high-resolution image block and an original image block with the size of N x N according to the step length M, wherein both M and N are constants.
9. The apparatus of claim 6, wherein: when the control module executes the operation to construct the cavity convolution super-resolution network, the method comprises the following operations:
replacing the pooling layer in the convolutional neural network and the convolutional layer behind the pooling layer with a hollow convolutional layer.
10. The apparatus of claim 8, wherein: when the control module executes the operation and utilizes the cavity convolution super-resolution network to carry out the super-resolution reconstruction on the preprocessed image, the control module comprises the following operations:
extracting image characteristics of the original image block and the corresponding high-resolution image block; establishing a single-layer neural network and learning a nonlinear mapping relation between two image features by using the mapping characteristics of a convolution kernel for the obtained image features of the original image block and the image features of the high-resolution image block;
and carrying out aggregation reconstruction on the mapped high-resolution image blocks to generate a high-resolution image.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811113028.0A CN109389556B (en) | 2018-09-21 | 2018-09-21 | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device |
| PCT/CN2018/107916 WO2020056791A1 (en) | 2018-09-21 | 2018-09-27 | Method and apparatus for super-resolution reconstruction of multi-scale dilated convolution neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811113028.0A CN109389556B (en) | 2018-09-21 | 2018-09-21 | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109389556A CN109389556A (en) | 2019-02-26 |
| CN109389556B true CN109389556B (en) | 2023-03-21 |
Family
ID=65419007
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811113028.0A Active CN109389556B (en) | 2018-09-21 | 2018-09-21 | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109389556B (en) |
| WO (1) | WO2020056791A1 (en) |
Families Citing this family (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110428428B (en) * | 2019-07-26 | 2022-03-25 | 长沙理工大学 | Image semantic segmentation method, electronic equipment and readable storage medium |
| CN110533051B (en) * | 2019-08-02 | 2023-01-17 | 中国民航大学 | Automatic detection method for contraband in X-ray security inspection image based on convolutional neural network |
| US11540798B2 (en) | 2019-08-30 | 2023-01-03 | The Research Foundation For The State University Of New York | Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising |
| CN112488916B (en) * | 2019-09-12 | 2023-12-26 | 武汉Tcl集团工业研究院有限公司 | Training method and computer equipment for image super-resolution reconstruction model |
| CN110659724B (en) * | 2019-09-12 | 2023-04-28 | 复旦大学 | Target detection depth convolution neural network construction method based on target scale |
| CN110781776B (en) * | 2019-10-10 | 2022-07-05 | 湖北工业大学 | Road extraction method based on prediction and residual refinement network |
| CN111028147B (en) * | 2019-11-25 | 2024-02-02 | 天津大学 | Three-dimensional medical image super-resolution reconstruction method based on cascade cavity convolution |
| CN111160413B (en) * | 2019-12-12 | 2023-11-17 | 天津大学 | Thyroid nodule classification method based on multi-scale feature fusion |
| CN111144460A (en) * | 2019-12-16 | 2020-05-12 | 华东师范大学 | Algorithm for improving classification performance of convolutional neural network |
| CN111047515B (en) * | 2019-12-29 | 2024-01-09 | 兰州理工大学 | Attention mechanism-based cavity convolutional neural network image super-resolution reconstruction method |
| EP3846477B1 (en) * | 2020-01-05 | 2023-05-03 | Isize Limited | Preprocessing image data |
| CN111242846B (en) * | 2020-01-07 | 2022-03-22 | 福州大学 | Fine-grained scale image super-resolution method based on non-local enhancement network |
| CN111414988B (en) * | 2020-02-13 | 2023-04-18 | 湖北工业大学 | Remote sensing image super-resolution method based on multi-scale feature self-adaptive fusion network |
| CN111429347A (en) * | 2020-03-20 | 2020-07-17 | 长沙理工大学 | Image super-resolution reconstruction method and device and computer-readable storage medium |
| CN111461983B (en) * | 2020-03-31 | 2023-09-19 | 华中科技大学鄂州工业技术研究院 | Image super-resolution reconstruction model and method based on different frequency information |
| CN111667409B (en) * | 2020-06-09 | 2024-03-22 | 云南电网有限责任公司电力科学研究院 | Super-resolution algorithm-based insulator image resolution enhancement method |
| CN112017116B (en) * | 2020-07-23 | 2024-02-23 | 西北大学 | Image super-resolution reconstruction network based on asymmetric convolution and construction method thereof |
| CN111985544A (en) * | 2020-08-07 | 2020-11-24 | 中移（杭州）信息技术有限公司 | Fractal graph style fusion method and device, electronic equipment and storage medium |
| CN112017191B (en) * | 2020-08-12 | 2023-08-22 | 西北大学 | Liver pathology image segmentation model establishment and segmentation method based on attention mechanism |
| CN112070667B (en) * | 2020-08-14 | 2024-06-18 | 深圳市九分文化传媒有限公司 | Multi-scale feature fusion video super-resolution reconstruction method |
| CN111915490A (en) * | 2020-08-14 | 2020-11-10 | 深圳清研智城科技有限公司 | License plate image super-resolution reconstruction model and method based on multi-scale features |
| CN112070689A (en) * | 2020-08-24 | 2020-12-11 | 中国人民解放军陆军装甲兵学院 | Data enhancement method based on depth image |
| CN112070669B (en) * | 2020-08-28 | 2024-04-16 | 西安科技大学 | Super-resolution image reconstruction method for arbitrary fuzzy core |
| CN112101366A (en) * | 2020-09-11 | 2020-12-18 | 湖南大学 | Real-time segmentation system and method based on hybrid expansion network |
| CN112200725B (en) * | 2020-10-26 | 2023-11-28 | 深圳大学 | Super-resolution reconstruction method and device, storage medium and electronic equipment |
| CN112288629B (en) * | 2020-10-27 | 2024-03-01 | 河南威虎智能科技有限公司 | Super-resolution method of cross-scale graph neural network |
| CN112330540A (en) * | 2020-11-08 | 2021-02-05 | 哈尔滨理工大学 | Image super-resolution method based on multi-scale optimization |
| CN112541878A (en) * | 2020-12-24 | 2021-03-23 | 北京百度网讯科技有限公司 | Method and device for establishing image enhancement model and image enhancement |
| CN112699967B (en) * | 2021-01-18 | 2024-03-12 | 武汉大学 | Remote airport target detection method based on improved deep neural network |
| CN112734915A (en) * | 2021-01-19 | 2021-04-30 | 北京工业大学 | Multi-view stereoscopic vision three-dimensional scene reconstruction method based on deep learning |
| CN112884654B (en) * | 2021-03-03 | 2024-02-20 | 鄂尔多斯应用技术学院 | WDSR image super-resolution reconstruction optimization method based on CNN |
| CN112950476A (en) * | 2021-03-12 | 2021-06-11 | 广州冠图视觉科技有限公司 | Method for improving resolution and definition of picture |
| CN113191947B (en) * | 2021-03-23 | 2024-05-14 | 西安电子科技大学 | Image super-resolution method and system |
| CN113298714B (en) * | 2021-05-24 | 2024-04-26 | 西北工业大学 | Image cross-scale super-resolution method based on deep learning |
| CN113450290B (en) * | 2021-09-01 | 2021-11-26 | 中科方寸知微(南京)科技有限公司 | Low-illumination image enhancement method and system based on image inpainting technology |
| CN113781306B (en) * | 2021-09-09 | 2024-03-08 | 西北工业大学 | Super-resolution reconstruction method for hyperspectral image based on double-stage strategy |
| CN114913069B (en) * | 2022-04-26 | 2024-05-03 | 上海大学 | Infrared image super-resolution reconstruction method based on deep neural network |
| CN115937018B (en) * | 2022-11-03 | 2023-11-03 | 江苏济远医疗科技有限公司 | Method for restoring image by using multichannel feedback network |
| CN115759202B (en) * | 2022-11-22 | 2023-11-28 | 江苏济远医疗科技有限公司 | Variable cavity convolution dense image super-resolution method |
| CN116309067B (en) * | 2023-03-21 | 2023-09-29 | 安徽易刚信息技术有限公司 | Light field image space super-resolution method |
| CN116452696B (en) * | 2023-06-16 | 2023-08-29 | 山东省计算中心（国家超级计算济南中心） | Image compressed sensing reconstruction method and system based on double-domain feature sampling |
| CN116703729B (en) * | 2023-08-09 | 2023-12-19 | 荣耀终端有限公司 | Image processing method, terminal, storage medium and program product |
| CN117132472B (en) * | 2023-10-08 | 2024-05-31 | 兰州理工大学 | Forward-backward separable self-attention-based image super-resolution reconstruction method |
| CN117172134B (en) * | 2023-10-19 | 2024-01-16 | 武汉大学 | Moon surface multiscale DEM modeling method based on fusion terrain features |
| CN118096556A (en) * | 2024-04-23 | 2024-05-28 | 华东交通大学 | Image reconstruction method, system, equipment and medium |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3507773A1 (en) * | 2016-09-02 | 2019-07-10 | Artomatix Ltd. | Systems and methods for providing convolutional neural network based image synthesis using stable and controllable parametric models, a multiscale synthesis framework and novel network architectures |
| US11024009B2 (en) * | 2016-09-15 | 2021-06-01 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN106952228B (en) * | 2017-03-10 | 2020-05-22 | 北京工业大学 | Super-resolution reconstruction method of single image based on image non-local self-similarity |
| US10803378B2 (en) * | 2017-03-15 | 2020-10-13 | Samsung Electronics Co., Ltd | System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions |
| CN106991648B (en) * | 2017-04-10 | 2024-01-02 | 中国石油大学(华东) | Image super-resolution reconstruction method based on wavelet transformation and convolutional neural network |
| CN107392852B (en) * | 2017-07-10 | 2020-07-07 | 深圳大学 | Super-resolution reconstruction method, device and equipment for depth image and storage medium |
| CN107464216A (en) * | 2017-08-03 | 2017-12-12 | 济南大学 | A kind of medical image ultra-resolution ratio reconstructing method based on multilayer convolutional neural networks |
| CN107679477B (en) * | 2017-09-27 | 2021-02-02 | 深圳市未来媒体技术研究院 | Face depth and surface normal vector prediction method based on cavity convolution neural network |
| CN108537731B (en) * | 2017-12-29 | 2020-04-14 | 西安电子科技大学 | Image super-resolution reconstruction method based on compressed multi-scale feature fusion network |
| CN108428212A (en) * | 2018-01-30 | 2018-08-21 | 中山大学 | A kind of image magnification method based on double laplacian pyramid convolutional neural networks |
| CN108537733B (en) * | 2018-04-11 | 2022-03-11 | 南京邮电大学 | Super-resolution reconstruction method based on multi-path deep convolutional neural network |
| CN108550115B (en) * | 2018-04-25 | 2020-06-16 | 中国矿业大学 | Image super-resolution reconstruction method |
-
2018
- 2018-09-21 CN CN201811113028.0A patent/CN109389556B/en active Active
- 2018-09-27 WO PCT/CN2018/107916 patent/WO2020056791A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020056791A1 (en) | 2020-03-26 |
| CN109389556A (en) | 2019-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109389556B (en) | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device | |
| CN112233038B (en) | True image denoising method based on multi-scale fusion and edge enhancement | |
| CN110232394B (en) | Multi-scale image semantic segmentation method | |
| CN108537733B (en) | Super-resolution reconstruction method based on multi-path deep convolutional neural network | |
| CN110599401A (en) | Remote sensing image super-resolution reconstruction method, processing device and readable storage medium | |
| CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
| US10198801B2 (en) | Image enhancement using self-examples and external examples | |
| CN110766632A (en) | Image denoising method based on channel attention mechanism and characteristic pyramid | |
| CN110136075B (en) | Remote sensing image defogging method for generating countermeasure network based on edge sharpening cycle | |
| CN111242846A (en) | Fine-grained scale image super-resolution method based on non-local enhancement network | |
| CN114757832A (en) | Face super-resolution method and device based on cross convolution attention antagonistic learning | |
| CN110443775B (en) | Discrete wavelet transform domain multi-focus image fusion method based on convolutional neural network | |
| CN113129212B (en) | Image super-resolution reconstruction method and device, terminal device and storage medium | |
| US20230252605A1 (en) | Method and system for a high-frequency attention network for efficient single image super-resolution | |
| CN114723630A (en) | Image deblurring method and system based on cavity double-residual multi-scale depth network | |
| CN113538246A (en) | Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network | |
| Zhao et al. | Deep pyramid generative adversarial network with local and nonlocal similarity features for natural motion image deblurring | |
| CN116452810A (en) | Multi-level semantic segmentation method and device, electronic equipment and storage medium | |
| Esmaeilzehi et al. | UPDResNN: A deep light-weight image upsampling and deblurring residual neural network | |
| CN109993701B (en) | Depth map super-resolution reconstruction method based on pyramid structure | |
| CN117593187A (en) | Remote sensing image super-resolution reconstruction method based on meta-learning and transducer | |
| CN113096032A (en) | Non-uniform blur removing method based on image area division | |
| CN116977200A (en) | Processing method and device of video denoising model, computer equipment and storage medium | |
| CN115205148A (en) | Image deblurring method based on double-path residual error network | |
| CN115760658A (en) | Image processing method, image processing device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |