WO2023010831A1 - 提高图像分辨率的方法、系统、装置及存储介质 - Google Patents

提高图像分辨率的方法、系统、装置及存储介质 Download PDF

Info

Publication number
WO2023010831A1
WO2023010831A1 PCT/CN2022/077832 CN2022077832W WO2023010831A1 WO 2023010831 A1 WO2023010831 A1 WO 2023010831A1 CN 2022077832 W CN2022077832 W CN 2022077832W WO 2023010831 A1 WO2023010831 A1 WO 2023010831A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature information
information
resolution
image
module
Prior art date
Application number
PCT/CN2022/077832
Other languages
English (en)
French (fr)
Inventor
王进
吴一鸣
何施茗
陈泽宇
王柳
Original Assignee
长沙理工大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 长沙理工大学 filed Critical 长沙理工大学
Publication of WO2023010831A1 publication Critical patent/WO2023010831A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the invention relates to the field of single image super-resolution, in particular to a method, system, device and storage medium for improving image resolution.
  • Single Image Super-Resolution (SISR) reconstruction algorithm is to restore low-resolution images into high-resolution images with good visual effects through a series of algorithms.
  • single image super-resolution is an ill-posed algorithm problem, that is, for any low-resolution image, there may be countless high-resolution images corresponding to it.
  • Single-image super-resolution technology can not only restore clarity from low-resolution pictures, but also save the cost of replacing or upgrading camera components because the super-resolution method processes the captured photos.
  • Disadvantages of the existing technology 1 large super-resolution network parameters and high calculation amount; disadvantages of the existing technology 2: lightweight super-resolution algorithm reduces the effect of super-resolution reconstruction.
  • the super-resolution reconstruction network based on convolutional neural network (SRCNN, Super-Resolution Convolution Neural Network) proposed by Dong et al. applies three-layer convolutional neural network to image super-resolution for the first time. rate field, and achieved better results than traditional methods.
  • the deep recursive convolution-based neural network (DRCN, Deep Recursive Convolution Network) proposed by Kim et al. applies the recurrent neural network to super-resolution, and makes the feature extraction part learn repeatedly through the cyclic structure without increasing the amount of network parameters. .
  • the waterfall residual network CARN (Cascading Residual Network) proposed by Ahn et al. uses the waterfall cascading method, although it has a small amount of parameters while increasing the information between the convolutional layers. Interaction, but the increased cascade operation greatly increases the amount of calculation.
  • the Information Distillation Network (IDN, Information Distillation Network) proposed by Hui et al. divides the feature extraction module into an information enhancement unit and a compression unit, and extracts and removes features respectively. redundant information.
  • most of the network is feed-forward propagation, and the feature information lacks mutual learning, resulting in a mediocre reconstruction effect. Therefore, there is an urgent need to adopt an image processing method that can further reduce the amount of parameters and calculations while improving the image super-resolution reconstruction effect.
  • the main purpose of the present invention is to provide a method, system, device, and storage medium for improving image resolution, aiming at solving the problem that existing low-resolution images cannot be guaranteed while improving the resolution reconstruction effect by using single-image super-resolution technology.
  • the present invention provides a method for improving image resolution, said method comprising the following steps:
  • the shallow layer feature information and the feedback feature information are carried out to enhance the information multiple distillation module (EIMDB, Enhance Information Multi-Distillation Network) operation to obtain the middle layer feature information;
  • EIMDB Enhance Information Multi-Distillation Network
  • IMDB Information Multi-Distillation Block
  • the steps of the information multiple distillation module (IMDB) operation include:
  • the processed first target feature information is subjected to coordinate attention (CA, Coordinate Attention) mechanism processing and 1 ⁇ 1 convolution, and is added to the received first initial image feature information to obtain an information multiple distillation module (IMDB) output image feature information after the operation.
  • CA Coordinate Attention
  • IMDB information multiple distillation module
  • the steps of the enhanced information multiple distillation module include:
  • the processed second target feature information is subjected to coordinate attention (CA, Coordinate Attention) mechanism processing and 1 ⁇ 1 convolution operation, and is added to the received second initial image feature information to obtain an enhanced information multiple
  • the output image feature information after the operation of the distillation module (EIMDB, Enhance Information Multi-Distillation Network).
  • the steps of the ghost module (Ghost Module) operation include:
  • the initial feature information and the group feature information are cascaded to obtain the output image feature information after one ghost Module operation.
  • the step of obtaining the feature information of the low-resolution image and performing a convolution operation to obtain shallow feature information includes:
  • the feature information of the obtained low-resolution image is output by 3 ⁇ 3 convolution and 1 ⁇ 1 convolution operations to output shallow feature information, and the operation formula is as follows:
  • FL represents shallow feature information
  • conv 1 ⁇ 1 and conv 3 ⁇ 3 represent convolution operations with convolution kernel sizes of 1 ⁇ 1 and 3 ⁇ 3, respectively
  • I LR represents the feature information of the input low-resolution image
  • ReLU() is the activation function
  • the step of performing the Enhanced Information Multiple Distillation Module (EIMDB) operation on the shallow layer feature information and the feedback feature information to obtain the middle layer feature information includes:
  • Adopt N EIMDB modules to carry out the extraction of middle-level feature information by described shallow feature information, feedback feature information, described operation formula is as follows:
  • F M represents the middle-level feature information
  • F i EIMDB represents the output information of the i-th EIMDB module in the middle-level feature extraction (1 ⁇ i ⁇ N)
  • F FB represents the feedback feature information
  • the step of performing an information multiple distillation module (IMDB) operation on the middle-level feature information to obtain deep-level feature information includes:
  • F H represents the deep feature information
  • F j IMDB represents the output of the jth IMDB module in the deep feature extraction (1 ⁇ j ⁇ M);
  • the step of obtaining the feedback feature information according to the shallow feature information, the middle feature information and the deep feature information includes:
  • the shallow feature information, the middle feature information and the deep feature information are cascaded and twice 1 ⁇ 1 convolution to obtain the feedback feature information, and the operation formula is as follows:
  • F FB conv 1 ⁇ 1 (concat(conv 1 ⁇ 1 (ReLU((concat(F M ,F H ))),F L ));
  • FFB represents feedback feature information
  • FL represents shallow feature information
  • FM represents middle-level feature information
  • FM represents middle-level feature information
  • the step of obtaining the reconstructed super-resolution picture information of the low-resolution image according to the middle-level feature information, the deep-level feature information and the feature information of the low-resolution image includes:
  • the deep feature information and the middle feature information are concatenated and 1 ⁇ 1 convolution to obtain the initial reconstruction feature information, and the initial reconstruction feature information is added to the shallow feature extraction, 3 ⁇ 3 convolution and a
  • the sub-pixel convolution operation obtains the super-resolution image information after reconstruction of the low-resolution image;
  • F R f sub (conv 3 ⁇ 3 (conv 1 ⁇ 1 (ReLU((concat(F M ,F H ))))+I LR )));
  • I SR represents super-resolution image information
  • FR represents image reconstruction feature information
  • f sub represents sub-pixel convolution operation
  • the system for improving the image resolution of the present invention includes:
  • the shallow feature extraction module is used to obtain the feature information of the low-resolution image and perform convolution operation to obtain the shallow feature information
  • the middle-level feature extraction module is used to perform the Enhanced Information Multiple Distillation Module (EIMDB) operation on the shallow-level feature information and the feedback feature information to obtain the middle-level feature information;
  • EIMDB Enhanced Information Multiple Distillation Module
  • a deep feature extraction module is used to perform an information multiple distillation module (IMDB) operation on the middle-level feature information to obtain deep-level feature information;
  • IMDB information multiple distillation module
  • a feedback mechanism module configured to obtain the feedback feature information according to the shallow feature information, the middle feature information and the deep feature information
  • a reconstruction module configured to obtain super-resolution picture information reconstructed from the low-resolution image according to the feature information of the middle layer, the feature information of the deep layer, and the feature information of the low-resolution image.
  • the device for improving image resolution of the present invention includes a memory, a processor, and a program for improving image resolution stored in the memory and operable on the processor.
  • the high-speed program is executed by the processor, the steps of the above-mentioned method for increasing image resolution are realized.
  • the present invention also provides a storage medium, on which a program for improving image resolution is stored, and when the program for improving image resolution is executed by a processor, the aforementioned improvement The steps of the method of image resolution.
  • the present invention provides a method for improving image resolution, by obtaining feature information of low-resolution images and performing convolution operation to obtain shallow feature information; performing enhanced information multiple distillation module (EIMDB) to obtain middle-level feature information; performing information multiple distillation module (IMDB) on the middle-level feature information to obtain deep-level feature information; obtaining the feedback feature information according to the shallow-level feature information, middle-level feature information and the deep-level feature information; Obtaining super-resolution picture information reconstructed from the low-resolution image according to the middle-level feature information, deep-level feature information, and feature information of the low-resolution image.
  • EIMDB enhanced information multiple distillation module
  • IMDB information multiple distillation module
  • the batch processing of low-resolution images effectively reduces the amount of data parameters and data calculations, and the feedback mechanism module can be used to improve the connection between low-level and high-level feature information, and then convert low-resolution images into super High-resolution images improve the reconstruction effect of low-resolution images and reduce the amount of computation and parameters in the reconstruction process.
  • Fig. 1 is a schematic structural diagram of a device for improving image resolution related to an embodiment of the present invention
  • Fig. 2 is a schematic flow chart of an embodiment of a method for improving image resolution in the present invention
  • Fig. 3 is a schematic diagram of a module structure of a system for improving image resolution related to an embodiment of the present invention
  • Fig. 4 is the structural representation of the IMDB module in Fig. 3;
  • Fig. 5 is the EIMDB module structure schematic diagram in Fig. 3;
  • Fig. 6 is a schematic structural diagram of the ghost Module module in Fig. 5;
  • Fig. 7 is a schematic structural diagram of the CA mechanism module in Fig. 4 and Fig. 5;
  • Fig. 8 is the comparison diagram after the first test picture in the Set14 test set in the verification experiment of the present application is processed in different ways;
  • Fig. 9 is a comparison diagram of the second test picture in the Urban100 test set of the verification experiment of the present application after different processing;
  • Figure 10 is a comparison chart of the third test picture processed in different ways in the Urban100 test set of the verification experiment of the present application;
  • Figure 11 is a comparison chart of the fourth test picture in the Urban100 test set of the verification experiment of this application after different processing
  • Figure 12 is a scatter diagram of the comparison of parameter quantities of the Urban100 test set in the verification experiment of the present application under twice the magnification.
  • the super-resolution reconstruction network (SRCNN) based on convolutional neural network (SRCNN) proposed by Dong et al. applied the three-layer convolutional neural network to the field of image super-resolution for the first time, and achieved a comparative
  • the traditional method is better.
  • the deep recursive convolution-based neural network (DRCN) proposed by Kim et al. applies the recurrent neural network to super-resolution, and makes the feature extraction part learn repeatedly through the loop structure without increasing the amount of network parameters.
  • the image was enlarged and directly input into the network, which resulted in a large feature map and increased the amount of parameters and calculations of the network.
  • the accelerated super-resolution convolutional neural network (FSRCNN) proposed by Dong et al. and the sub-pixel convolutional neural network (ESPCN) proposed by Shi et al. respectively adopt the method of deconvolution and sub-pixel convolution.
  • the image is directly extracted through the network, which reduces the increase in the amount of calculation caused by the enlargement of the feature map.
  • the network structure is usually deepened. Even if the reconstruction method of deconvolution or sub-pixel convolution is used, the amount of computation and parameters of the network will inevitably increase.
  • the CARN proposed by Ahn et al. uses the waterfall cascade method.
  • the information distillation network (IDN) proposed by Hui et al. divides the feature extraction module into an information enhancement unit and a compression unit, which extracts features and removes redundant information in features, respectively.
  • IDN information distillation network
  • the present invention provides a solution to obtain the shallow feature information by obtaining the feature information of the low-resolution image and performing a convolution operation; the shallow feature information and the feedback feature information are subjected to an enhanced information multiple distillation module (EIMDB) to obtain middle-level feature information; performing information multiple distillation module (IMDB) on the middle-level feature information to obtain deep-level feature information; obtaining the feedback feature information according to the shallow-level feature information, middle-level feature information and the deep-level feature information; according to the The middle-level feature information, the deep-level feature information and the feature information of the low-resolution image are used to obtain super-resolution picture information after reconstruction of the low-resolution image.
  • EIMDB enhanced information multiple distillation module
  • IMDB information multiple distillation module
  • the batch processing of low-resolution images effectively reduces the amount of data parameters and data calculations, and the feedback mechanism module can be used to improve the connection between low-level and high-level feature information, and then convert low-resolution images into super High-resolution images improve the reconstruction effect of low-resolution images and reduce the amount of computation and parameters in the reconstruction process.
  • FIG. 1 is a schematic structural diagram of an apparatus for improving image resolution according to an embodiment of the present invention.
  • the device may include: a processor 1001 , such as a CPU, a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 .
  • the communication bus 1002 is used to realize connection and communication between these components.
  • the user interface 1003 may be an infrared receiving module for receiving control commands triggered by the user through a remote controller, and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory.
  • the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
  • the device can be applied in the camera equipment of the Internet of Things, thereby reducing the cost of replacing the high-definition pixel camera; at the same time, based on the low amount of parameters and calculation in the device, it can also be applied to low-computing power devices for high-resolution rate image reconstruction operations.
  • Figure 3 is a schematic diagram of a module structure of a system for improving image resolution according to an embodiment of the present invention. Methods include:
  • Step S10 acquiring feature information of the low-resolution image and performing a convolution operation to obtain shallow feature information.
  • the shallow feature information of the low-resolution image is obtained, such as the shallow feature extraction module FL shown in FIG. Layer feature information; specifically, the feature information of the low-resolution image is output using 3 ⁇ 3 convolution and 1 ⁇ 1 convolution operations to output shallow feature information, wherein the extraction of shallow feature information is the low-resolution input of the network
  • the red, green, and blue (RGB, Red Blue Green) channels of the image increase the number of channels through convolution, so that features can be extracted from more channels in the subsequent mid-level and deep feature information extraction, and in the 3 ⁇ 3 convolution
  • the received feature information will be activated in advance, that is, the activation function (Rectified Linear Unit, ReLU) operation is used; among them, a 3 ⁇ 3 convolution and a 1 ⁇ 1 convolution, the number of output channels is 256 and 64 respectively, the operation formula is as follows:
  • FL represents the output of the shallow feature extraction module
  • conv 1 ⁇ 1 and conv 3 ⁇ 3 represent convolution operations with convolution kernel sizes of 1 ⁇ 1 and 3 ⁇ 3, respectively
  • I LR represents the input low-resolution image
  • ReLU() is the activation function.
  • Step S20 performing an Enhance Information Multi-Distillation Network (EIMDB) operation on the shallow layer feature information and feedback feature information to obtain middle layer feature information.
  • EIMDB Enhance Information Multi-Distillation Network
  • the middle-level feature information is obtained, such as the middle-level feature extraction module F M and the feedback mechanism module F FB shown in Figure 3, the middle-level feature extraction module is used to input the shallow-level feature extraction module and the feedback mechanism module
  • the feature information of the system is processed to obtain the middle-level feature information.
  • the EIMDB module is used to process the received shallow-level feature information and feedback feature information.
  • the feedback feature information is generated by the feedback mechanism module.
  • the acquisition of middle-level feature information is carried out by using N EIMDB modules, and the input dimension of middle-level feature extraction is 48 ⁇ 48 ⁇ 64 (length ⁇ width ⁇ number of channels), and the output dimension is 48 ⁇ 48 ⁇ 64.
  • Middle-level feature extraction can be performed by the following The formula says:
  • F M represents the output of the middle-level feature extraction module
  • F i EIMDB represents the output of the i-th EIMDB module in the middle-level feature extraction (1 ⁇ i ⁇ N)
  • F FB represents the output of the feedback mechanism module
  • F L represents the shallow feature Extract the output of the module.
  • Step S30 performing an Information Multi-Distillation Block (IMDB, Information Multi-Distillation Block) operation on the middle-level feature information to obtain deep-level feature information;
  • IMDB Information Multi-Distillation Block
  • deep feature information is obtained, such as the deep feature extraction module F H shown in FIG.
  • the amount of parameters and the amount of calculation are greatly reduced, but the ability to effect the deep feature information is also weakened. Therefore, by using the original M IMDB modules to process the middle-level feature information, the obtained feature information will be deeper. Therefore, under the joint action of the EIMDB module and the IMDB module, the final feature information can not only reduce the amount of parameters and calculation, but also ensure the extraction effect of feature information.
  • its input dimension is 48 ⁇ 48 ⁇ 64 (length ⁇ width ⁇ number of channels), and the output dimension is 48 ⁇ 48 ⁇ 64.
  • the operation formula is as follows:
  • F H represents the output of the deep extraction module
  • F j IMDB represents the output of the jth IMDB module in the deep feature extraction (1 ⁇ j ⁇ M)
  • F M represents the output of the middle layer extraction module.
  • Step S40 obtaining the feedback characteristic information according to the shallow characteristic information, the middle characteristic information and the deep characteristic information;
  • obtaining the feedback feature information needs to be realized by contacting multiple modules, including the shallow feature extraction module, the middle feature extraction module and the deep feature extraction module.
  • the middle feature information F M and the deep feature information F H passes cascade operation, ReLU operation and a 1 ⁇ 1 convolution operation, and then combines shallow feature information FL to continue cascade operation and 1 ⁇ 1 convolution to obtain feedback feature information.
  • the deep layer obtained in the previous steps Feature information and middle-level feature information perform feature fusion operations to improve the contextual relevance of features without increasing the amount of parameters.
  • F FB conv 1 ⁇ 1 (concat(conv 1 ⁇ 1 (ReLU((concat(F M ,F H ))),F L ));
  • F FB represents the output of the feedback mechanism module
  • FL represents the output of the shallow feature extraction module
  • F M represents the output of the middle layer feature extraction module
  • F H represents the output of the deep feature extraction module
  • concat() represents the cascade operation
  • conv 1 ⁇ 1 indicates a convolution operation with a convolution kernel size of 1 ⁇ 1.
  • Step S50 obtaining super-resolution image information reconstructed from the low-resolution image according to the feature information of the middle layer, the feature information of the deep layer, and the feature information of the low-resolution image.
  • the super-resolution picture information after reconstruction of the low-resolution image is obtained, such as the reconstruction module F R of FIG.
  • the feature information of the high-resolution image is added to obtain the super-resolution picture information after reconstruction of the low-resolution image.
  • the deep-level feature information and the middle-level feature information are obtained through cascade operation, ReLU operation and 1 ⁇ 1 convolution to obtain the initial Reconstructing feature information, and performing an addition operation, 3 ⁇ 3 convolution, and a sub-pixel convolution operation on the initial reconstruction feature information and shallow feature extraction to obtain super-resolution image information after reconstruction of the low-resolution image;
  • F R f sub (conv 3 ⁇ 3 (conv 1 ⁇ 1 (ReLU((concat(F M ,F H ))))+I LR )));
  • I SR represents the super-resolution image information
  • FR represents the output of the reconstruction module
  • f sub represents the sub-pixel convolution operation
  • the shallow feature information is obtained by obtaining the feature information of the low-resolution image and performing a convolution operation; and performing the enhanced information multiple distillation module (EIMDB) on the shallow feature information and feedback feature information to obtain the middle feature information; and performing information multiple distillation module (IMDB) on the middle-level feature information to obtain deep-level feature information; wherein, the shallow-level feature information, middle-level feature information and the deep-level feature information are used to obtain the feedback feature information; and according to the The super-resolution picture information reconstructed from the low-resolution image is obtained by combining the middle-level feature information, the deep-level feature information, and the feature information of the low-resolution image.
  • EIMDB enhanced information multiple distillation module
  • IMDB information multiple distillation module
  • the batch processing of low-resolution images effectively reduces the amount of data parameters and data calculations, and the feedback mechanism module can be used to improve the connection between low-level and high-level feature information, and then convert low-resolution images into super High-resolution images improve the reconstruction effect of low-resolution images and reduce the amount of computation and parameters in the reconstruction process.
  • IMDB information multiple distillation module
  • the received first initial image feature information is sequentially performed four times of 3 ⁇ 3 convolution operations to obtain the corresponding first sub-processing target feature information; in the actual image processing, the first 3 ⁇ 3 convolution operation is performed for the first time
  • the initial image feature information is the image feature information output by the middle-level feature extraction module FM
  • the input image feature information of the subsequent three 3 ⁇ 3 convolution operations is the image feature information output after the previous convolution operation is completed.
  • the first three 3 ⁇ 3 convolution operations Before the ⁇ 3 convolution operation, the ReLU operation will be performed in advance, and after each 3 ⁇ 3 convolution, some channels will be separated and used as the input of the next layer of convolution, and the remaining channels will be reserved.
  • the specific operation formula is as follows:
  • F i IMDB_in represents the input of the i-th IMDB
  • F i refine_1 and F i coarse_1 represent the trimmed feature map and the feature map for further feature extraction, respectively
  • f split represents the feature channel segmentation operation.
  • the first sub-processing target feature information obtained by the above convolution operation is concatenated to obtain the processed first target feature information, that is, the 16-layer channels retained by the above four convolutions are re-concatenated.
  • the number of channels is combined into 64 layers; and the processed first target feature information is subjected to coordinate attention (CA, Coordinate Attention) mechanism processing and 1 ⁇ 1 convolution operation, and is combined with the received first initial image feature information
  • CA Coordinate Attention
  • IMDB information multiple distillation module
  • F i IMDB represents the output of the i-th IMDB.
  • EIMDB Enhanced Information Multiple Distillation Module
  • the received second initial image feature information is sequentially performed four ghost module (Ghost Module) operations to obtain the second target feature information; in the actual image processing, the second ghost module (Ghost Module) operation
  • the initial image feature information is the image feature information output by the shallow feature extraction module FL
  • the input image feature information of the subsequent three afterimage module operations is the image feature information output after the previous image afterimage module operation is completed.
  • the number of input feature channels for the first ghost Module operation is 64
  • the number of input feature channels for the last three ghost Module operations is 48 layers
  • the output channel number of the first three ghost Module operations is 64 layers
  • the last ghost Module operation The output of the number of feature channels is 16.
  • the 64 layers of the first three output feature layers are divided into 48 layers and 16 layers, and the 48 layers are used as the input for the last four ghost Module operations.
  • the specific operation formula is as follows:
  • F' i EIMDB_in represents the input of the i-th EIMDB
  • F' i refine_1 and F' i coarse_1 represent the trimmed feature map and the feature map for further feature extraction, respectively
  • f split represents the feature channel segmentation operation.
  • F' i EIMDB represents the output of the i-th EIMDB.
  • FIG. 6 it is a schematic structural diagram of the ghost Module module in the enhanced information multiple distillation module (EIMDB) shown in Figure 5, and the specific details are as follows:
  • the first third initial image feature information refers to the shallow feature extraction module Image feature information output by FL .
  • the received third initial image feature information is divided into feature channels. For example, assuming that the number of input feature channels is M, the number of feature channels of the output ghost Module is N, and the partial group convolution is responsible for removing redundant feature channels.
  • the number of feature channels after group convolution is d(0 ⁇ d ⁇ M), and the convolution operation in ghost Module operation will perform ReLU operation in advance.
  • the specific operation formula is as follows:
  • Feat 1 conv 1 ⁇ 1 (ReLU(I F ));
  • Feat 1 represents the feature map after the primary convolution
  • I F represents the input of the ghost Module module.
  • Feat 2 g_conv 3 ⁇ 3 (ReLU(Feat 1 ));
  • Feat 2 represents the feature map after the cheap operation
  • g_conv 3 ⁇ 3 represents the group convolution with a convolution kernel size of 3 ⁇ 3.
  • the initial feature information and the group feature information are cascaded to obtain the output image feature information after one ghost Module operation.
  • the specific operation formula is as follows:
  • F GM concat(Feat 1 ,Feat 2 ) ;
  • F GM represents the output of the ghost Module module.
  • the Ghost Module operation in this embodiment divides the received feature information into feature channels, part of which is used to remove redundant feature channels, and part of which is used for convolution operations, thereby reducing the cost of running the entire system. Calculations and parameters.
  • FIG. 7 it is a schematic structural diagram of the CA mechanism module shown in FIG. 4 or 5, and the specific details are as follows:
  • the fourth initial image feature information includes vertical input feature information and horizontal input feature information, and the vertical input feature information and horizontal input feature information are respectively Two one-dimensional global average pooling processes are performed to obtain perceptual feature information in the vertical direction and perceptual feature information in the horizontal direction.
  • the fourth initial image feature information refers to the image feature information obtained after cascading operation in EIMDB operation or IMDB operation, and the obtained image feature information is subjected to vertical input feature information and horizontal input feature information
  • the division, and two one-dimensional global average pooling are used to process the feature information of the vertical input and the feature information of the horizontal input to obtain the perceptual feature information of the vertical direction and the perceptual feature information of the horizontal direction.
  • the specific operation formula is as follows:
  • F X pool (W) and F Y pool (h) represent one-dimensional pooling with the width as the direction and the length as the direction, respectively, W and H are the width and length of the input feature map, respectively, and ⁇ c() represents the fixed position The pixel value of .
  • the perceptual feature information in the vertical direction and the perceptual feature information in the horizontal direction are extracted.
  • the perceptual feature information in the vertical direction and the perceptual feature information in the horizontal direction are cascaded, and the vertical or horizontal dependency is extracted through the channel attention extraction operation, and then the generated attention maps are respectively
  • the coordinate attention information is stored in the image feature information after the CA operation of the fourth initial image feature information through horizontal and vertical pixel-by-pixel multiplication, so as to enable subsequent convolution operations.
  • the super-resolution image output of two iterations is used, and the L loss function is calculated separately with the original low-resolution image and then averaged, which can be expressed by the following formula:
  • I LR and I t SR represent the original low-resolution image and the super-resolution reconstructed image respectively.
  • an embodiment of the present invention also proposes a storage medium, on which a program for increasing image resolution is stored, and when the program for increasing image resolution is executed by a processor, the above-mentioned method for increasing image resolution can be realized. step.
  • Experimental content Initialize the network parameters in advance, where the initial learning rate is 10 -4 , the learning rate is set to 0.5 every 200 times of training, and a total of 1000 iterations are performed.
  • the experimental environment uses the GPU version of Pytorch1.6.0, the GPU uses RTX2070Super for training, and the operating system is Ubuntu16.08.
  • the training set used in the experiment is DIV2K, which includes 800 pictures of 2K resolution such as people, handmade products, buildings (cities, villages), animals and plants, and natural scenery.
  • DIV2K includes 800 pictures of 2K resolution such as people, handmade products, buildings (cities, villages), animals and plants, and natural scenery.
  • the test sets used in the experiment are five widely used super-resolution benchmark test sets of Set5, Set14, BSD100, Urban100, and Manga109 for model performance evaluation.
  • Urban100 contains 100 challenging urban scene pictures, containing dense high-frequency feature details.
  • Manga109 is 109 manga cover pictures, with high-frequency and low-frequency information and text information, which tests the comprehensive processing ability of the model for text and pictures.
  • EIMDN-L Ehance Information Multi-distillation Network Large
  • EIMDN-S Ehance Information Multi-distillation Network-Small
  • the number of EIMDBs is N
  • the number of IMDBs is M
  • PSNR peak signal-to-noise ratio
  • SSIM structure similarity
  • MAX is the maximum value representing the color of the image point
  • MSE is the mean square error between the original image and the processed image
  • ⁇ x is the mean value of x
  • ⁇ y is the mean value of y
  • ⁇ x is the variance of x
  • ⁇ y is the variance of y
  • ⁇ xy is the covariance of x and y
  • c 1 and c 2 are constants.
  • EIMDN-S has achieved a moderate reconstruction effect. It can be seen that EIMDN-L is superior to most models under the condition of magnification of 3 times and 4 times.
  • the PSNR value of the model corresponding to this application in the test set Manga109 is 2.35dB higher than that of the classic model VDSR, and 0.11dB higher than the second CARN. It shows that the model corresponding to this application can pass the high-frequency details that are difficult to learn at high magnifications through the feedback mechanism module to deepen the learning depth of high-frequency information, thereby achieving good reconstruction effects at high magnifications.
  • the model EIMDN-L corresponding to this application is significantly better than other datasets in the Urban100 test set, because the Urban100 dataset contains pictures of urban buildings with more high-frequency details.
  • the CA attention mechanism used in the corresponding model of this application can filter and retain features containing more high-frequency information in channels and spaces, so it can achieve better reconstruction results in the Urban test set.
  • the method used in the corresponding model of this application does not always achieve the best results compared to other models, indicating that although the Ghost Module used by EIMDB in this application can reduce the amount of parameters and remove Redundant channels of feature channels, and each channel retains more high-frequency details in lower magnifications, which will cause some important feature channels to be lost in the redundant removal step, which cannot be enlarged at 2 times The best effect under multiples.
  • the information distillation network IDN Information Distillation Network
  • CARN-M Cascading Residual Network-Mobile
  • waterfall residual network CARN (Cascading Residual Network)
  • information multiple distillation network IMDN Information Multi-distillation Network
  • the model EIMDN-L corresponding to this application can better restore the correct texture of women's headscarves and glass grilles.
  • EIMDN-S is also similar to other lightweight methods for recovery.
  • the computational complexity of the model is evaluated by using the Multi-Add operation proposed in the CARN method, that is, the statistics of the number of composite multiplication and accumulation operations under the condition of a single image size of 720p.
  • the Multi-Add operation proposed in the CARN method that is, the statistics of the number of composite multiplication and accumulation operations under the condition of a single image size of 720p.
  • Table 3 compared with the SOTA algorithm, EIMDN-S in the model corresponding to this application maintains a relatively low calculation amount.
  • the model EIMDN-L corresponding to this application has slightly increased the amount of calculation while obtaining a better reconstruction effect.
  • the second group compares the effects of different numbers of IMDB and EIMDB modules on the effect of super-resolution reconstruction, as shown in Table 5. It can be seen that the greater the number of IMDB and EIMDB modules in the second group, the better the PSNR and SSIM effects obtained.
  • the term “comprises”, “comprises” or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or system comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or system. Without further limitations, an element defined by the phrase “comprising a " does not preclude the presence of additional identical elements in the process, method, article or system comprising that element.
  • the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation.
  • the technical solution of the present invention can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , magnetic disk, optical disk), including several instructions to enable a terminal device (which may be a system for improving image resolution, an Internet of Things camera device, etc.) to execute the methods described in various embodiments of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

一种提高图像分辨率的方法、系统、装置及存储介质，所述方法包括：获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息（S10）；将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)操作得到中层特征信息（S20）；将所述中层特征信息进行信息多重蒸馏模块(IMDB)操作得到深层特征信息（S30）；根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息（S40）；根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息（S50）。该方法旨在提高低分辨率图片的重建效果的同时进一步减少参数量和计算量。

Description

提高图像分辨率的方法、系统、装置及存储介质 技术领域
本发明涉及单图像超分辨率领域，尤其涉及一种提高图像分辨率的方法、系统、装置及存储介质。
背景技术
图片是记录人们生活和历史场景重现的重要载体，特别是在交通流量预测、遥感以及刑侦取证等城市治理方面拥有非常多的图像数据。随着高分辨率的显示设备的涌现，低分辨率的图片无法在高分辨率设备上获得良好的视觉效果。单图像超分辨率(Single Image Super-Resolution，SISR)重建算法目的是将低分辨率图片通过一系列算法恢复成视觉效果良好的高分辨率图像。实际上，单图像超分辨率属于不适定的算法问题，即对于任意的低分辨率图像，都有可能存在无数的高分辨率图像与之对应。单图像超分辨率技术不仅能够从清晰度较低的图片中恢复清晰度，而且由于超分辨率方法是对所拍摄的照片进行处理，能够节约更换或升级摄像头元器件的成本。现有技术缺点①：大型超分辨率网络参数量和计算量高；现有技术缺点②：轻量级超分辨率算法降低了超分辨率重建效果。
针对现有技术缺点①提出的改进，由Dong等人提出的基于卷积神经网络的超分辨率重建网络(SRCNN，Super-Resolution Convolution Neural Network)首次将三层卷积神经网络应用于图像超分辨率领域，并取得了比传统方法更好的效果。由Kim等人提出的基于深度递归卷积的神经网络(DRCN，Deep Recursive Convolution Network)将递归神经网络应用于超分辨率，在不增加网络参数量的情况下使得特征提取部分通过循环结构重复学习。然而早期基于深度学习的超分辨率将图像放大后直接输入进网络，这导致特征映射较大，增加了网络的参数量和计算量。由Dong等人提出的加速超分辨率卷积神经网络(FSRCNN，,Faster Super-Resolution Convolution Neural Network)和由Shi等人提出的亚像素卷积神经网络(ESPCN，Enhance Sub-Pixel Convolution Network)分别采用了反卷积和亚像素卷积的方法，低分辨率图像直接通过网 络进行特征提取，减少了由于特征映射放大导致的计算量增加。但是，为了提高超分辨率重建的效果，通常会加深网络结构。即便是使用了反卷积或亚像素卷积的重建方法，网络的计算量与参数量还是不可避免的增加。
针对现有技术缺点②提出的改进，由Ahn等人提出的瀑布残差网络CARN(Cascading Residual Network)使用瀑布级联的方式，虽然在拥有少量参数量的同时增加了卷积层之间的信息交互，但是增加的级联操作大幅提高了计算量，由Hui等人提出的信息蒸馏网络(IDN，Information Distillation Network)将特征提取模块分成信息增强单元和压缩单元，分别将特征进行提取和去除特征中冗余信息。但是网络大部分都是前馈传播，特征信息缺少相互学习，导致重建效果一般。因此，亟需采用一种提高图像超分辨率重建效果的同时，还可以进一步减少参数量和计算量的图像处理方法。
发明内容
本发明的主要目的在于提供一种提高图像分辨率的方法、系统、装置及存储介质，旨在解决现有的低分辨率图片利用单图像超分辨率技术无法保障在提高分辨率重建效果的同时进一步减少参数量和计算量的技术问题。
为实现上述目的，本发明提供一种提高图像分辨率的方法，所述方法包括以下步骤：
获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息；
将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作得到中层特征信息；
将所述中层特征信息进行信息多重蒸馏模块(IMDB，Information Multi-Distillation Block)操作得到深层特征信息；
根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；
根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。
可选地，所述信息多重蒸馏模块(IMDB)操作的步骤，包括：
将接收到的第一初始图像特征信息依次进行四次3×3卷积操作得到对应的第一分处理目标特征信息；
将各第一分处理目标特征信息进行级联操作得到处理后的第一目标特征信息；
将所述处理后的第一目标特征信息进行坐标注意力(CA,Coordinate Attention)机制处理和1×1卷积，并与接收到的第一初始图像特征信息进行相加得到一次信息多重蒸馏模块(IMDB)操作后的输出图像特征信息。
可选地，所述增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作的步骤，包括：
将接收到的第二初始图像特征信息依次进行四次残影模块(Ghost Module)操作得到第二目标特征信息；
将所述处理后的第二目标特征信息进行坐标注意力(CA,Coordinate Attention)机制处理和1×1卷积操作，并与接收到的第二初始图像特征信息进行相加得到一次增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作后的输出图像特征信息。
可选地，所述残影模块(Ghost Module)操作的步骤，包括：
将接收的第三初始图像特征信息进行1×1卷积操作得到卷积后的初始特征信息；
将所述主要特征信息进行1×1分组卷积操作得到分组特征信息；
将所述初始特征信息和分组特征信息进行级联操作得到一次残影模块(Ghost Module)操作后的输出图像特征信息。
可选地，所述获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息的步骤，包括：
将获取的低分辨率图像的特征信息采用3×3卷积和1×1卷积操作输出浅层特征信息，所述操作公式如下：
F L＝conv 1×1(ReLU((conv 3×3(ReLU(I LR)))))；
其中，F L表示浅层特征信息，conv 1×1和conv 3×3分别表示卷积核大小为1×1和3×3的卷积操作，I LR表示输入的低分辨率图像的特征信息，ReLU()是激活函数；
所述将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)操作得到中层特征信息的步骤，包括：
将所述浅层特征信息、反馈特征信息采用N个EIMDB模块来进行中层 特征信息的提取，所述操作公式如下：
Figure PCTCN2022077832-appb-000001
其中，F M表示中层特征信息，F i EIMDB表示中层特征提取里第i个EIMDB模块的输出信息(1<i≤N),F FB表示反馈特征信息。
可选地，所述将所述中层特征信息进行信息多重蒸馏模块(IMDB)操作得到深层特征信息的步骤，包括：
将所述中层特征信息采用M个IMDB模块进行深层特征信息的提取，所述操作公式如下：
Figure PCTCN2022077832-appb-000002
其中，F H表示深层特征信息，F j IMDB表示深层特征提取里第j个IMDB模块的输出(1<j≤M)；
所述根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息的步骤，包括：
将所述浅层特征信息、中层特征信息和所述深层特征信息经过级联操作和两次1×1卷积得到反馈特征信息，所述操作公式如下：
F FB＝conv 1×1(concat(conv 1×1(ReLU((concat(F M,F H))),F L))；
其中，F FB表示反馈特征信息，F L表示浅层特征信息，F M表示中层特征信息，F M表示中层特征信息。
可选地，所述根据所述中层特征信息、深层特征信息进和所低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息的步骤，包括：
将所述深层特征信息和中层特征信息通过级联操作和1×1卷积得到初始重建特征信息并将所述初始重建特征信息与浅层特征提取进行相加操作、3×3卷积和一个亚像素卷积操作得到低分辨率图像重建后的超分辨率图片信息；
所述操作公式如下：
F R＝f sub(conv 3×3(conv 1×1(ReLU((concat(F M,F H))))+I LR)))；
I SR＝F R；
其中I SR表示超分辨率图片信息，F R表示图像重建特征信息，f sub表示亚像素卷积操作。
此外，为实现上述目的，本发明提高图像分辨率的系统，包括：
浅层特征提取模块，用于获取低分辨率图像的特征信息并进行卷积操作 得到浅层特征信息；
中层特征提取模块，用于将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)操作得到中层特征信息；
深层特征提取模块，用于将所述中层特征信息进行信息多重蒸馏模块(IMDB)操作得到深层特征信息；
反馈机制模块，用于根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；
重建模块，用于根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。
此外，为实现上述目的，本发明提高图像分辨率的装置，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的提高图像分辨率的程序，所述提高图像分辨率的程序被所述处理器执行时实现如上述所述的提高图像分辨率的方法的步骤。
此外，为实现上述目的，本发明还提供一种存储介质，所述存储介质上存储有提高图像分辨率的程序，所述提高图像分辨率的程序被处理器执行时实现如上述所述的提高图像分辨率的方法的步骤。
本发明提供一种提高图像分辨率的方法，通过获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息；将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)得到中层特征信息；将所述中层特征信息进行信息多重蒸馏模块(IMDB)得到深层特征信息；根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。采用低分辨率图像分批次处理有效地降低了数据参数量和数据计算量，并引用了反馈机制模块可以实现提高低层和高层特征信息之间的联系，进而实现将低分辨率图片转换成超分辨率图片，使得提高了低分辨率图片的重建效果的同时也降低了重建过程中的计算量和参数量。
附图说明
图1是本发明实施例方案涉及的提高图像分辨率的装置的结构示意图；
图2为本发明提高图像分辨率的方法的实施例的流程示意图；
图3为本发明实施例方案涉及的提高图像分辨率的系统的模块结构示意 图；
图4为图3中的IMDB模块结构示意图；
图5为图3中的EIMDB模块结构示意图；
图6为图5中的Ghost Module模块结构示意图；
图7为图4和图5中的CA机制模块结构示意图；
图8为本申请验证实验中的Set14测试集中的第一测试图片不同方式处理后的对比图；
图9为本申请验证实验的Urban100测试集中的第二测试图片图片不同方式处理后的对比图；
图10为本申请验证实验的Urban100测试集中的第三测试图片不同方式处理后的对比图；
图11为本申请验证实验的Urban100测试集中的第四测试图片不同方式处理后的对比图；
图12为本申请验证实验中的Urban100测试集在放大两倍数下的参数量对比散点图。
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
本发明实施例的主要解决方案是：
由于现有技术缺点①：大型超分辨率网络参数量和计算量高；现有技术缺点②：轻量级超分辨率算法降低了超分辨率重建效果。
针对现有技术缺点①提出的改进，由Dong等人提出的基于卷积神经网络的超分辨率重建网络(SRCNN)首次将三层卷积神经网络应用于图像超分辨率领域，并取得了比传统方法更好的效果。由Kim等人提出的基于深度递归卷积的神经网络(DRCN)将递归神经网络应用于超分辨率，在不增加网络参数量的情况下使得特征提取部分通过循环结构重复学习。然而早期基于深度学习的超分辨率将图像放大后直接输入进网络，这导致特征映射较大，增加 了网络的参数量和计算量。由Dong等人提出的加速超分辨率卷积神经网络(FSRCNN)和由Shi等人提出的亚像素卷积神经网络(ESPCN)分别采用了反卷积和亚像素卷积的方法，低分辨率图像直接通过网络进行特征提取，减少了由于特征映射放大导致的计算量增加。但是，为了提高超分辨率重建的效果，通常会加深网络结构。即便是使用了反卷积或亚像素卷积的重建方法，网络的计算量与参数量还是不可避免的增加。针对现有技术缺点②提出的改进，由Ahn等人提出的CARN使用瀑布级联的方式，虽然在拥有少量参数量的同时增加了卷积层之间的信息交互，但是增加的级联操作大幅提高了计算量，由Hui等人提出的信息蒸馏网络(IDN)将特征提取模块分成信息增强单元和压缩单元，分别将特征进行提取和去除特征中冗余信息。但是网络大部分都是前馈传播，特征信息缺少相互学习，导致重建效果一般。
本发明提供一种解决方案，使通过获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息；将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)得到中层特征信息；将所述中层特征信息进行信息多重蒸馏模块(IMDB)得到深层特征信息；根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。采用低分辨率图像分批次处理有效地降低了数据参数量和数据计算量，并引用了反馈机制模块可以实现提高低层和高层特征信息之间的联系，进而实现将低分辨率图片转换成超分辨率图片，使得提高了低分辨率图片的重建效果的同时也降低了重建过程中的计算量和参数量。
如图1所示，图1是本发明实施例方案涉及的提高图像分辨率的装置的结构示意图。
如图1所示，该装置可以包括：处理器1001，例如CPU，通信总线1002，用户接口1003，网络接口1004，存储器1005。其中，通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以红外接收模块，用于接收用户通过遥控器触发的控制指令，可选的用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如WI-FI接口)。存储器1005可以是高速RAM存储器，也可以是稳定的存储器(non-volatile memory)，例如磁盘存储器。存储器1005可选的还可以是独立 于前述处理器1001的存储装置。其中该装置可以应用在物联网摄像头设备中，进而可以减少更换高清像素摄像头的开销；同时，基于该装置中参数量和计算量较低，因此，还可以应用在低算力设备上进行高分辨率图像重建操作。
本领域技术人员可以理解，图1中示出的提高图像分辨率的装置的结构并不构成对提高图像分辨率的系统的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
本发明提高图像分辨率的装置的具体实施例与下述提高图像分辨率的方法各实施例基本相同，在此不作赘述。
参照图2至3，图3为本发明实施例方案涉及的提高图像分辨率的系统的模块结构示意图，图2为本发明提高图像分辨率的方法提供第一实施例的操作流程示意图，所述方法包括：
步骤S10，获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息。
本实施例中，获取低分辨率图像的浅层特征信息，如图3所示的浅层特征提取模块F L，所述浅层特征提取模块用于将接收的低分辨率图像进行处理得到浅层特征信息；具体地，将低分辨率图像的特征信息采用3×3卷积和1×1卷积操作输出浅层特征信息，其中，浅层特征信息的提取是将网络输入的低分辨率图像的红绿蓝(RGB,Red Blue Green)通道通过卷积的方式增加通道数，使得在后续中层和深层特征信息提取时能从更多的通道数中提取特征，并且在进行3×3卷积和1×1卷积操作之前，会预先将接收到的特征信息进行激活处理，即采用激活函数(Rectified Linear Unit,ReLU)操作；其中，上述中采用了一个3×3卷积和一个1×1卷积，输出通道数分别为和256和64，所述操作公式如下：
F L＝conv 1×1(ReLU((conv 3×3(ReLU(I LR)))))；
其中，F L表示浅层特征提取模块的输出，conv 1×1和conv 3×3分别表示卷积核大小为1×1和3×3的卷积操作，I LR表示输入的低分辨率图像，ReLU()是激活函数。
步骤S20，将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作得到中层特征信息。
本实施例中，获取中层特征信息，如图3所示的中层特征提取模块F M和反馈机制模块F FB，所述中层特征提取模块用于将所述浅层特征提取模块和反馈机制模块输入的特征信息进行处理得到中层特征信息，具体地，采用了EIMDB模块来对接收的浅层特征信息和反馈特征信息进行处理，其中所述反馈特征信息是由反馈机制模块生成的，在具体地进行中层特征信息的获取是采用N个EIMDB模块来进行，且中层特征提取的输入维度为48×48×64(长×宽×通道数)，输出维度为48×48×64，中层特征提取可由下列公式表示：
Figure PCTCN2022077832-appb-000003
其中，F M表示中层特征提取模块的输出，F i EIMDB表示中层特征提取里第i个EIMDB模块的输出(1<i≤N),F FB表示反馈机制模块的输出，F L表示浅层特征提取模块的输出。
步骤S30，将所述中层特征信息进行信息多重蒸馏模块(IMDB，Information Multi-Distillation Block)操作得到深层特征信息；
本实施例中，获取深层特征信息，如图3所示的深层特征提取模块F H，所述深层特征提取模块用于将中层特征提取模块的特征信息进行处理得到深层特征信息，具体地，基于步骤S20中的EIMDB模块操作后拥有大幅下降的参数量和计算量，但是深层特征信息的效果的能力也随之减弱。因此，通过使用原始的M个IMDB模块对中层特征信息进行处理，得到的将是更深层次的特征信息。因此，在EIMDB模块和IMDB模块共同作用下，最终得到的特征信息既可以减少了参数量和计算量，又保证了特征信息的提取效果。其中，IMDB模块操作中，其输入维度为48×48×64(长×宽×通道数)，输出维度为48×48×64，所述操作公式如下：
Figure PCTCN2022077832-appb-000004
其中，F H表示深层提取模块的输出，F j IMDB表示深层特征提取里第j个IMDB模块的输出(1<j≤M)，F M表示中层提取模块的输出。
步骤S40，根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；
本实施例中，获取反馈特征信息是需要接触多个模块共同作用来实现，包括浅层特征提取模块、中层特征提取模块和深层特征提取模块，具体地，将中层特征信息F M和深层特征信息F H通过通过级联操作、ReLU操作和一次 1×1卷积操作，进而结合浅层特征信息F L继续进行级联操作和1×1卷积得到反馈特征信息，将前述步骤中获得的深层特征信息与中层特征信息进行特征融合操作，实现在不增加较多参数量的前提下提高特征的上下文的关联性。
所述操作公式如下：
F FB＝conv 1×1(concat(conv 1×1(ReLU((concat(F M,F H))),F L))；
其中，F FB表示反馈机制模块的输出，F L表示浅层特征提取模块的输出，F M表示中层特征提取模块的输出，F H表示深层特征提取模块的输出，concat()表示级联操作，conv 1×1表示卷积核大小为1×1的卷积操作。
步骤S50，根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。
本实施例中，获取低分辨率图像重建后的超分辨率图片信息，如图3的重建模块F R，将根据所述中层特征提取模块和深层特征提取模块输出的特征信息与所述低分辨率图像的特征信息相加得到得到低分辨率图像重建后的超分辨率图片信息，具体地，将所述深层特征信息和中层特征信息通过级联操作、ReLU操作和1×1卷积得到初始重建特征信息，并将所述初始重建特征信息与浅层特征提取进行相加操作、3×3卷积和一个亚像素卷积操作得到低分辨率图像重建后的超分辨率图片信息；
所述操作公式如下：
F R＝f sub(conv 3×3(conv 1×1(ReLU((concat(F M,F H))))+I LR)))；
I SR＝F R；
其中I SR表示超分辨率图片信息，F R表示重建模块的输出，f sub表示亚像素卷积操作。
本实施例中，通过将获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息；并将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)得到中层特征信息；以及将所述中层特征信息进行信息多重蒸馏模块(IMDB)得到深层特征信息；其中，所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；并根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。采用低分辨率图像分批次处理有效地降低了数据参数量和数据计算量，并引用了反馈机制模块可以实现提高低层和高层特征信息之间的 联系，进而实现将低分辨率图片转换成超分辨率图片，使得提高了低分辨率图片的重建效果的同时也降低了重建过程中的计算量和参数量。
进一步地，如图4所示为上述实施例中的信息多重蒸馏模块(IMDB)结构示意图，具体的详情如下：
首先，将接收到的第一初始图像特征信息依次进行四次3×3卷积操作得到对应的第一分处理目标特征信息；在图像实际处理中，首次进行3×3卷积操作的第一初始图像特征信息为中层特征提取模块F M输出的图像特征信息，后续三次3×3卷积操作的输入的图像特征信息就是前一次卷积操作完成后输出的图像特征信息，同时，前三次3×3卷积操作之前会预先进行ReLU操作，以及每一次3×3卷积后都将部分通道分离出来，作为下一层卷积的输入，剩余通道保留。具体的操作公式如下：
第一次卷积操作的公式：
Figure PCTCN2022077832-appb-000005
第二次卷积操作的公式；
Figure PCTCN2022077832-appb-000006
第三次卷积操作的公式：
Figure PCTCN2022077832-appb-000007
第四次卷积操作的公式：
Figure PCTCN2022077832-appb-000008
其中F i IMDB_in表示第i个IMDB的输入，F i refine_1和F i coarse_1分别代表被剪裁的特征映射和进行进一步特征提取特征映射，f split表示特征通道分割操作。
其次，将上述卷积操作得到的将各第一分处理目标特征信息进行级联操作得到处理后的第一目标特征信息，即将上述四次卷积所保留的16层通道重新通过concatenate操作将特征通道数合并为64层；以及将所述处理后的第一目标特征信息进行坐标注意力(CA,Coordinate Attention)机制处理和1×1卷积操作，并与接收到的第一初始图像特征信息进行相加得到一次信息多重蒸馏模块(IMDB)操作后的输出图像特征信息。
具体的操作公式如下：
Figure PCTCN2022077832-appb-000009
其中F i IMDB表示第i个IMDB的输出。
进一步地，如图5所示为上述实施例中的增强信息多重蒸馏模块(EIMDB)结构示意图，具体的详情如下：
首先，将接收到的第二初始图像特征信息依次进行四次残影模块(Ghost Module)操作得到第二目标特征信息；在图像实际处理中，首次进行残影模块(Ghost Module)操作的第二初始图像特征信息为浅层特征提取模块F L输出的图像特征信息，后续三次残影模块操作的输入的图像特征信息就是前一次残影模块操作完成后输出的图像特征信息。具体地，第一次Ghost Module操作的输入特征通道数为64，后三次Ghost Module操作的输入特征通道数为48层，前三次Ghost Module操作输出为通道数为64层，最后一次Ghost Module操作的特征通道数的输出为16。然后将前三次的输出特征层数64层分割为48层和16层，将48层作为最后将四次Ghost Module操作的输入。具体的操作公式如下：
第一次残影模块操作的公式：
Figure PCTCN2022077832-appb-000010
第二次残影模块操作的公式；
Figure PCTCN2022077832-appb-000011
第三次残影模块操作的公式：
Figure PCTCN2022077832-appb-000012
第四次残影模块操作的公式：
Figure PCTCN2022077832-appb-000013
其中F’ i EIMDB_in表示第i个EIMDB的输入，F’ i refine_1和F’ i coarse_1分别代表被剪裁的特征映射和进行进一步特征提取特征映射，f split表示特征通道分割操作。
其次，将上述残影模块操作得到的将各第一分处理目标特征信息进行级联操作得到处理后的第一目标特征信息，即将上述四次Ghost Module操作所保留的16层通道重新通过concatenate操作将特征通道数合并为64层。以及将所述处理后的第二目标特征信息进行坐标注意力(CA,Coordinate Attention)机制处理和1×1卷积操作，并与接收到的第二初始图像特征信息进行相加得到一次增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation  Network)操作后的输出图像特征信息。
具体的操作公式如下：
Figure PCTCN2022077832-appb-000014
其中F’ i EIMDB表示第i个EIMDB的输出。
进一步地，如图6所示，为图5所示的增强信息多重蒸馏模块(EIMDB)中的Ghost Module模块结构示意图，具体详情如下：
首先，将接收的第三初始图像特征信息进行1×1卷积操作得到卷积后的初始特征信息；在实际的图像处理过程中，其中首次第三初始图像特征信息是指浅层特征提取模块F L输出的图像特征信息。具体地，将接收的第三初始图像特征信息进行特征通道的划分，例如，假设输入特征通道数为M，输出Ghost Module的特征通道数为N，部分分组卷积负责去除冗余的特征通道，经过分组卷积后的特征通道数为d(0<d<M)，其中Ghost Module操作中卷积操作会预先进行ReLU操作，具体的操作公式如下：
Feat 1＝conv 1×1(ReLU(I F))；
其中Feat 1表示primary卷积后的特征映射，I F表示Ghost Module模块的输入。
其次，将所述初始特征信息进行3×3分组卷积操作得到分组特征信息，具体的操作公式如下：
Feat 2＝g_conv 3×3(ReLU(Feat 1))；
其中Feat 2表示cheap operation后的特征映射，g_conv 3×3表示卷积核大小为3×3的分组卷积。
第三，将所述初始特征信息和分组特征信息进行级联操作得到一次残影模块(Ghost Module)操作后的输出图像特征信息。具体的操作公式如下：
F GM＝concat(Feat 1,Feat 2) ；
其中，F GM表示Ghost Module模块的输出。
本实施例中的残影模块(Ghost Module)操作通过将接收的特征信息进行特征通道的划分，部分用于去除冗余的特征通道，部分用于卷积操作，进而可以实现减少整个系统运行的计算量和参数量。
进一步地，如图7所示，为图4或5所示的CA机制模块结构示意图，具体详情如下：
首先，通过接收输入的第四初始图像特征信息，所述第四初始图像特征信息包括垂直输入的特征信息和水平输入的特征信息，以及将所述垂直输入的特征信息和水平输入的特征信息分别进行两个一维全局平均池化处理得到垂直方向的感知特征信息和水平方向的感知特征信息。
具体地，所述第四初始图像特征信息是指在EIMDB操作或IMDB操作中进行级联操作后得到的图像特征信息，并将得到的图像特征信息进行垂直输入的特征信息和水平输入的特征信息的划分，以及采用两个一维全局平均池化对垂直输入的特征信息和水平输入的特征信息进行处理得到垂直方向的感知特征信息和水平方向的感知特征信息。具体的操作公式如下：
Figure PCTCN2022077832-appb-000015
Figure PCTCN2022077832-appb-000016
其中F X pool(W)和F Y pool(h)分别代表以宽为方向和以长为方向的一维池化，W和H分别为输入特征映射的宽和长，χc()表示固定位置的像素点之值。
其次，将所述垂直方向的感知特征信息和水平方向的感知特征信息进行萃取操作。
具体地，将所述垂直方向的感知特征信息和水平方向的感知特征信息进行级联操作，并分别通过通道注意力的萃取操作提取垂直或水平方向的依赖关系，进而将生成的注意力映射分别通过水平和垂直的逐像素乘法将坐标注意力信息保存在所述第四初始图像特征信息通过CA操作后的图像特征信息中，以便开启进行后续的卷积操作。
采用上述CA操作，不仅能提取重要通道的信息，还能获取空间位置信息，并且只增加了较少的计算量和参数量。
此外，为了验证得到上述提高图像分辨率的系统的处理图像信息的情况，常采用损失函数来衡量。
具体地，用到两次迭代的超分辨率图片输出，和原低分辨率图片分别进行L损失函数计算然后取平均值，可由下列公式表示：
Figure PCTCN2022077832-appb-000017
其中为θ网络参数，T＝2为总体迭代次数，t为此次迭代，I LR和I t SR分别 代表原始低分辨率图片和超分辨重建图片。
此外，本发明实施例还提出一种存储介质，所述存储介质上存储有提高图像分辨率的程序，所述提高图像分辨率的程序被处理器执行时实现如上述提高图像分辨率的方法的步骤。
本发明可读存储介质的具体实施例与上述提高图像分辨率的方法各实施例基本相同，在此不作赘述。
最后，为了验证上述提供的提高图像分辨率的方法的实际适用性情况，采用下述具体的验证实验进行说明。
实验内容：预先对网络参数进行初始化，其中，初始化学习率为10 -4，每训练200次将学习率以0.5，总共迭代1000次。使用Adam算法(β1＝0.9，β2＝0.999)对网络参数进行优化。设置batch-size大小为16，图像块大小为48×48。
实验环境采用GPU版本的Pytorch1.6.0，GPU使用RTX2070Super进行训练，操作系统为Ubuntu16.08。
如表1实验采用的训练集为DIV2K，包括800张类型为人物、手工制品、建筑(城市，村庄)、动植物以及自然风光等2K分辨率的图片。通过对DIV2K进行数据增强处理，包括旋转、翻转以及等比例缩小，进行数据增强后的图片为8000张。实验采用的测试集为Set5、Set14、BSD100、Urban100、Manga109五个广泛使用的超分辨率基准测试集进行模型性能评估。其中Urban100包含100张具有挑战性的城市场景图片，包含密集的高频特征细节。Manga109为109张漫画封面图片，具有高频和低频信息以及文字信息，考验模型对文字和图片的综合的处理能力。
Figure PCTCN2022077832-appb-000018
表1-数据集来源
实验对比
通过两种模型EIMDN-L(Ehance Information Multi-distillation Network  Large)和EIMDN-S(Ehance Information Multi-distillation Network-Small)一大一小作为实验的网络模型，EIMDB个数为N，IMDB个数为M。EIMDN-L使用的EIMDB和IMDB个数分别为N＝6、M＝6，EIMDN-S使用的EIMDB和IMDB个数分别为N＝3、M＝3。
采用峰值信噪比(PSNR,peak signal-to-noise ratio)和结构相似度(SSIM,structure similarity)作为评价标准在YCbCr色彩编码格式下选取Y通道进行评估，具体的公式如下：
Figure PCTCN2022077832-appb-000019
Figure PCTCN2022077832-appb-000020
MAX是表示图像点颜色的最大数值，MSE为原图像与处理图像之间均方误差；μ x为x的均值，μ y为y的均值，δ x为x的方差，δ y为y的方差，δ xy为x和y的协方差，c 1、c 2为常数。
并随着放大倍数的增加，图像超分辨率重建的难度也会随之增加，如下述表2展示了近几年较为优秀的轻量级超分辨率算法与本申请中的采用的提高图像分辨率的方法得到的图片效果对比，其中EIMDN-S取得了中等的重建效果。可以看出EIMDN-L在放大倍数为3倍和4倍的条件下要优于大部分模型。
Figure PCTCN2022077832-appb-000021
Figure PCTCN2022077832-appb-000022
表2在测试集上PSNR和SSIM的效果对比
基于上述表2中的内容，在放大倍数为4倍的条件下，本申请对应的模型在测试集Manga109中PSNR值比经典模型VDSR提升了2.35dB，比第二的CARN提升了0.11dB。说明本申请对应的模型能够将高放大倍数中难以学习到的高频细节通过反馈机制模块，加深对高频信息学习的深度，从而在高放大倍数中取得良好的重建效果。本申请对应的模型EIMDN-L在Urban100测试集中效果明显高于其他数据集，是因为Urban100数据集包含城市建筑的图片，高频细节较多。而本申请对应的模型中使用的CA注意力机制能够筛选通道和空间中包含高频信息较多的特征并保留，所以能够在Urban测试集中取得更好的重建效果。在放大倍数为2倍的情况下，本申请对应的模型中采用的方法相比其他模型并未总是取得最好效果，说明本申请中的EIMDB采用的Ghost Module虽然能够减少参数量，并去除特征通道的冗余通道，而较低放大倍数中每个通道保留的高频细节较多，这会导致某些重要的特征通道在去除冗余步骤中有所丢失，达不到在2倍放大倍数下最好的效果。
视觉效果对比
通过选取了信息蒸馏网络IDN(Information Distillation Network)、CARN-M(Cascading Residual Network-Mobile)、瀑布残差网络CARN(Cascading Residual Network)、信息多重蒸馏网络IMDN(Information  Multi-distillation Network)方法，在放大倍数2倍、3倍和4倍条件下，Set14和Urban100数据集中的重建图片进行视觉效果对比。如图8和图9所示，对于Set14中的第一测试图片和Urban100中的第二测试图片，本申请对应的模型EIMDN-L能较好地恢复女性的头巾以及玻璃格栅的正确纹理。而EIMDN-S也和其他轻量级方法恢复效果相似。此外，如图10和图11所示，对于在数据集Urban中的第三测试图片和第四测试图片，也可以观察到本申请对应的模型的重建效果更有利，可以恢复更多的细节。本申请对应的模型EIMDN-S与对比方法相比具有相似的性能，但EIMDN-L方法明显优于对比方法。
进一步地，构建一个轻量级的SR模型，需要在减少网络参数量的同时保证重建效果。如图12所示，在Urban100数据集中建立在2倍放大倍数下的参数量对比，对比SOTA方法，本申请对应的模型EIMDN-L在参数量略微增加的条件下获得了不错的PSNR效果。而EIMDN-S也在更低参数量的情况下取得了较好的效果，达到了图像超分辨率重建与模型大小之间的权衡。
进一步地，通过使用CARN方法中提出的Multi-Add运算来评估模型的计算复杂度，即对单个图像尺寸为720p的条件下进行复合乘法累加运算的次数统计。如表3所示，与SOTA算法对比，本申请对应的模型中EIMDN-S保持了较低的计算量。而本申请对应的模型EIMDN-L在获得了更好的重建效果情况下略微提升了计算量。
Figure PCTCN2022077832-appb-000023
表3在2倍、3倍和4倍放大倍数下模型的计算量对比
结果分析
通过两组消融实验对结果进行分析。
第一组，加入反馈机制模块和使用CA注意力机制代替IMDB原有的CCA注意力机制对超分辨率图像重建的影响，如表4所示。可见，第一组中采用EIMDN-S模型的图像，最终得到的图像重建效果更好。
Figure PCTCN2022077832-appb-000024
表4加入反馈机制模块和CA注意力机制对网络重建效果的影响
第二组，对比不同个数的IMDB和EIMDB模块对超分辨率重建效果的影响，如表5所示。可见，第二组中IMDB和EIMDB模块的数量越多最后得到的PSNR和SSIM效果越好。
Figure PCTCN2022077832-appb-000025
表5 EIMDB和IMDB在网络中的个数PSNR和SSIM效果对比
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者系统不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者系统所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者系统中还存在另外的相同要素。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是提高图像分辨率的系统、物联网摄像头设备等)执行本发明各个实施例所述的方法。
以上仅为本发明的优选实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。

Claims (10)

  1. 一种提高图像分辨率的方法，其特征在于，所述方法包括以下步骤：
    获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息；
    将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作得到中层特征信息；
    将所述中层特征信息进行信息多重蒸馏模块(IMDB，Information Multi-Distillation Block)操作得到深层特征信息；
    根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；
    根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。
  2. 根据权利要求1中所述的提高图像分辨率的方法，其特征在于，所述信息多重蒸馏模块(IMDB)操作的步骤，包括：
    将接收到的第一初始图像特征信息依次进行四次3×3卷积操作得到对应的第一分处理目标特征信息；
    将各第一分处理目标特征信息进行级联操作得到处理后的第一目标特征信息；
    将所述处理后的第一目标特征信息进行坐标注意力(CA,Coordinate Attention)机制处理和1×1卷积，并与接收到的第一初始图像特征信息进行相加得到一次信息多重蒸馏模块(IMDB)操作后的输出图像特征信息。
  3. 根据权利要求1或2所述的提高图像分辨率的方法，其特征在于，所述增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作的步骤，包括：
    将接收到的第二初始图像特征信息依次进行四次残影模块(Ghost Module)操作得到第二分处理目标特征信息；
    将各第二分处理目标特征信息进行级联操作得到处理后的第二目标特征 信息；
    将所述处理后的第二目标特征信息进行坐标注意力(CA,Coordinate Attention)机制处理和1×1卷积操作，并与接收到的第二初始图像特征信息进行相加得到一次增强信息多重蒸馏模块(EIMDB，Enhance Information Multi-Distillation Network)操作后的输出图像特征信息。
  4. 根据权利要求3中所述的提高图像分辨率的方法，其特征在于，所述残影模块(Ghost Module)操作的步骤，包括：
    将接收的第三初始图像特征信息进行1×1卷积操作得到卷积后的初始特征信息；
    将所述初始特征信息进行3×3分组卷积操作得到分组特征信息；
    将所述初始特征信息和分组特征信息进行级联操作得到一次残影模块(Ghost Module)操作后的输出图像特征信息。
  5. 根据权利要求1所述的提高图像分辨率的方法，其特征在于，所述获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息的步骤，包括：
    将获取的低分辨率图像的特征信息采用3×3卷积和1×1卷积操作输出浅层特征信息，所述操作公式如下：
    F L＝conv 1×1(ReLU((conv 3×3(ReLU(I LR)))))；
    其中，F L表示浅层特征信息，conv 1×1和conv 3×3分别表示卷积核大小为1×1和3×3的卷积操作，I LR表示输入的低分辨率图像的特征信息，ReLU()是激活函数；
    所述将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)操作得到中层特征信息的步骤，包括：
    将所述浅层特征信息、反馈特征信息采用N个EIMDB模块来进行中层特征信息的提取，所述操作公式如下：
    Figure PCTCN2022077832-appb-100001
    其中，F M表示中层特征信息，F i EIMDB表示中层特征提取里第i个EIMDB模块的输出信息(1<i≤N),F FB表示反馈特征信息。
  6. 根据权利要求1或5所述的提高图像分辨率的方法，其特征在于，所述将所述中层特征信息进行信息多重蒸馏模块(IMDB)操作得到深层特征信息的步骤，包括：
    将所述中层特征信息采用M个IMDB模块进行深层特征信息的提取，所述操作公式如下：
    Figure PCTCN2022077832-appb-100002
    其中，F H表示深层特征信息，F j IMDB表示深层特征提取里第j个IMDB模块的输出(1<j≤M)；
    所述根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息的步骤，包括：
    将所述浅层特征信息、中层特征信息和所述深层特征信息经过级联操作和两次1×1卷积得到反馈特征信息，所述操作公式如下：
    F FB＝conv 1×1(concat(conv 1×1(ReLU((concat(F M,F H))),F L))；
    其中，F FB表示反馈特征信息，F L表示浅层特征信息，F M表示中层特征信息，F M表示中层特征信息。
  7. 根据权利要求6所述的提高图像分辨率的方法，其特征在于，所述根据所述中层特征信息、深层特征信息和所低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息的步骤，包括：
    将所述深层特征信息和中层特征信息通过级联操作和1×1卷积得到初始重建特征信息并将所述初始重建特征信息与浅层特征提取进行相加操作、3×3卷积和一个亚像素卷积操作得到低分辨率图像重建后的超分辨率图片信息；
    所述操作公式如下：
    F R＝f sub(conv 3×3(conv 1×1(ReLU((concat(F M,F H))))+I LR)))；
    I SR＝F R；
    其中I SR表示超分辨率图片信息，F R表示图像重建特征信息，f sub表示亚像素卷积操作。
  8. 一种提高图像分辨率的系统，其特征在于，包括：
    浅层特征提取模块，用于获取低分辨率图像的特征信息并进行卷积操作得到浅层特征信息；
    中层特征提取模块，用于将所述浅层特征信息和反馈特征信息进行增强信息多重蒸馏模块(EIMDB)操作得到中层特征信息；
    深层特征提取模块，用于将所述中层特征信息进行信息多重蒸馏模块(IMDB)操作得到深层特征信息；
    反馈机制模块，用于根据所述浅层特征信息、中层特征信息和所述深层特征信息得到所述反馈特征信息；
    重建模块，用于根据所述中层特征信息、深层特征信息和所述低分辨率图像的特征信息得到低分辨率图像重建后的超分辨率图片信息。
  9. 一种提高图像分辨率的装置，其特征在于，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的提高图像分辨率的程序，所述提高图像分辨率的程序被所述处理器执行时实现如权利要求1至7中任一项所述的提高图像分辨率的方法的步骤。
  10. 一种存储介质，其特征在于，所述存储介质上存储有提高图像分辨率的程序，所述提高图像分辨率的程序被处理器执行时实现如权利要求1至7中任一项所述的提高图像分辨率的方法的步骤。
PCT/CN2022/077832 2021-08-03 2022-02-25 提高图像分辨率的方法、系统、装置及存储介质 WO2023010831A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110884842.8 2021-08-03
CN202110884842.8A CN113658044B (zh) 2021-08-03 2021-08-03 提高图像分辨率的方法、系统、装置及存储介质

Publications (1)

Publication Number Publication Date
WO2023010831A1 true WO2023010831A1 (zh) 2023-02-09

Family

ID=78490640

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/077832 WO2023010831A1 (zh) 2021-08-03 2022-02-25 提高图像分辨率的方法、系统、装置及存储介质

Country Status (2)

Country Link
CN (1) CN113658044B (zh)
WO (1) WO2023010831A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117078516A (zh) * 2023-08-11 2023-11-17 济宁安泰矿山设备制造有限公司 基于残差混合注意力的矿井图像超分辨率重建方法
CN117132472A (zh) * 2023-10-08 2023-11-28 兰州理工大学 基于前向-反向可分离自注意力的图像超分辨率重建方法
CN117152546A (zh) * 2023-10-31 2023-12-01 江西师范大学 一种遥感场景分类方法、系统、存储介质及电子设备
CN117132472B (zh) * 2023-10-08 2024-05-31 兰州理工大学 基于前向-反向可分离自注意力的图像超分辨率重建方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113658044B (zh) * 2021-08-03 2024-02-27 长沙理工大学 提高图像分辨率的方法、系统、装置及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111353938A (zh) * 2020-02-29 2020-06-30 杭州电子科技大学 一种基于网络反馈的图像超分辨率学习方法
CN111882485A (zh) * 2020-06-19 2020-11-03 北京交通大学 分级特征反馈融合的深度图像超分辨率重建方法
US20200394759A1 (en) * 2019-06-11 2020-12-17 Samsung Electronics Co., Ltd. Asymmetric normalized correlation layer for deep neural network feature matching
CN112200724A (zh) * 2020-10-22 2021-01-08 长沙理工大学 一种基于反馈机制的单图像超分辨率重建系统及方法
CN113658044A (zh) * 2021-08-03 2021-11-16 长沙理工大学 提高图像分辨率的方法、系统、装置及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7812303B2 (en) * 2007-12-06 2010-10-12 The United States Of America As Represented By The Secretary Of The Army Method and system for creating an image using quantum properties of light based upon spatial information from a second light beam which does not illuminate the subject
CN112819910B (zh) * 2021-01-08 2022-12-27 上海理工大学 基于双鬼注意力机制网络的高光谱图像重建方法
CN113052254B (zh) * 2021-04-06 2022-10-04 安徽理工大学 多重注意力幽灵残差融合分类模型及其分类方法
CN113160062B (zh) * 2021-05-25 2023-06-06 烟台艾睿光电科技有限公司 一种红外图像目标检测方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200394759A1 (en) * 2019-06-11 2020-12-17 Samsung Electronics Co., Ltd. Asymmetric normalized correlation layer for deep neural network feature matching
CN111353938A (zh) * 2020-02-29 2020-06-30 杭州电子科技大学 一种基于网络反馈的图像超分辨率学习方法
CN111882485A (zh) * 2020-06-19 2020-11-03 北京交通大学 分级特征反馈融合的深度图像超分辨率重建方法
CN112200724A (zh) * 2020-10-22 2021-01-08 长沙理工大学 一种基于反馈机制的单图像超分辨率重建系统及方法
CN113658044A (zh) * 2021-08-03 2021-11-16 长沙理工大学 提高图像分辨率的方法、系统、装置及存储介质

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117078516A (zh) * 2023-08-11 2023-11-17 济宁安泰矿山设备制造有限公司 基于残差混合注意力的矿井图像超分辨率重建方法
CN117078516B (zh) * 2023-08-11 2024-03-12 济宁安泰矿山设备制造有限公司 基于残差混合注意力的矿井图像超分辨率重建方法
CN117132472A (zh) * 2023-10-08 2023-11-28 兰州理工大学 基于前向-反向可分离自注意力的图像超分辨率重建方法
CN117132472B (zh) * 2023-10-08 2024-05-31 兰州理工大学 基于前向-反向可分离自注意力的图像超分辨率重建方法
CN117152546A (zh) * 2023-10-31 2023-12-01 江西师范大学 一种遥感场景分类方法、系统、存储介质及电子设备
CN117152546B (zh) * 2023-10-31 2024-01-26 江西师范大学 一种遥感场景分类方法、系统、存储介质及电子设备

Also Published As

Publication number Publication date
CN113658044A (zh) 2021-11-16
CN113658044B (zh) 2024-02-27

Similar Documents

Publication Publication Date Title
WO2023010831A1 (zh) 提高图像分辨率的方法、系统、装置及存储介质
CN113392711B (zh) 一种基于高层语义与噪声抑制的烟雾语义分割方法及系统
CN110717921B (zh) 改进型编码解码结构的全卷积神经网络语义分割方法
CN108875751B (zh) 图像处理方法和装置、神经网络的训练方法、存储介质
CN110958469A (zh) 视频处理方法、装置、电子设备及存储介质
WO2023212997A1 (zh) 基于知识蒸馏的神经网络训练方法、设备及存储介质
CN111951164A (zh) 一种图像超分辨率重建网络结构及图像重建效果分析方法
CN109949234A (zh) 基于深度网络的视频复原模型训练方法及视频复原方法
CN113902925A (zh) 一种基于深度卷积神经网络的语义分割方法及系统
CN113436210A (zh) 一种融合上下文逐级采样的道路图像分割方法
CN115601236A (zh) 一种基于特征信息蒸馏网络的遥感图像超分辨重建方法
CN115294055A (zh) 图像处理方法、装置、电子设备和可读存储介质
CN111951171A (zh) Hdr图像生成方法、装置、可读存储介质及终端设备
CN114266952A (zh) 基于深监督的实时语义分割方法
CN117097853A (zh) 一种基于深度学习的实时抠像方法及系统
CN115965844B (zh) 基于视觉显著性先验知识的多聚焦图像融合方法
CN115861922B (zh) 一种稀疏烟火检测方法、装置、计算机设备及存储介质
CN116977208A (zh) 双分支融合的低照度图像增强方法
CN110675320A (zh) 一种空间参数变化及复杂场景下目标图像清晰化方法
Liang et al. Multi-scale and multi-patch transformer for sandstorm image enhancement
CN112488115B (zh) 一种基于two-stream架构的语义分割方法
CN115170812A (zh) 图像去噪模型训练、降噪方法及其设备、存储介质
CN113436198A (zh) 一种协同图像超分辨率重建的遥感图像语义分割方法
Sun et al. Two-Stage Deep Single Image Super-Resolution with Multiple Blur Kernels for the Internet of Things
CN115601242B (zh) 一种适用于硬件部署的轻量级图像超分辨率重建方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22851558

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE