WO2023217270A1 - 图像超分方法、超分网络参数调整方法、相关装置及介质 - Google Patents

图像超分方法、超分网络参数调整方法、相关装置及介质 Download PDF

Info

Publication number
WO2023217270A1
WO2023217270A1 PCT/CN2023/093880 CN2023093880W WO2023217270A1 WO 2023217270 A1 WO2023217270 A1 WO 2023217270A1 CN 2023093880 W CN2023093880 W CN 2023093880W WO 2023217270 A1 WO2023217270 A1 WO 2023217270A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
feature
resolution
super
features
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/093880
Other languages
English (en)
French (fr)
Inventor
孔方圆
刘松伟
李明蹊
拜阳
陈芳民
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202210524706.2A external-priority patent/CN117114981A/zh
Priority claimed from CN202210524546.1A external-priority patent/CN117114980A/zh
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to US18/858,296 priority Critical patent/US20250272786A1/en
Publication of WO2023217270A1 publication Critical patent/WO2023217270A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks

Definitions

  • the present disclosure relates to the field of image processing technology, and in particular, to an image super-resolution method, a super-resolution network parameter adjustment method, related devices and media.
  • Image super-resolution technology is a technology that restores high-resolution images from low-resolution images. Since the image super-resolution business has now become a key business in image quality enhancement, image super-resolution technology is one of the current research hotspots in the field of image processing.
  • CNN Convolutional Neural Network
  • embodiments of the present disclosure provide an image super-resolution method, including:
  • the first image features are processed by a feature processing module to obtain second image features
  • the feature processing module includes: features composed of at least one set of concatenated first convolutional layers and first activation function layers.
  • the thinning unit is used to process the input features of the feature processing module.
  • the feature fusion unit is used to perform additive fusion of the input features of the feature processing module and the output features of the feature refinement unit.
  • Attention Unit used to process the output features of the feature fusion unit and obtain the output features of the feature processing module;
  • a super-resolution image corresponding to the image to be super-resolved is generated according to the first image feature and the second image feature.
  • the feature processing module further includes: a channel number adjustment unit, wherein the channel number adjustment unit is used to adjust the output features of the feature fusion unit before the attention unit processes the output features of the feature fusion unit.
  • the output feature of the feature fusion unit is adjusted to a feature with the same number of feature channels as the first image feature.
  • the first image feature is processed by a feature processing module, and obtaining the second image feature includes:
  • the first image features are processed through the multi-stage feature processing modules connected in series to obtain the second image features.
  • the activation function of the first activation function layer is a linear rectification function ReLU function.
  • the channel number adjustment unit includes: a second convolution layer, wherein the size of the convolution kernel of the second convolution layer is 1*1.
  • performing feature extraction on the image to be super-resolved, and obtaining the first image features includes:
  • the image to be super-resolved is convolved through a third convolution layer to obtain the first image feature.
  • generating a super-resolution image corresponding to the image to be super-resolved based on the first image feature and the second image feature includes:
  • a super-resolution image corresponding to the image to be super-resolved is generated according to the sixth image feature.
  • embodiments of the present disclosure provide a method for adjusting parameters of a super-resolution network.
  • the method includes: using a feature extractor to respectively extract feature information of the first reconstructed image, feature information of the high-resolution image, and feature information of the upsampled image.
  • the feature extractor includes at least one convolution activation module, which is composed of a convolution layer and an activation function.
  • the first reconstructed image is a super-resolution image obtained by reconstructing a low-resolution image through a super-resolution network.
  • the high-resolution image and the low-resolution image are the training sample images of the super-resolution network
  • the high-resolution image and the low-resolution image are images with the same visual content at different resolutions
  • the upsampling image is the pair
  • the image obtained by upsampling the low-resolution image, the first reconstructed image, the high-resolution image and the upsampled image have the same resolution; based on the feature information of the first reconstructed image, the features of the high-resolution image information and the feature information of the upsampled image, calculate the contrast loss function, and obtain the first contrast loss function value based on the first reconstructed image; based on the first contrast loss function value, adjust the parameters of the super-resolution network so that the adjusted The super-resolution network reconstructs the low-resolution image to obtain a second reconstructed image, and the second contrast loss function value based on the second reconstructed image is smaller than the first contrast loss function value.
  • the convolutional layers of each convolutional activation module in the at least one convolutional activation module are the same.
  • the at least one convolutional activation module is connected in series.
  • the upsampling process is a bicubic interpolation process.
  • the contrast loss function is a ratio of a first difference term and a second difference term.
  • the first difference term is used to characterize the difference between the first reconstructed image and the high-resolution image
  • the second difference term is used to characterize The difference between this upsampled image and this high-resolution image.
  • the first difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term is the L1 distance between the feature information of the upsampled image and the high-resolution image. L1 distance of feature information.
  • the contrast loss function is a ratio of a third difference term to a fourth difference term.
  • the third difference term is used to characterize the difference between the first reconstructed image and the high-resolution image.
  • the fourth difference term is used to characterize The difference between the first reconstructed image and this upsampled image.
  • the third difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image; the fourth difference term is the feature information of the first reconstructed image and the features of the upsampled image. L1 distance of information.
  • an image super-resolution device including:
  • the feature extraction module is used to extract features from the super-resolution image to be processed and obtain the first image features
  • a feature processing module configured to process the first image features through a feature processing module to obtain second image features
  • the feature processing module includes: a first convolution layer and a third convolution layer connected in series.
  • a feature refinement unit composed of an activation function layer, used to process the input features of the feature processing module, and a feature fusion unit, used to process the input features of the feature processing module and the output features of the feature refinement unit Perform additive fusion, and the attention unit is used to process the output features of the feature fusion unit and obtain the output features of the feature processing module;
  • An image reconstruction group configured to generate a super-resolution image corresponding to the image to be super-resolved according to the first image feature and the second image feature.
  • the feature processing module further includes: a channel number adjustment unit;
  • the channel number adjustment unit is configured to adjust the output features of the feature fusion unit to features with the same number of channels as the first image feature before the attention unit processes the output features of the feature fusion unit. .
  • the feature processing module 102 is specifically configured to process the first image feature through a multi-stage serially connected feature processing module to obtain the second image feature.
  • the activation function of the first activation function layer is a linear rectification function ReLU function.
  • the channel number adjustment unit includes: a second convolution layer
  • the size of the convolution kernel of the second convolution layer is 1*1.
  • the feature extraction module is specifically configured to perform convolution processing on the image to be super-resolved through a third convolution layer to obtain the first image features.
  • the image reconstruction group is specifically configured to process the second image feature through a fourth convolution layer to obtain a third image feature; compare the first image feature and the third image feature.
  • the image features are additively fused to obtain the fourth image feature; the fourth image feature is processed through the fifth convolution layer to obtain the fifth image feature; the fifth image feature is upsampled to obtain the sixth image feature ; Generate a super-resolution image corresponding to the image to be super-resolved according to the sixth image feature.
  • inventions of the present disclosure provide a super-resolution network parameter adjustment device.
  • the device includes: a feature extraction module, a calculation module and an adjustment module, wherein the feature extraction module is used to respectively extract the first parameters through a feature extractor.
  • the feature information of the reconstructed image, the feature information of the high-resolution image and the feature information of the upsampled image, the feature extractor includes at least one convolution activation module, the convolution activation module consists of a convolution layer and an activation function, the first reconstruction
  • the image is a super-resolution image obtained by reconstructing a low-resolution image through a super-resolution network.
  • the high-resolution image and the low-resolution image are the training sample images of the super-resolution network.
  • the high-resolution image and the low-resolution image are The images are images with the same visual content at different resolutions.
  • the upsampled image is an image obtained by upsampling the low-resolution image.
  • the first reconstructed image, the high-resolution image and the upsampled image have the same resolution. rate; the calculation module is used to calculate the contrast loss function based on the feature information of the first reconstructed image extracted by the feature extraction module, the feature information of the high-resolution image and the feature information of the upsampled image, and obtain the result based on the first reconstructed image.
  • the first contrast loss function value of the image is used to adjust the parameters of the super-resolution network based on the first contrast loss function value obtained by the calculation module, so that the low-resolution image can be processed through the adjusted super-resolution network.
  • the image is reconstructed to obtain a second reconstructed image, and the second contrast loss function value based on the second reconstructed image is smaller than the first contrast loss function value.
  • the convolutional layers of each convolutional activation module in the at least one convolutional activation module are the same.
  • the at least one convolutional activation module is connected in series.
  • the upsampling process is a bicubic interpolation process.
  • the contrast loss function is a ratio of a first difference term and a second difference term.
  • the first difference term is used to characterize the difference between the first reconstructed image and the high-resolution image
  • the second difference term is used to characterize The difference between this upsampled image and this high-resolution image.
  • the first difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term is the L1 distance between the feature information of the upsampled image and the high-resolution image. L1 distance of feature information.
  • the contrast loss function is a ratio of a third difference term to a fourth difference term.
  • the third difference term is used to characterize the difference between the first reconstructed image and the high-resolution image.
  • the fourth difference term is used to characterize The difference between the first reconstructed image and this upsampled image.
  • the third difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image; the fourth difference term is the feature information of the first reconstructed image and the features of the upsampled image. L1 distance of information.
  • embodiments of the present disclosure provide an electronic device, including: a memory and a processor, the memory is used to store a computer program; the processor is used to enable the electronic device to implement any of the foregoing when calling the computer program.
  • embodiments of the present disclosure provide a computer-readable storage medium, which when the computer program is executed by a computing device, causes the computing device to implement the image super-resolution method described in any of the foregoing embodiments and/or any of the foregoing.
  • embodiments of the present disclosure provide a computer program product.
  • the computer program product When the computer program product is run on a computer, it causes the computer to implement the image super-resolution method described in any of the foregoing embodiments and/or any of the foregoing embodiments. Described super-resolution network parameter adjustment method
  • an embodiment of the present disclosure provides a chip.
  • the chip includes a processor and a communication interface.
  • the communication interface is coupled to the processor.
  • the processor is used to run program instructions to implement the super-resolution network as described in the first aspect. Parameter adjustment method.
  • Figure 1 is a flow chart of an image super-resolution method provided by some embodiments of the present disclosure
  • Figure 2 is a schematic structural diagram of a feature processing module provided by some embodiments of the present disclosure.
  • Figure 3 is a schematic structural diagram of a feature processing module provided by other embodiments of the present disclosure.
  • Figure 4 is a schematic structural diagram of a feature processing module provided by some embodiments of the present disclosure.
  • Figure 5 is a schematic structural diagram of a feature processing module provided by some further embodiments of the present disclosure.
  • Figure 6 is a schematic structural diagram of a feature processing module provided by some embodiments of the present disclosure.
  • Figure 7 is a step flow chart of an image super-resolution method provided by other embodiments of the present disclosure.
  • Figure 8 is a schematic structural diagram of an image super-resolution network model provided by some embodiments of the present disclosure.
  • Figure 9 is a schematic structural diagram of an image super-resolution network model provided by other embodiments of the present disclosure.
  • Figure 10 is a schematic structural diagram of an image super-resolution device provided by some embodiments of the present disclosure.
  • Figure 11 is a schematic diagram of the hardware structure of an electronic device provided by some embodiments of the present disclosure.
  • Figure 12 is a schematic structural diagram of a super-resolution network parameter adjustment system provided by some embodiments of the present disclosure.
  • Figure 13 is a schematic flowchart of a method for adjusting parameters of a super-resolution network provided by some embodiments of the present disclosure
  • Figure 14 is a schematic structural diagram of a feature extractor provided by some embodiments of the present disclosure.
  • Figure 15 is a structural block diagram of a super-resolution network parameter adjustment device provided by some embodiments of the present disclosure.
  • Figure 16 is a structural block diagram of an electronic device provided by some embodiments of the present disclosure.
  • words such as “first” and “second” are used to describe the same or similar items that have basically the same functions and effects.
  • words such as “first” and “second” do not limit the quantity and execution order.
  • the first image feature and the second image feature are only used to distinguish different features, but do not limit the order of features.
  • words such as “exemplary” or “for example” are used to represent examples, illustrations or explanations. bright. Any embodiment or design described as “exemplary” or “such as” in the present disclosure is not intended to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words “exemplary” or “such as” is intended to present the concept in a concrete manner. Furthermore, in the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of “plurality” means two or more.
  • the main reason why the image super-resolution model runs slowly is that in the process of image super-resolution through the image super-resolution model, after feature extraction of low-resolution images, the extracted image features need to be further processed to obtain information for reconstruction.
  • the characteristics of the super-resolution image corresponding to the low-resolution image, and the model structure of the image super-resolution model in the existing technology is very unfavorable for model acceleration. Therefore, how to improve the running speed of the image super-resolution model is an urgent problem to be solved.
  • Embodiments of the present disclosure provide an image super-resolution method. As shown in FIG. 1 , the image super-resolution method includes the following steps: S11 to S13. A super-resolution image can be generated through a super-resolution network (image super-resolution model).
  • step S11 feature extraction is performed on the image to be super-resolved to obtain first image features.
  • the image to be super-resolved in the embodiment of the present disclosure refers to the low-resolution image corresponding to the high-resolution image ultimately desired.
  • the image to be super-resolved can be an image of any resolution and any format.
  • the image to be super-resolved can be an RGB image with a resolution of 960*540.
  • the feature extraction method for feature extraction of the super-resolution image is not limited, as long as the feature extraction of the super-resolution image to be performed can be performed.
  • the above step S11 (performing feature extraction on the target super-resolution image and obtaining the first image features) includes:
  • the image to be super-resolved is processed through a third convolution layer to obtain the first image feature.
  • the size of the convolution kernel of the third convolution layer may be 3*3, and the step size of the third convolution layer may be 2.
  • step S12 the first image features are processed by a feature processing module to obtain second image features.
  • the feature processing module includes: a feature refinement unit composed of at least a set of concatenated first convolution layers and first activation function layers, used to process the input features of the feature processing module; feature fusion Unit, used for additive fusion of the input features of the feature processing module and the output features of the feature refinement unit; attention unit (Expanded Self Attention, ESA), used for the output features of the feature fusion unit Perform processing to obtain the output features of the feature processing module.
  • a feature refinement unit composed of at least a set of concatenated first convolution layers and first activation function layers, used to process the input features of the feature processing module
  • feature fusion Unit used for additive fusion of the input features of the feature processing module and the output features of the feature refinement unit
  • attention unit Exponed Self Attention, ESA
  • step S12 processing the first image features through a feature processing module
  • Processing, obtaining the second image features includes:
  • the first image feature is processed by a first-level feature processing module to obtain the second image feature.
  • the model structure to implement the above step S12 includes: feature processing module 200, and the feature processing module 200
  • the input feature is the first image feature F 1 obtained by performing feature extraction on the to-be-super-resolved image
  • the output feature of the feature processing module 200 is the second image feature F 2 .
  • the feature processing module 200 is composed of at least one set of concatenated first convolutional layers 211 and first activation function layers 212 and a feature refining unit 21 (in Figure 2, the feature refining unit 21 includes three concatenated sets of first convolutional layers).
  • the product layer 211 and the first activation function layer 212 are shown as an example), the feature refinement unit 21 is used to process the input features (first image features F 1 ) of the feature processing module; the feature fusion unit 22 is used to process The input features of the feature processing module (first image feature F 1 ) and the output features of the feature refinement unit are additively fused; the attention unit 23 is used to process the output features of the feature fusion unit, Obtain the output feature (second image feature F 2 ) of the feature processing module.
  • the feature processing process of the feature processing module 200 includes: first processing the first image feature F 1 through the first set of concatenated first convolution layer 211 and the first activation function layer 212 , and outputting the feature F re1 , and then through The second set of concatenated first convolution layers 211 and first activation function layers 212 processes F re1 and outputs the feature F re2 , which is then passed through the third set of concatenated first convolution layers 211 and first activation functions.
  • the layer 212 processes F re2 and outputs the feature F re3 (the output feature of the feature refinement unit 21); then the first image feature F 1 and the output feature F re3 of the feature refinement unit 21 are processed by the feature fusion unit 22 Through additive fusion, the output feature F re of the feature fusion unit 22 is obtained; finally, the output feature F re of the feature fusion unit 22 is processed by the attention unit 23 to obtain the first image feature F 2 .
  • RM 1 the processing of features by the first concatenated convolution layer 211 and the first activation function layer 212
  • RM 2 the processing of features by the third group of concatenated first convolution layer 211 and the first activation function layer 212
  • ESA the processing of features by the attention unit 23
  • F re1 RM 1 (F in ) (1)
  • F re2 RM 2 (F re1 ) (2)
  • F re3 RM 3 (F re2 ) (3)
  • F re F 1 +F re3 (4)
  • F 2 ESA(F re ) (5)
  • step S12 processing the first image features through a feature processing module to obtain second image features
  • step S12 includes:
  • the first image features are processed through multi-level series-connected feature processing modules to obtain the second image features.
  • the multi-level series-connected feature processing module may include a 6-level series-connected feature processing module, an 8-level series-connected feature processing module, etc., which are not limited in this embodiment of the disclosure.
  • the model structure that implements the above step S12 includes: a multi-level serially connected feature processing module 200 .
  • the input feature of the first-level feature processing module 200 is the first image feature F 1
  • the input feature F in of the n-th level feature processing module 200 is the output feature of the n-1 level feature processing module 200
  • the n-th level feature The output feature F out of the processing module 200 is the input feature of the n+1-th level feature processing module 200.
  • the processing method of each feature processing module 200 on the input features is the same as above, and will not be described again here.
  • the feature processing module 200 further includes: a channel number adjustment unit 24.
  • the channel number adjustment unit 24 is configured to adjust the output feature F re of the feature fusion unit 22 to be consistent with the first feature F re before the attention unit 23 processes the output feature F re of the feature fusion unit 22 .
  • An image feature F1 has the same number of channels.
  • the channel number of the output feature F re of the feature fusion unit 22 is also adjusted by the channel number adjustment unit 24 to a feature that is the same as the channel number of the first image feature F 1 , and then input into the attention unit 23.
  • the number of channels of the output feature of the channel number adjustment unit 24 and the number of channels of the first feature F 1 may both be 52.
  • the channel number adjustment unit 24 includes: a second convolution layer, where the size of the convolution kernel of the second convolution layer is 1*1.
  • the output feature F re of the feature fusion unit 22 is a feature with the same number of channels as the first image feature F 1 through the second convolution layer with a convolution kernel size of 1*1, the above embodiment can The output feature of the feature fusion unit 22 is adjusted to a feature with the same number of feature channels as the first image feature without changing the spatial size of the output feature F re of the feature fusion unit 22 .
  • the size of the convolution kernel of the first convolution layer 211 is 3*3.
  • the activation function of the first activation function layer 212 is a linear rectification function (Linear rectification function, ReLU function).
  • the network model structure that implements step S12 above can be shown in Figure 5.
  • the attention unit 23 includes:
  • the size of the convolution kernel of the sixth convolution layer 231 is 1*1, which is used to process the output features of the channel number adjustment unit.
  • the size of the convolution kernel of the seventh convolution layer 232 is 3*3, and the step size is a preset value, which is used to process the output features of the sixth convolution layer.
  • the pooling layer 233 is used to process the output features of the seventh convolution layer.
  • the convolution group 234 includes a plurality of convolution layers with a convolution kernel size of 3*3 connected in series, and is used to process the output features of the pooling layer.
  • the upsampling layer 235 is used to upsample the output features of the pooling layer.
  • the first fusion layer 236 is used for additive fusion of the output features of the sixth convolution layer and the output features of the upsampling layer.
  • the size of the convolution kernel of the eighth convolution layer 237 is 1*1, and is used to process the output features of the first fusion layer.
  • the second activation function layer 238 is used to process the output features of the first fusion layer.
  • the second fusion layer 239 is used to multiply and fuse the output features of the second activation function layer and the output features of the channel number adjustment unit to obtain the output features of the corresponding feature processing module.
  • step S13 a super-resolution image corresponding to the image to be super-resolved is generated according to the first image feature and the second image feature.
  • the image super-resolution method provided by the embodiments of the present disclosure first performs feature extraction on the image to be super-resolved to obtain the first image features, and then processes the first image features through the feature processing module to obtain the second image features.
  • An image feature and the second image feature generate a super-resolution image corresponding to the image to be super-resolved.
  • the feature processing module includes: a feature refinement unit composed of at least one set of concatenated first convolution layers and first activation function layers, used to process input features of the feature processing module; feature fusion unit, use for performing additive fusion on the input features of the feature processing module and the output features of the feature refinement unit; the attention unit is used for processing the output features of the feature fusion unit and obtaining the features of the feature processing module.
  • Output features Compared with the image super-resolution model (i.e., super-resolution network) in the prior art, which requires parallel processing of image features, the image super-resolution method provided by the embodiment of the present disclosure processes the first image feature through the feature processing module.
  • each unit in the feature processing module performs feature processing in a serial manner. Therefore, embodiments of the present disclosure can improve the running speed of the image super-resolution model.
  • the feature processing module in the above embodiments can also be replaced with a model structure that distills and refines features in parallel at each stage, and processes features progressively in multiple stages.
  • the replaced image super The sub-model can achieve an image super-resolution effect similar to that of the image super-resolution model before replacement.
  • each stage requires parallel distillation and refinement of features. Therefore, the running speed of the image super-resolution model after replacement will be much lower than the image super-resolution model before replacement.
  • the image super-resolution method includes the following steps: S701 to S707.
  • step S701 feature extraction is performed on the image to be super-resolved to obtain first image features.
  • the above step S701 (performing feature extraction on the to-be-super-resolved image and obtaining the first image features) includes:
  • the image to be super-resolved is processed through a third convolution layer to obtain the first image feature.
  • the size of the convolution kernel of the third convolution layer is 3*3.
  • the image to be super-resolved is processed through a convolution layer with a convolution kernel of 3*3 to obtain the image features of the image to be super-resolved, and the image features of the image to be super-resolved are used as the first in the embodiment of the present disclosure.
  • Image features are used as the first in the embodiment of the present disclosure.
  • step S702 the first image features are processed by a feature processing module to obtain second image features.
  • the feature processing module includes: a feature refinement unit composed of at least a set of concatenated first convolutional layers and first activation function layers, used to perform input features on the feature processing module. Processing; a feature fusion unit, used for additive fusion of the input features of the feature processing module and the output features of the feature refinement unit; an attention unit, used for processing the output features of the feature fusion unit, Obtain the output features of the feature processing module.
  • step S703 the second image feature is processed through the fourth convolution layer to obtain the third image feature.
  • the size of the convolution kernel of the fourth convolution layer is 3*3.
  • step S704 the first image feature and the third image feature are additively fused to obtain a fourth image feature.
  • step S705 the fourth image feature is processed through the fifth convolution layer to obtain the fifth image feature.
  • the size of the convolution kernel of the eighth convolution layer is 3*3.
  • step S706 the fifth image feature is upsampled to obtain a sixth image feature.
  • step S707 a super-resolution image corresponding to the image to be super-resolved is generated according to the sixth image feature.
  • the image super-resolution network model (i.e., super-resolution network) that implements the image super-resolution method shown in Figure 8 includes: a feature extraction module 81, a multi-stage series The feature processing module 82, the fourth convolution layer 83, the feature fusion module 84, the fifth convolution layer 85, the upsampling module 86 and the image reconstruction module 87 are connected.
  • the process of realizing image super-resolution by the image super-resolution network model shown in Figure 8 includes:
  • the feature extraction module 81 performs feature extraction on the super-resolution image P in to be processed to obtain the first image feature F 1
  • the multi-level serial feature processing module 82 processes the first image feature F 1 to obtain the second image feature F 2 , then process the second image feature F 2 through the fourth convolution layer 83 to obtain the third image feature F 3 , and then fuse the first image feature F 1 and the third image feature F 3 through the feature fusion module 84 to obtain the fourth image.
  • Feature F 4 , and the fourth image feature F 4 is processed through the fifth convolution layer 85 to obtain the fifth image feature F 5
  • the fifth image feature F 5 is upsampled through the upsampling module 86 to obtain the sixth image feature.
  • F 6 and the image reconstruction module 87 performs image reconstruction according to the sixth image feature F 6 to obtain the super-resolution image P out corresponding to the image P in to be super-resolved.
  • the image super as shown in Figure 8 is achieved.
  • the image super-resolution network model of the sub-method can be implemented as shown in Figure 9.
  • an embodiment of the present disclosure also provides an image super-resolution device.
  • This device embodiment corresponds to the foregoing method embodiment.
  • this device embodiment no longer refers to the foregoing method.
  • the details in the embodiments will be described one by one, but it should be clear that the image super-resolution device in this embodiment can correspondingly implement all the contents in the foregoing method embodiments.
  • the super-resolution images constructed through image sample data (low-resolution images and high-resolution images) and contrast loss functions are usually first used.
  • the network is trained to obtain a super-resolution network with better results (reconstructed image features are as close as possible to high-resolution image features, and reconstructed image features are as far away from low-resolution image features as possible).
  • the network reconstructs the low-resolution image to obtain the reconstructed image, and then extracts the feature information of the high-resolution image, the feature information of the reconstructed image and the feature information of the low-resolution image respectively through the feature extractor, and then based on the feature information of the high-resolution image , the characteristic information of the reconstructed image, the characteristic information of the low-resolution image, and the contrast loss function value calculated by the contrast loss function adjust the parameters of the super-resolution network to achieve training of the super-resolution network.
  • the present disclosure also provides a super-resolution network parameter adjustment, device, system, etc., which will be described below.
  • Figure 10 shows a possible super-resolution network parameter adjustment system according to an embodiment of the present disclosure.
  • the super-resolution network parameter adjustment system includes a super-resolution network, an upsampling unit, a feature extractor, a contrast loss function value calculation unit, and a parameter adjustment unit.
  • the system can also include other units, such as other loss functions.
  • the specific unit can be determined according to the actual situation and is not limited here.
  • the super-resolution network is used to reconstruct low-resolution images to obtain a reconstructed image with a preset resolution.
  • the upsampling unit is used to upsample the low-resolution image to obtain an upsampled image with a preset resolution.
  • the device is used to respectively extract the feature information of the reconstructed image, the feature information of the high-resolution image (the resolution of the high-resolution image is the preset resolution) and the feature information of the upsampled image.
  • the contrast loss function value calculation unit is used to calculate the value of the reconstructed image based on the reconstructed image.
  • the characteristic information of the high-resolution image and the characteristic information of the upsampled image, as well as the contrast loss function calculate the contrast loss function value, and then the parameter adjustment unit is used to determine the parameter adjustment scheme for the super-resolution network based on the contrast loss function value. , and adjust the parameters of the super-resolution network to obtain a better super-resolution network.
  • the electronic device in the embodiment of the present disclosure may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA). ), personal computer (PC), etc.; the embodiments of this disclosure are not specifically limited.
  • UMPC ultra-mobile personal computer
  • PDA personal digital assistant
  • PC personal computer
  • the execution subject of the super-resolution network parameter adjustment method provided by the embodiments of the present disclosure may be the above-mentioned electronic device, or may be a functional module and/or functional entity in the electronic device that can implement the super-resolution network parameter adjustment method. Specifically, It is determined according to actual usage requirements and is not limited by the embodiments of the present disclosure.
  • an embodiment of the present disclosure provides a method for adjusting parameters of a super-resolution network, which may include the following steps S1101 to S1103.
  • step S1101 the feature information of the first reconstructed image, the feature information of the high-resolution image, and the feature information of the upsampled image are respectively extracted through a feature extractor.
  • the feature extractor includes at least one convolutional activation module consisting of a convolutional layer and an activation function.
  • the feature extractor may also include other operators that do not change the resolution.
  • the details may be determined according to the actual situation and are not limited here.
  • the feature extractor consists of at least one convolutional activation module, each convolutional activation module consisting of a convolutional layer and an activation function.
  • the feature extractor consists of at least one convolutional activation module in series.
  • each convolution activation module is composed of at least one convolution layer and at least one activation function.
  • the number of convolution layers included in different convolution activation modules may be the same or different.
  • Different convolution activation modules The number of activation functions included in the modules may be the same or different; the details can be determined according to the actual situation and are not limited here.
  • the feature extractor consists of two convolution activation modules connected in series.
  • a convolution activation module consists of a convolution layer and an activation module.
  • the part indicated by the dotted box in the figure is a convolution product activation module.
  • the feature extractor since the feature extractor includes at least one convolution activation module, there are no operators that change the resolution, such as conv_s2, max_pooling, avg_pooling, etc., in the feature extractor. Therefore, the feature extractor performs the first reconstruction During the feature extraction process of images, high-resolution images and upsampled images, the high resolution has been maintained without downsampling, so the feature information of the first reconstructed image, the feature information of the high-resolution image and the upsampled image are obtained There is no feature loss in the feature information of The contrast loss function value obtained by sampling the feature information of the image can correctly adjust the parameters of the super-resolution network, and then the second reconstructed image reconstructed through the adjusted super-resolution network will not produce more artifacts, so there will be no The second reconstructed image is distorted due to artifacts, and the subjective quality and objective index peak signal-to-noise ratio of the second reconstructed image can be improved.
  • the convolutional layers of each convolutional activation module in the at least one convolutional activation module may be the same. At the same time, the convolution layers of each convolution activation module may also be different. The details can be determined according to the actual situation and are not limited here.
  • the convolution layers of each convolution activation module are the same, which can make the structure of the feature extractor simple, facilitate feature extraction, and extract fine and accurate image edges and high-frequency features.
  • the convolutional layers of each convolutional activation module are the same, the convolutional kernel of the convolutional layer of each convolutional activation module is 3 ⁇ 3, and the number of channels is 64.
  • the convolution kernel of each convolution layer can also be other values, and the number of channels can also be other values.
  • the details can be determined according to the actual situation, and are not limited here.
  • the convolution kernel of the convolution layer of each convolution activation module is 3 ⁇ 3, and the number of channels is 64, which can make the structure of the feature extractor simple, facilitate feature extraction, and can extract fine Accurate image edges and high frequency features.
  • the first reconstructed image is a super-resolution image obtained by reconstructing a low-resolution image through a super-resolution network
  • the high-resolution image and the low-resolution image are training sample images of the super-resolution network
  • the The high-resolution image and the low-resolution image are images with the same visual content at different resolutions.
  • the upsampled image is an image obtained by upsampling the low-resolution image.
  • the first reconstructed image, the high-resolution image Have the same resolution as this upsampled image.
  • the first reconstructed image is an image of a preset resolution obtained by reconstructing a low-resolution image through a super-resolution network
  • the high-resolution image is a sample image of the preset resolution corresponding to the low-resolution image
  • the upsampled image is an image of the preset resolution obtained by upsampling the low-resolution image.
  • the resolutions of the first reconstructed image, the high-resolution image and the upsampled image are all preset resolutions, and the resolution of the low-resolution image is lower than the preset resolution.
  • the feature extractor provided in the embodiments of the present disclosure may be trained, or the feature extractor provided in the embodiments of the present disclosure may not be trained. If no training is performed, all weights in the feature extractor will be Just initialize randomly.
  • the feature extractor can extract fine and accurate image edges and high-frequency features. Based on the high-frequency feature extractor The contrast loss function can provide more accurate supervision information during the training process, correctly and effectively guide the training process of the super-resolution network, greatly reduce the artifacts of the reconstructed image and improve the quality of the reconstructed image; on the other hand, This feature extractor does not require training, which can save the training process of the feature extractor and improve the training efficiency of the super-resolution network.
  • the upsampling process is a bicubic interpolation process.
  • the upsampling process can also be nearest neighbor interpolation, bilinear interpolation, transposed convolution, etc.
  • the details can be determined according to the actual situation and are not limited here.
  • the upsampled image obtained based on the bicubic interpolation process improves the resolution of the image through the content of the low-resolution image without introducing more information.
  • the feature extractor is used to extract the features of the upsampled image.
  • the characteristic information is closer to the characteristic information of the low-resolution image, so that a more accurate contrast loss function value can be calculated, and then the parameters of the super-resolution network can be correctly adjusted based on the contrast loss function value, so that the adjusted super-resolution network can
  • the characteristics of the second reconstructed image obtained by reconstructing the low-resolution image are closer to the characteristics of the high-resolution image and further away from the characteristics of the low-resolution image.
  • step S1102 a contrast loss function is calculated based on the feature information of the first reconstructed image, the feature information of the high-resolution image, and the feature information of the upsampled image, and a first contrast loss function value based on the first reconstructed image is obtained.
  • the contrast loss function is a ratio of a first difference term and a second difference term.
  • the first difference term is used to characterize the difference between the first reconstructed image and the high-resolution image
  • the second difference term is used to characterize The difference between this upsampled image and this high-resolution image.
  • the contrast loss function is the ratio of the first difference term to the second difference term, which can make the training process of the super-resolution network more stable and achieve better image reconstruction effects.
  • the first difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term is the L1 distance between the feature information of the upsampled image and the high-resolution image. L1 distance of feature information.
  • the L1 distance is the Manhattan Distance, which is the sum of the absolute values of the differences between the pixel values of the corresponding pixels in the feature information of the two images.
  • the formula is:
  • d L1 is the L1 distance between the feature information of image 1 and the feature information of image 2
  • K is the number of pixels in the feature information of image 1 and the feature information of image 2.
  • anchor represents the first reconstructed image
  • ⁇ (anchor) represents the feature information of the first reconstructed image
  • positive represents the high-resolution image
  • ⁇ (positive) represents the feature information of the high-resolution image
  • negative represents the upsampled image
  • ⁇ ( negative) represents the feature information of the upsampled image
  • L1( ⁇ (anchor), ⁇ (positive)) represents the first difference term
  • L1( ⁇ (positive), ⁇ (negative)) represents the second difference term.
  • the first difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term is the L1 distance between the feature information of the upsampled image and the high-resolution image.
  • the L1 distance of feature information can make the training process of the super-resolution network more stable and achieve better image reconstruction results.
  • a high-frequency feature extractor and the above-mentioned contrast loss function are designed.
  • This feature extractor can extract fine and accurate high-frequency features.
  • the above-mentioned contrast loss function based on such high-frequency features can It provides more accurate supervision information during the training process, correctly and effectively guides the training process of the super-resolution network, greatly reduces artifacts in reconstructed images and improves the quality of reconstructed images.
  • the first difference term may be the L2 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term may be the L2 distance between the feature information of the upsampled image and the high-resolution image. L2 distance of the feature information of the image.
  • the L2 distance is the Euclidean distance, that is, the sum of the squares of the difference between the pixel values of the corresponding pixels in the feature information of the two images and then the square root of the difference, the formula is:
  • d L2 is the L2 distance between the feature information of image 1 and the feature information of image 2
  • K is the number of pixels in the feature information of image 1 and image 2.
  • anchor represents the first reconstructed image
  • ⁇ (anchor) represents the feature information of the first reconstructed image
  • positive represents the high-resolution image
  • ⁇ (positive) represents the feature information of the high-resolution image
  • negative represents the upsampled image
  • ⁇ ( negative) represents the feature information of the upsampled image
  • L2( ⁇ (anchor), ⁇ (positive)) represents the first difference term
  • L2( ⁇ (positive), ⁇ (negative)) represents the second difference term.
  • the first difference term is the L2 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term is the L2 distance between the feature information of the upsampled image and the high-resolution image.
  • the L2 distance of feature information can make the training process of the super-resolution network more stable and the effect of reconstructing the image is better.
  • the first difference term may also be other formulas that characterize the difference between the first reconstructed image and the high-resolution image. The details may be determined according to the actual situation and are not limited here.
  • the second difference term can also be other formulas that characterize the difference between the high-resolution image and the low-resolution image, which can be determined based on the actual situation.
  • the contrast loss function is a ratio of a third difference term to a fourth difference term.
  • the third difference term is used to characterize the difference between the first reconstructed image and the high-resolution image.
  • the fourth difference term is used to characterize The difference between the first reconstructed image and this upsampled image.
  • the contrast loss function is the ratio of the third difference term to the fourth difference term, which can make the training process of the super-resolution network more stable and achieve better image reconstruction effects.
  • the third difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image; the fourth difference term is the feature information of the first reconstructed image and the features of the upsampled image. L1 distance of information.
  • anchor represents the first reconstructed image
  • ⁇ (anchor) represents the feature information of the first reconstructed image
  • positive represents the high-resolution image
  • ⁇ (positive) represents the feature information of the high-resolution image
  • negative represents the upsampled image
  • ⁇ ( negative) represents the feature information of the upsampled image
  • L1( ⁇ (anchor), ⁇ (positive)) represents the third difference term
  • L1( ⁇ (anchor), ⁇ (negative)) represents the fourth difference term.
  • the third difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image; the fourth difference term is the feature information of the first reconstructed image and the feature of the upsampled image.
  • the L1 distance of the information can make the training process of the super-resolution network more stable and the effect of reconstructing the image is better.
  • the third difference term may be the characteristic information of the first reconstructed image and the high-resolution image.
  • the L2 distance of the feature information; the fourth difference term may be the L2 distance of the feature information of the first reconstructed image and the feature information of the upsampled image.
  • anchor represents the first reconstructed image
  • ⁇ (anchor) represents the feature information of the first reconstructed image
  • positive represents the high-resolution image
  • ⁇ (positive) represents the feature information of the high-resolution image
  • negative represents the upsampled image
  • ⁇ ( negative) represents the feature information of the upsampled image
  • L2( ⁇ (anchor), ⁇ (positive)) represents the third difference term
  • L2( ⁇ (anchor), ⁇ (negative)) represents the fourth difference term.
  • the third difference term is the L2 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image; the fourth difference term is the feature information of the first reconstructed image and the feature information of the upsampled image.
  • the L2 distance of the information can make the training process of the super-resolution network more stable and the effect of reconstructing the image is better.
  • the third difference term may also be other formulas that characterize the difference between the first reconstructed image and the high-resolution image. The details may be determined according to the actual situation and are not limited here.
  • the fourth difference term can also be other formulas characterizing the difference between the first reconstructed image and the low-resolution image, which can be determined according to the actual situation.
  • the contrast loss function can also be other formulas, which can be determined according to actual conditions and are not limited here.
  • step S1103 based on the first contrast loss function value, adjust the parameters of the super-resolution network so that the low-resolution image is reconstructed through the adjusted super-resolution network to obtain a second reconstructed image, and based on the second reconstructed image
  • the second contrast loss function value is smaller than the first contrast loss function value.
  • the process of obtaining the second contrast loss function value may refer to the process of obtaining the first contrast loss function value in the above steps S1101 to S1102, which will not be described again here.
  • the second contrast loss function value based on the second reconstructed image is smaller than the first contrast loss function value, so the difference between the second reconstructed image and the high-resolution image is smaller than the difference between the first reconstructed image and the high-resolution image.
  • the difference between the second reconstructed image and the low-resolution image is greater than the difference between the first reconstructed image and the low-resolution image. That is to say, compared to the first reconstructed image, the characteristics of the second reconstructed image are closer to the high-resolution image. The characteristics of the image are further away from the characteristics of the low-resolution image. This shows that after adjusting the super-resolution network based on the first contrast loss function value, the reconstructed image effect of the adjusted super-resolution network is better.
  • the above steps S1101 to S1102 are re-executed based on the second reconstructed image (after executing During the process, the contrast loss function value based on the second reconstructed image is obtained by replacing the first reconstructed image with the second reconstructed image. If the contrast loss function value based on the second reconstructed image is less than or equal to the preset threshold, then the super-resolution network training Complete, otherwise execute the above step S1103 again to obtain the new adjusted super-resolution network, and then repeatedly execute the above steps S1101 to step S1103 until the obtained contrast loss function value is less than or equal to the preset threshold, the super-resolution network training is completed, otherwise Keep training.
  • the technical solution provided by the embodiment of the present disclosure has the following advantages compared with the existing technology:
  • the feature information of the first reconstructed image, the feature information of the high-resolution image and the feature information of the upsampled image are respectively extracted through a feature extractor.
  • the feature extractor includes at least one convolution activation module, which is composed of a convolution layer and an activation function; based on the feature information of the first reconstructed image, the feature information of the high-resolution image and the upsampling Based on the characteristic information of the image, the contrast loss function is calculated to obtain the first contrast loss function value based on the first reconstructed image; based on the first contrast loss function value, the parameters of the super-resolution network are adjusted, and based on the adjusted super-resolution network, the low-resolution A second reconstructed image is obtained by reconstructing the rate image, and the second contrast loss function value based on the second reconstructed image is smaller than the first contrast loss function value.
  • convolution activation module which is composed of a convolution layer and an activation function
  • the feature extractor includes at least one convolution activation module and there is no operator that changes the resolution in the feature extractor, the feature extractor performs feature extraction on the first reconstructed image, the high-resolution image and the upsampled image.
  • the high resolution has been maintained during the process without downsampling, so there is no feature loss in the obtained feature information of the first reconstructed image, the feature information of the high-resolution image and the feature information of the upsampled image. In this way, fine features can be extracted.
  • the contrast loss function value obtained based on the feature information of the first reconstructed image, the feature information of the high-resolution image, and the feature information of the upsampled image extracted by the feature extractor can be accurately Adjust the parameters of the super-resolution network accurately, and then the second reconstructed image reconstructed through the adjusted super-resolution network will not produce more artifacts, so the second reconstructed image will not be distorted due to artifacts, and the second reconstructed image can be improved.
  • FIG. 12 is a schematic structural diagram of the image super-resolution device. As shown in Figure 12, the image super-resolution device 120 includes: a feature extraction module 121 and a feature processing module 122 , image reconstruction group 123.
  • the feature extraction module 121 is used to extract features of the super-resolution image to be processed and obtain first image features.
  • the feature processing module 122 is used to process the first image features through a feature processing module to obtain second image features, wherein the feature processing module includes: a first convolution layer and a third convolution layer connected in series.
  • a feature refinement unit composed of an activation function layer, used to process the input features of the feature processing module, and a feature fusion unit, used to process the input features of the feature processing module and the output features of the feature refinement unit conduct Additive fusion, attention unit, used to process the output features of the feature fusion unit and obtain the output features of the feature processing module;
  • the image reconstruction group 123 is configured to generate a super-resolution image corresponding to the image to be super-resolved according to the first image feature and the second image feature.
  • the feature processing module further includes: a channel number adjustment unit;
  • the channel number adjustment unit is configured to adjust the output features of the feature fusion unit to features with the same number of channels as the first image feature before the attention unit processes the output features of the feature fusion unit. .
  • the feature processing module 102 is specifically configured to process the first image feature through a multi-stage serially connected feature processing module to obtain the second image feature.
  • the activation function of the first activation function layer is a linear rectification function ReLU function.
  • the channel number adjustment unit includes: a second convolution layer
  • the size of the convolution kernel of the second convolution layer is 1*1.
  • the feature extraction module 101 is specifically configured to perform convolution processing on the image to be super-resolved through a third convolution layer to obtain the first image features.
  • the image reconstruction group 103 is specifically configured to process the second image feature through a fourth convolution layer to obtain a third image feature; compare the first image feature and the third image feature.
  • the three image features are additively fused to obtain the fourth image feature; the fourth image feature is processed through the fifth convolution layer to obtain the fifth image feature; the fifth image feature is upsampled to obtain the sixth image Features: Generate a super-resolution image corresponding to the image to be super-resolved according to the sixth image feature.
  • the image super-resolution device provided in this embodiment can execute the image super-resolution method provided in the above method embodiment. Its implementation principles and technical effects are similar and will not be described again here.
  • FIG. 14 is a structural block diagram of a super-resolution network parameter adjustment device according to an embodiment of the present disclosure. As shown in Figure 14, it includes: a feature extraction module 141, a calculation module 142 and an adjustment module 143; the feature extraction module 141 is used to The feature information of the first reconstructed image, the feature information of the high-resolution image and the feature information of the upsampled image are respectively extracted through a feature extractor.
  • the feature extractor includes at least one convolution activation module.
  • the convolution activation module is composed of a convolution layer. and an activation function.
  • the first reconstructed image is a super-resolution image obtained by reconstructing a low-resolution image through a super-resolution network.
  • the high-resolution image and the low-resolution image are the training sample images of the super-resolution network.
  • the high-resolution image is the training sample image of the super-resolution network.
  • the high-resolution image and the low-resolution image are images with the same visual content at different resolutions, and the upsampled image is obtained by upsampling the low-resolution image.
  • the calculation module 142 is configured to based on the feature information of the first reconstructed image extracted by the feature extraction module, the high-resolution image The characteristic information of the image and the characteristic information of the upsampled image are used to calculate the contrast loss function to obtain the first contrast loss function value based on the first reconstructed image; the adjustment module 143 is used to obtain the first contrast loss function based on the calculation module value, adjust the parameters of the super-resolution network so that the low-resolution image is reconstructed through the adjusted super-resolution network to obtain a second reconstructed image, and the second contrast loss function value based on the second reconstructed image is smaller than the first contrast loss function value.
  • the convolutional layers of each convolutional activation module in the at least one convolutional activation module are the same.
  • the at least one convolutional activation module is connected in series.
  • the upsampling process is a bicubic interpolation process.
  • the contrast loss function is a ratio of a first difference term and a second difference term.
  • the first difference term is used to characterize the difference between the first reconstructed image and the high-resolution image
  • the second difference term is used to characterize The difference between this upsampled image and this high-resolution image.
  • the first difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image
  • the second difference term is the L1 distance between the feature information of the upsampled image and the high-resolution image. L1 distance of feature information.
  • the contrast loss function is a ratio of a third difference term to a fourth difference term.
  • the third difference term is used to characterize the difference between the first reconstructed image and the high-resolution image.
  • the fourth difference term is used to characterize The difference between the first reconstructed image and this upsampled image.
  • the third difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the high-resolution image; the fourth difference term is the feature information of the first reconstructed image and the features of the upsampled image. L1 distance of information.
  • each module can implement the super-resolution network parameter adjustment method provided in the above method embodiment, and can achieve the same technical effect. To avoid duplication, it will not be described again here.
  • FIG. 15 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device provided by this embodiment includes: a memory 151 and a processor 152.
  • the memory 151 is used to store computer programs; the processing
  • the processor 152 is configured to execute the image super-resolution method and/or the super-resolution network parameter adjustment method provided in the above embodiments when calling a computer program.
  • Figure 16 is a schematic structural diagram of an electronic device provided by other embodiments of the present disclosure. It is used to illustrate the electronic device implementing any super-resolution network parameter adjustment method in the embodiments of the present disclosure. It should not be understood as Specific limitations of embodiments of the present disclosure.
  • the electronic device 160 may include a processor (eg, central processing unit, graphics processor, etc.) 161 , which may be loaded into a random access memory according to a program stored in a read-only memory (ROM) 162 or from a storage device 168 .
  • the program in the memory (RAM) 163 executes various appropriate actions and processes.
  • various programs and data required for the operation of the electronic device 160 are also stored.
  • the processor 161, ROM 162 and RAM 163 are connected to each other through a bus 164.
  • An input/output (I/O) interface 165 is also connected to bus 164 .
  • I/O interface 165 the following devices may be connected to the I/O interface 165: input devices 166 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 167 such as a computer; a storage device 168 including a magnetic tape, a hard disk, etc.; and a communication device 169. Communication device 169 may allow electronic device 160 to communicate wirelessly or wiredly with other devices to exchange data.
  • input devices 166 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.
  • LCD liquid crystal display
  • An output device 167 such as a computer
  • storage device 168 including a magnetic tape, a hard disk, etc.
  • communication device 169 may allow electronic device 160 to communicate wirelessly or wiredly with other devices to exchange data.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 169, or from storage device 168, or from ROM 162.
  • the processor 161 When the computer program is executed by the processor 161, the functions defined in any hyper-resolution network parameter adjustment method provided by the embodiments of the present disclosure can be executed.
  • Embodiments of the present disclosure also provide a computer-readable storage medium.
  • a computer program is stored on the computer-readable storage medium.
  • the computing device implements the image super-resolution provided by the above embodiments. methods and/or super-resolution network parameter adjustment methods.
  • Embodiments of the present disclosure also provide a computer program product, including: instructions, which when executed by a processor, cause the processor to implement the image super-resolution method and/or super-resolution network parameter adjustment provided in the above embodiments. method.
  • An embodiment of the present disclosure also provides a computer program, including: instructions.
  • the processor implements the image super-resolution method and/or the super-resolution network parameter adjustment method provided in the above embodiments. .
  • embodiments of the present disclosure may be provided as methods, systems, or computer program products. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present disclosure may employ computer-usable program code embodied therein in one or more In the form of a computer program product implemented on a computer-usable storage medium.
  • the processor can be a Central Processing Unit (CPU), other general-purpose processors, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), or off-the-shelf programmable processors. Gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
  • Memory may include non-volatile memory in computer-readable media, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash memory
  • Computer-readable media includes permanent and non-permanent, removable and non-removable storage media.
  • Storage media can be implemented by any method or technology to store information, and information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • read-only memory read-only memory
  • ROM read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other memory technology
  • compact disc read-only memory CD-ROM
  • DVD digital versatile disc
  • Magnetic tape cassettes disk storage or other magnetic storage devices, or any other non-transmission medium, can be used to store information that can be accessed by a computing device.
  • computer-readable media does not include transitory media, such as modulated data signals and carrier waves.
  • the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Data communications e.g., communications network
  • communications networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or developed in the future network of.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the present disclosure also provides the following methods.
  • An image super-resolution method including:
  • feature extraction is performed on the super-resolution image to be processed to obtain the first image features
  • the first image features are processed by a feature processing module to obtain second image features; wherein the feature processing module includes: features composed of at least one set of concatenated first convolutional layers and first activation function layers. thin
  • the feature fusion unit is used to add and fuse the input features of the feature processing module and the output features of the feature refinement unit, and the attention unit is , used to process the output features of the feature fusion unit and obtain the output features of the feature processing module;
  • a super-resolution image corresponding to the image to be super-resolved is generated.
  • the feature processing module further includes: a channel number adjustment unit, wherein the channel number adjustment unit is used to process the features in the attention unit. Before the output features of the fusion unit are processed, the output features of the feature fusion unit are adjusted to have the same number of feature channels as the first image feature.
  • the channel number adjustment unit includes: a second convolution layer, wherein the size of the convolution kernel of the second convolution layer is 1*1 .
  • processing the first image features through a feature processing module to obtain the second image features includes:
  • the first image features are processed through the multi-stage feature processing modules connected in series to obtain the second image features.
  • the image to be super-resolved is processed through a third convolution layer to obtain the first image feature.
  • Sub-images include:
  • a super-resolution image corresponding to the image to be super-resolved is generated according to the sixth image feature.
  • the feature information of the first reconstructed image, the feature information of the high-resolution image and the feature information of the upsampled image are respectively extracted through a feature extractor, wherein the feature extractor includes at least one convolution activation module, and the convolution The product activation module is composed of a convolution layer and an activation function.
  • the first reconstructed image is a super-resolution image obtained by reconstructing a low-resolution image through the super-resolution network.
  • the high-resolution image is different from the low-resolution image.
  • the image is a training sample image of the super-resolution network, the high-resolution image and the low-resolution image are images with the same visual content at different resolutions, and the upsampling image is obtained by performing the processing on the low-resolution image.
  • the image obtained by the upsampling process, the first reconstructed image, the high-resolution image and the upsampled image have the same resolution;
  • a contrast loss function is calculated to obtain a first contrast loss function value based on the first reconstructed image ;
  • the parameters of the super-resolution network are adjusted so that the low-resolution image is reconstructed through the adjusted super-resolution network to obtain a second reconstructed image, and based on the second The second contrast loss function value of the reconstructed image is smaller than the first contrast loss function value.
  • the second difference term is the L1 distance between the feature information of the upsampled image and the feature information of the high-resolution image.
  • the fourth difference term is the L1 distance between the feature information of the first reconstructed image and the feature information of the upsampled image.
  • the present disclosure also provides the following means.
  • An image super-resolution device including:
  • the feature extraction module is used to extract features of the super-resolution image to be processed in the super-resolution network and obtain the first image features;
  • a feature processing module configured to process the first image features through a feature processing module to obtain second image features
  • the feature processing module includes: a first convolution layer and a third convolution layer connected in series.
  • a feature refinement unit composed of an activation function layer, used to process the input features of the feature processing module, and a feature fusion unit, used to process the input features of the feature processing module and the output features of the feature refinement unit Perform additive fusion, and the attention unit is used to process the output features of the feature fusion unit and obtain the output features of the feature processing module;
  • An image reconstruction group configured to generate a super-resolution image corresponding to the image to be super-resolved according to the first image feature and the second image feature.
  • the feature extraction module is used to respectively extract the feature information of the first reconstructed image, the feature information of the high-resolution image and the feature information of the upsampled image through a feature extractor, where the feature extractor includes at least one convolution activation module,
  • the convolution activation module is composed of a convolution layer and an activation function.
  • the first reconstructed image is a super-resolution image obtained by reconstructing a low-resolution image through a super-resolution network.
  • the high-resolution image is different from the low-resolution image. is the training sample image of the super-resolution network, the high-resolution image and the low-resolution image are images with the same visual content at different resolutions, and the upsampling image is an upsampling of the low-resolution image.
  • the first reconstructed image, the high-resolution image and the upsampled image have the same resolution;
  • the calculation module is configured to calculate a contrast loss function based on the feature information of the first reconstructed image, the feature information of the high-resolution image, and the feature information of the upsampled image extracted by the feature extraction module, and obtain the result based on the first reconstructed image. - the first contrast loss function value of the reconstructed image;
  • the adjustment module is configured to adjust the parameters of the super-resolution network based on the first contrast loss function value obtained by the calculation module, so as to reconstruct the low-resolution image through the adjusted super-resolution network.
  • a second reconstructed image, and a second contrast loss function value based on the second reconstructed image is smaller than the first contrast loss function value.
  • An electronic device which includes: a memory and a processor, the memory is used to store a computer program; the processor is used to perform the image super-resolution described in any one of (1) to (15) when running the computer program. method.
  • a computer program product wherein a computer program is stored on the computer program product, and the computer program is used by a processor to execute the image super-resolution method described in any one of (1) to (15).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本公开实施例提供了一种图像超分方法、超分网络参数调整方法、相关装置及介质。该图像超分方法包括：对待超分图像进行特征提取，获取第一图像特征；通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征；所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理；特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合；注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。

Description

图像超分方法、超分网络参数调整方法、相关装置及介质
相关申请的交叉引用
本公开是以CN申请号为202210524546.1，申请日为2022年5月13日的申请，以及CN申请号为202210524706.2，申请日为2022年5月13日的申请为基础，并主张其优先权，该CN申请的公开内容在此作为整体引入本公开中。
技术领域
本公开涉及图像处理技术领域，尤其涉及一种图像超分方法、超分网络参数调整方法、相关装置及介质。
背景技术
图像超分技术是一种由低分辨率图像恢复出高分辨率图像的技术。由于图像超分辨率业务目前已成为图像画质增强中的重点业务，因此图像超分技术是当前图像处理领域的研究热点之一。
目前，基于卷积神经网络(Convolutional Neural Network，CNN)的图像超分模型已经在图像超分辨率领域取得了极大的进展。
发明内容
本公开第一方面，本公开的实施例提供了一种图像超分方法，包括：
对待超分图像进行特征提取，获取第一图像特征；
通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征，其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。
在一些实施例中，所述特征处理模块还包括：通道数调节单元，其中，所述通道数调节单元用于在所述注意力单元对所述特征融合单元的输出特征进行处理之前，将 所述特征融合单元的输出特征调节为与所述第一图像特征通道数相同的特征。
在一些实施例中，所述通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征包括：
通过多级串接的所述特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
在一些实施例中，所述第一激活函数层的激活函数为线性整流函数ReLU函数。
在一些实施例中，所述通道数调节单元，包括：第二卷积层，其中，所述第二卷积层的卷积核的尺寸为1*1。
在一些实施例中，所述对待超分图像进行特征提取，获取第一图像特征包括：
通过第三卷积层对所述待超分图像进行卷积处理，以获取所述第一图像特征。
在一些实施例中，所述根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像包括：
通过第四卷积层对所述第二图像特征进行处理，获取第三图像特征；
对所述第一图像特征和所述第三图像特征进行相加融合，获取第四图像特征；
通过第五卷积层对所述第四图像特征进行处理，获取第五图像特征；
对所述第五图像特征进行上采样获取第六图像特征；
根据所述第六图像特征生成所述待超分图像对应的超分图像。
第二方面，本公开实施例提供一种超分网络参数调整方法，该方法包括：通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，其中，该特征提取器包括至少一个卷积激活模块，该卷积激活模块由卷积层和激活函数组成，第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，该高分辨率图像与该低分辨率图像为该超分网络的训练样本图像，该高分辨率图像与该低分辨率图像为不同分辨率下相同视觉内容的图像，该上采样图像为对该低分辨率图像进行上采样处理得到的图像，第一重建图像、该高分辨率图像与该上采样图像具有相同的分辨率；基于第一重建图像的特征信息、该高分辨率图像的特征信息和该上采样图像的特征信息，计算对比损失函数，得到基于第一重建图像的第一对比损失函数值；基于第一对比损失函数值，调整该超分网络的参数，以便通过该调整后的超分网络对该低分辨率图像进行重建得到第二重建图像，且基于第二重建图像的第二对比损失函数值小于第一对比损失函数值。
在一些实施例中，该至少一个卷积激活模块中的每个卷积激活模块的卷积层相同。
在一些实施例中，该至少一个卷积激活模块串联连接。
在一些实施例中，该上采样处理为双三次插值处理。
在一些实施例中，该对比损失函数为第一差异项与第二差异项的比值，第一差异项用于表征第一重建图像与该高分辨率图像的差异，第二差异项用于表征该上采样图像与该高分辨率图像的差异。
在一些实施例中，第一差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第二差异项为该上采样图像的特征信息与该高分辨率图像的特征信息的L1距离。
在一些实施例中，该对比损失函数为第三差异项与第四差异项的比值，第三差异项用于表征第一重建图像与该高分辨率图像的差异，第四差异项用于表征第一重建图像与该上采样图像的差异。
在一些实施例中，第三差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第四差异项为第一重建图像的特征信息与该上采样图像的特征信息的L1距离。
第三方面，本公开实施例提供了一种图像超分装置，包括：
特征提取模组，用于对待超分图像进行特征提取，获取第一图像特征；
特征处理模组，用于通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征，其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
图像重建模组，用于根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。
在一些实施例中，所述特征处理模块还包括：通道数调节单元；
所述通道数调节单元用于在所述注意力单元对所述特征融合单元的输出特征进行处理之前，将所述特征融合单元的输出特征调节为与所述第一图像特征通道数相同的特征。
在一些实施例中，所述特征处理模组102，具体用于通过多级串接的所述特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
在一些实施例中，所述第一激活函数层的激活函数为线性整流函数ReLU函数。
在一些实施例中，所述通道数调节单元，包括：第二卷积层；
其中，所述第二卷积层的卷积核的尺寸为1*1。
在一些实施例中，所述特征提取模组，具体用于通过第三卷积层对所述待超分图像进行卷积处理，以获取所述第一图像特征。
在一些实施例中，所述图像重建模组，具体用于通过第四卷积层对所述第二图像特征进行处理，获取第三图像特征；对所述第一图像特征和所述第三图像特征进行相加融合，获取第四图像特征；通过第五卷积层对所述第四图像特征进行处理，获取第五图像特征；对所述第五图像特征进行上采样获取第六图像特征；根据所述第六图像特征生成所述待超分图像对应的超分图像。
第四方面，本公开实施例提供了一种超分网络参数调整装置，该装置包括：特征提取模块，计算模块和调整模块，其中，该特征提取模块，用于通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，该特征提取器包括至少一个卷积激活模块，该卷积激活模块由卷积层和激活函数组成，第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，该高分辨率图像与该低分辨率图像为该超分网络的训练样本图像，该高分辨率图像与该低分辨率图像为不同分辨率下相同视觉内容的图像，该上采样图像为对该低分辨率图像进行上采样处理得到的图像，第一重建图像、该高分辨率图像与该上采样图像具有相同的分辨率；该计算模块，用于基于该特征提取模块提取的第一重建图像的特征信息、该高分辨率图像的特征信息和该上采样图像的特征信息，计算对比损失函数，得到基于第一重建图像的第一对比损失函数值；该调整模块，用于基于该计算模块得到的第一对比损失函数值，调整该超分网络的参数，以便通过该调整后的超分网络对该低分辨率图像进行重建得到第二重建图像，且基于第二重建图像的第二对比损失函数值小于第一对比损失函数值。
在一些实施例中，该至少一个卷积激活模块中的每个卷积激活模块的卷积层相同。
在一些实施例中，该至少一个卷积激活模块串联连接。
在一些实施例中，该上采样处理为双三次插值处理。
在一些实施例中，该对比损失函数为第一差异项与第二差异项的比值，第一差异项用于表征第一重建图像与该高分辨率图像的差异，第二差异项用于表征该上采样图像与该高分辨率图像的差异。
在一些实施例中，第一差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第二差异项为该上采样图像的特征信息与该高分辨率图像的特征信息的L1距离。
在一些实施例中，该对比损失函数为第三差异项与第四差异项的比值，第三差异项用于表征第一重建图像与该高分辨率图像的差异，第四差异项用于表征第一重建图像与该上采样图像的差异。
在一些实施例中，第三差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第四差异项为第一重建图像的特征信息与该上采样图像的特征信息的L1距离。
第五方面，本公开实施例提供了一种电子设备，包括：存储器和处理器，所述存储器用于存储计算机程序；所述处理器用于在调用计算机程序时，使得所述电子设备实现前述任意实施例所述的图像超分方法和/或前述任意实施例所述的超分网络参数调整方法。
第六方面，本公开实施例提供一种计算机可读存储介质，当所述计算机程序被计算设备执行时，使得所述计算设备实现前述任意实施例所述的图像超分方法和/或前述任意实施例所述的超分网络参数调整方法
第七方面，本公开实施例提供一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机实现前述任意实施例所述的图像超分方法和/或前述任意实施例所述的超分网络参数调整方法
第八方面，本公开实施例提供一种芯片，该芯片包括处理器和通信接口，该通信接口和该处理器耦合，该处理器用于运行程序指令，实现如第一方面所述的超分网络参数调整方法。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开一些实施例提供的图像超分方法的流程图；
图2为本公开一些实施例提供的特征处理模块的结构示意图；
图3为本公开另一些实施例提供的特征处理模块的结构示意图；
图4为本公开又一些实施例提供的特征处理模块的结构示意图；
图5为本公开再一些实施例提供的特征处理模块的结构示意图；
图6为本公开又一些实施例提供的特征处理模块的结构示意图；
图7为本公开另一些实施例提供的图像超分方法的步骤流程图；
图8为本公开一些实施例提供的图像超分网络模型的结构示意图；
图9为本公开另一些实施例提供的图像超分网络模型的结构示意图；
图10为本公开一些实施例提供的图像超分装置的结构示意图；
图11为本公开一些实施例提供的电子设备的硬件结构示意图；
图12为本公开一些实施例提供的一种超分网络参数调整系统的结构示意图；
图13为本公开一些实施例提供的一种超分网络参数调整方法的流程示意图；
图14为本公开一些实施例提供的一种特征提取器的结构示意图；
图15为本公开一些实施例提供的一种超分网络参数调整装置的结构框图；
图16为本公开一些实施例提供的一种电子设备的结构框图。
具体实施方式
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。
需要说明的是，为了便于清楚描述本公开实施例的技术方案，在本公开的实施例中，采用了“第一”、“第二”等字样对功能和作用基本相同的相同项或相似项进行区分，本领域技术人员可以理解“第一”、“第二”等字样并不是在对数量和执行次序进行限定。例如：第一图像特征和第二图像特征仅仅是为了区分不同的特征，而不是在对特征的顺序等进行限定。
在本公开实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说 明。本公开实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。此外，在本公开实施例的描述中，除非另有说明，“多个”的含义是指两个或两个以上。
发明人发现：虽然图像超分模型取得了较好的性能与视觉效果，但是图像超分模型的运行速度普遍很慢，非常影响实际使用中的图像超分的效率。图像超分模型的运行速度慢的主要原因为：通过图像超分模型进行图像超分过程中，对低分辨率图像进行特征提取后，需要进一步对提取的图像特征进行处理，以获取用于重建低分辨率图像对应的超分图像的特征，而现有技术中的图像超分模型的模型结构非常不利于进行模型加速。因此如何提升图像超分模型的运行速度是一个亟待解决的问题。
本公开实施例提供了一种图像超分方法，参照图1所示，该图像超分方法包括如下步骤：S11～S13，可以通过超分网络(图像超分模型)生成超分图像。
在步骤S11中、对待超分图像进行特征提取，获取第一图像特征。
本公开实施例中的待超分图像是指最终想要得到的高分辨率图像所对应的低分辨率图像。待超分图像可以为任意分辨率、任意格式的图像。例如：待超分图像可以为分辨率为960*540的RGB图像。
本公开实施例中不限定对待超分图像进行特征提取的特征提取方式，以能够对待超分图像进行特征提取为准。
在一些实施例中，上步骤S11(对待超分图像进行特征提取，获取第一图像特征)包括：
通过第三卷积层对所述待超分图像进行处理，以获取所述第一图像特征。
示例性的，第三卷积层的卷积核的尺寸可以为3*3，第三卷积层的步长可以为2。
在步骤S12中、通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征。
例如，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理；特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合；注意力单元(Expanded Self Attention，ESA)，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征。
在一些实施例中，上述步骤S12(通过特征处理模块对所述第一图像特征进行处 理，获取第二图像特征)包括：
通过一级特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
示例性的，参照图2所示，当仅通过一级特征处理模块对所述第一图像特征进行处理时，实现上步骤S12的模型结构包括：特征处理模块200，所述特征处理模块200的输入特征为对待超分图像进行特征提取得到的第一图像特征F1，所述特征处理模块200的输出特征为第二图像特征F2。特征处理模块200由至少一组串接的第一卷积层211和第一激活函数层212组成的特征细化单元21(图2中以特征细化单元21包括三组串接的第一卷积层211和第一激活函数层212为例示出)，特征细化单元21用于对所述特征处理模块的输入特征(第一图像特征F1)进行处理；特征融合单元22，用于对所述特征处理模块的输入特征(第一图像特征F1)和所述特征细化单元的输出特征进行相加融合；注意力单元23，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征(第二图像特征F2)。
如图2所示，当仅通过一级特征处理模块对所述第一图像特征进行处理，且特征细化单元21包括三组串接的第一卷积层211和第一激活函数层212时，特征处理模块200的特征处理过程包括：首先通过第一组串接的第一卷积层211和第一激活函数层212对第一图像特征F1进行处理，并输出特征Fre1，然后通过第二组串接的第一卷积层211和第一激活函数层212对Fre1进行处理，并输出特征Fre2，再通过第三组串接的第一卷积层211和第一激活函数层212对Fre2进行处理，并输出特征Fre3(特征细化单元21的输出特征)；然后再通过特征融合单元22对第一图像特征F1和特征细化单元21的输出特征Fre3进行相加融合，获取特征融合单元22的输出特征Fre；最后通过注意力单元23对所述特征融合单元22的输出特征Fre进行处理，获取第一图像特征F2。
进一步的，第1组串接的第一卷积层211和第一激活函数层212对特征的处理表示为RM1，第2组串接的第一卷积层211和第一激活函数层212对特征的处理表示为RM2，第3组串接的第一卷积层211和第一激活函数层212对特征的处理表示为RM3，注意力单元23对特征的处理表示为ESA，则有如下公式：
Fre1＝RM1(Fin)            (1)
Fre2＝RM2(Fre1)             (2)
Fre3＝RM3(Fre2)               (3)
Fre＝F1+Fre3               (4)
F2＝ESA(Fre)             (5)
在一些实施例中，上述步骤S12(通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征)包括：
通过多级串接的特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
示例性的，多级串接的特征处理模块可以包括6级串接的特征处理模块、8级串接的特征处理模块等，本公开实施例对此不做限定。
示例性的，参照图3所示，当通过多级特征处理模块对所述第一图像特征进行处理时，实现上步骤S12的模型结构包括：多级串接的特征处理模块200。其中，第1级特征处理模块200的输入特征为第一图像特征F1，第n级特征处理模块200的输入特征Fin为第n-1级特征处理模块200的输出特征，第n级特征处理模块200的输出特征Fout为第n+1级特征处理模块200的输入特征，各个特征处理模块200对输入特征的处理方式与上述相同，在此不再赘述。
在一些实施例中，在上述图2或图3所示实施例的基础上，参照图4所示，所述特征处理模块200还包括：通道数调节单元24。
所述通道数调节单元24用于在所述注意力单元23对所述特征融合单元22的输出特征Fre进行处理之前，将所述特征融合单元22的输出特征Fre调节为与所述第一图像特征F1通道数相同的特征。
即，在通过注意力单元23进行特征处理之前，还通过通道数调节单元24将特征融合单元22的输出特征Fre的通道数调节为与所述第一图像特征F1的通道数相同的特征，然后再输入注意力单元23中。
示例性的，所述通道数调节单元24的输出特征的通道数与所述第一特征F1的通道数均可以为52。
在一些实施例中，所述通道数调节单元24，包括：第二卷积层，其中，所述第二卷积层的卷积核的尺寸为1*1。
由于上述实施例中通过卷积核尺寸为1*1的第二卷积层将特征融合单元22的输出特征Fre为所述第一图像特征F1通道数相同的特征，因此上述实施例可以在避免改变特征融合单元22的输出特征Fre的空间尺寸的情况下将所述特征融合单元的输出特征调节为与所述第一图像特征通道数相同的特征。
在一些实施例中，所述第一卷积层211的卷积核的尺寸为3*3。
在一些实施例中，所述第一激活函数层212的激活函数为线性整流函数(Linear rectification function，ReLU函数)。
进一步的，当所述第一卷积层211的卷积核的尺寸为3*3，第一激活函数层212的激活函数为ReLU，所述通道数调节单元23包括卷积核的尺寸为1*1第二卷积层，则实现上步骤S12的网络模型结构可以如图5所示。
在一些实施例中，参照图6所示，所述注意力单元23包括：
第六卷积层231、第七卷积层232、池化层233、卷积组234、上采样层235、第一融合层236、第八卷积层237、第二激活函数层238以及第二融合层239。
其中，所述第六卷积层231的卷积核的尺寸为1*1，用于对所述通道数调节单元的输出特征进行处理。
所述第七卷积层232的卷积核的尺寸为3*3，步长为预设值，用于对所述第六卷积层的输出特征进行处理。
所述池化层233用于对所述第七卷积层的输出特征进行处理。
所述卷积组234包括多个串接的卷积核尺寸为3*3的卷积层，用于对所述池化层的输出特征进行处理。
所述上采样层235用于对所述池化层的输出特征进行上采样。
所述第一融合层236用于对所述第六卷积层的输出特征和所述上采样层的输出特征进行相加融合。
所述第八卷积层237的卷积核的尺寸为1*1，用于对所述第一融合层的输出特征进行处理。
所述第二激活函数层238用于对所述第一融合层的输出特征进行处理。
所述第二融合层239用于对所述第二激活函数层的输出特征和所述通道数调节单元的输出特征进行相乘融合以获取所属特征处理模块的输出特征。
在步骤S13中、根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。
本公开实施例提供的图像超分方法首先对待超分图像进行特征提取获取第一图像特征，然后通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征，再根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理；特征融合单元，用 于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合；注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征。相比于现有技术中图像超分模型(即超分网络)中需要对并行进行图像特征的处理，由于本公开实施例提供的图像超分方法在通过特征处理模块对第一图像特征进行处理时，特征处理模块中的各个单元是以串行的方式进行特征处理的，因此本公开实施例可以提升图像超分模型的运行速度。
在一些实施例中，还可以将上述实施例中的特征处理模块替换为每一阶段并行的对特征进行蒸馏和细化，并多阶段渐进式对特征进行处理的模型结构，替换后的图像超分模型能够达到与替换前的图像超分模型能类似的图像超分效果，但由于替换后的图像超分模型在进行特征处理时，每一阶段都需要并行的对特征进行蒸馏和细化，因此替换后的图像超分模型的运行速度会远低于替换前的图像超分模型。
作为对上述实施例的扩展和细化，本公开实施例提供了另一种图像超分方法，参照图7所示，该图像超分方法包括如下步骤：S701～S707。
在步骤S701中、对待超分图像进行特征提取，获取第一图像特征。
在一些实施例中，上步骤S701(对待超分图像进行特征提取，获取第一图像特征)包括：
通过第三卷积层对所述待超分图像进行处理，获取所述第一图像特征。
在一些实施例中，所述第三卷积层的卷积核的尺寸为3*3。
即，通过一个卷积核为3*3的卷积层对待超分图像进行处理，以获取待超分图像的图像特征，并将待超分图像的图像特征作为本公开实施例中的第一图像特征。
在步骤S702中、通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征。
在一些实施例中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理；特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合；注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征。
在步骤S703中、通过第四卷积层对所述第二图像特征进行处理，获取第三图像特征。
在一些实施例中，所述第四卷积层的卷积核的尺寸为3*3。
在步骤S704中、对所述第一图像特征和所述第三图像特征进行相加融合，获取第四图像特征。
在步骤S705中、通过第五卷积层对所述第四图像特征进行处理，获取第五图像特征。
在一些实施例中，所述第八卷积层的卷积核的尺寸为3*3。
在步骤S706中、对所述第五图像特征进行上采样获取第六图像特征。
在步骤S707中、根据所述第六图像特征生成所述待超分图像对应的超分图像。
示例性的，参照图8所示，结合图2至图6所示，实现图8所示图像超分方法的图像超分网络模型(即超分网络)包括：特征提取模块81、多级串接的特征处理模块82、第四卷积层83、特征融合模块84、第五卷积层85、上采样模块86以及图像重建模块87。
图8所示图像超分网络模型实现图像超分的过程包括：
首先通过特征提取模块81对待超分图像Pin进行特征提取获取第一图像特征F1，其次通过多级串接的特征处理模块82对第一图像特征F1进行处理获取第二图像特征F2，然后通过第四卷积层83对第二图像特征F2进行处理获取第三图像特征F3，再通过特征融合模块84融合第一图像特征F1和第三图像特征F3获取第四图像特征F4，以及通过第五卷积层85对第四图像特征F4进行处理获取第五图像特征F5，最后通过上采样模块86对第五图像特征F5进行上采样获取第六图像特征F6，以及通过图像重建模块87根据第六图像特征F6进行图像重建获取待超分图像Pin对应的超分图像Pout。
进一步的，当所述第三卷积层、所述第四卷积层以及所述第五卷积层均为卷积核尺寸为3*3的卷积层时，实现图8所示图像超分方法的图像超分网络模型可以如实现图9所示。
基于同一发明构思，作为对上述方法的实现，本公开实施例还提供了一种图像超分装置，该装置实施例与前述方法实施例对应，为便于阅读，本装置实施例不再对前述方法实施例中的细节内容进行逐一赘述，但应当明确，本实施例中的图像超分装置能够对应实现前述方法实施例中的全部内容。
针对超分网络的训练，相关技术中，在通过超分网络实现图像超分辨率技术之前，通常先通过图像样本数据(低分辨率图像和高分辨率图像)以及对比损失函数对构建的超分网络进行训练，以得到效果较好(重建图像特征尽可能接近高分辨率图像特征，且重建图像特征尽可能远离低分辨率图像特征)的超分网络，具体地，先通过超分网 络对低分辨图像进行重建得到重建图像，然后通过特征提取器分别提取出高分辨率图像的特征信息，重建图像的特征信息以及低分辨率图像的特征信息，再基于高分辨率图像的特征信息，重建图像的特征信息，低分辨率图像的特征信息，以及对比损失函数计算得到的对比损失函数值对超分网络的参数进行调整，以实现对超分网络的训练。
发明人发现，当前在对超分网络进行训练的过程中，通常采用了较为深层的特征提取器，然而较为深层的特征提取器提取的深层特征失去了精细准确的图像边缘和高频特征，无法正确调整超分网络的参数，因此通过超分网络重建得到的重建图像通常会产生较多的伪影，这些伪影不仅会导致重建图像失真，而且对重建图像的主观质量和客观指标峰值信噪比(Peak Signal-to-Noise Ratio，PSNR)都有较大的影响。
有鉴于此，本公开还提供了一种超分网络参数调整、装置、系统等，下面进行描述。
首先，图10为本公开实施例示出的一种可能的超分网络参数调整系统。如图10所示，该超分网络参数调整系统包括超分网络、上采样单元、特征提取器、对比损失函数值计算单元以及参数调整单元，该系统还可以包括其他的单元，如其他损失函数单元，具体可以根据实际情况确定，此处不做限定。其中，超分网络用于对低分辨率图像进行重建，得到预设分辨率的重建图像，上采样单元用于对低分辨率图像进行上采样，得到预设分辨率的上采样图像，特征提取器用于分别提取重建图像的特征信息、高分辨率图像(高分辨率图像的分辨率为预设分辨率)的特征信息和上采样图像的特征信息，对比损失函数值计算单元用于基于重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，以及对比损失函数，计算对比损失函数值，然后参数调整单元用于基于对比损失函数值确定对超分网络的参数调整方案，并对超分网络的参数进行调整，以得到效果更好的超分网络。
本公开实施例中的电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)、个人计算机(personal computer，PC)等；本公开实施例不作具体限定。
本公开实施例提供的超分网络参数调整方法的执行主体可以为上述的电子设备，也可以为该电子设备中能够实现该超分网络参数调整方法的功能模块和/或功能实体，具体的可以根据实际使用需求确定，本公开实施例不作限定。
下面结合附图，通过具体的实施例及其应用场景对本公开实施例提供的超分网络 参数调整方法进行详细地说明。
如图11所示，本公开实施例提供一种超分网络参数调整方法，该方法可以包括下述的步骤S1101至步骤S1103。
在步骤S1101中，通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息。
在一些实施例中，该特征提取器包括至少一个卷积激活模块，该卷积激活模块由卷积层和激活函数组成。
在一些实施例中，该特征提取器还可以包括其他不会改变分辨率的算子，具体可以根据实际情况确定，此处不做限定。
在一些实施例中，该特征提取器由至少一个卷积激活模块组成，每个卷积激活模块由一个卷积层和一个激活函数组成。
在一些实施例中，该特征提取器由至少一个卷积激活模块串联组成。
在一些实施例中，每个卷积激活模块由至少一个卷积层和至少一个激活函数组成，不同的卷积激活模块包括的卷积层数量可以相同，也可以不相同，不同的卷积激活模块包括的激活函数的数量可以相同，也可以不相同；具体可以根据实际情况确定，此处不做限定。
示例性地，如图13所示，该特征提取器由2个卷积激活模块串联组成，一个卷积激活模块由一个卷积层和一个激活模块组成，图中虚线框指示的部分为一个卷积激活模块。
本公开实施例中，由于该特征提取器包括至少一个卷积激活模块，该特征提取器中不存在改变分辨率的算子，例如conv_s2，max_pooling，avg_pooling等，因此特征提取器在对第一重建图像、高分辨率图像和上采样图像进行特征提取的过程中一直保持了高分辨率，没有做下采样，所以得到的第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息均没有特征损失，如此，可以提取出精细准确的图像边缘和高频特征，进而使得基于该特征提取器提取到的第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息得到的对比损失函数值，可以正确地调整超分网络的参数，进而通过调整后的超分网络重建得到的第二重建图像不会产生较多的伪影，也就不会因伪影导致第二重建图像失真，而且可以提升第二重建图像的主观质量和客观指标峰值信噪比。
在一些实施例中，至少一个卷积激活模块中的每个卷积激活模块的卷积层可以相 同，该每个卷积激活模块的卷积层也可以不相同，具体可以根据实际情况确定，此处不做限定。
本公开实施例中，每个卷积激活模块的卷积层相同，可以使特征提取器的结构简单，便于进行特征提取，而且可以提取出精细准确的图像边缘和高频特征。
在一些实施例中，该每个卷积激活模块的卷积层相同，该每个卷积激活模块的卷积层的卷积核为3×3，通道数为64。
可以理解，每个卷积层的卷积核也可以为其他值，通道数也可以为其他值，具体可以根据实际情况确定，此处不做限定。
本公开实施例中，该每个卷积激活模块的卷积层的卷积核为3×3，通道数为64，可以使特征提取器的结构简单，便于进行特征提取，而且可以提取出精细准确的图像边缘和高频特征。
在一些实施例中，第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，该高分辨率图像与该低分辨率图像为该超分网络的训练样本图像，该高分辨率图像与该低分辨率图像为不同分辨率下相同视觉内容的图像，该上采样图像为对该低分辨率图像进行上采样处理得到的图像，第一重建图像、该高分辨率图像与该上采样图像具有相同的分辨率。
可以理解，第一重建图像为通过超分网络对低分辨率图像进行重建得到的预设分辨率的图像，该高分辨率图像为与该低分辨率图像对应的该预设分辨率的样本图像，该上采样图像为对该低分辨率图像进行上采样处理得到的该预设分辨率的图像。
可以理解，第一重建图像、高分辨率图像和上采样图像的分辨率均为预设分辨率，低分辨率图像的分辨率低于预设分辨率。
低分辨率图像和高分辨率图像的分辨率不同，但是展示的视觉内容(图像画面、图像内容)相同。
在一些实施例中，可以对本公开实施例中提供的特征提取器进行训练，也可以不对本公开实施例中提供的特征提取器进行训练，若不进行训练，特征提取器中的所有的权重采用随机初始化即可。
本公开实施例中，无需对特征提取器进行训练，所有的权重采用随机初始化即可，一方面该特征提取器能够提取出精细准确的图像边缘和高频特征，基于该高频特征提取器的对比损失函数能够在训练过程中提供更加精确的监督信息，正确有效的引导对超分网络的训练过程，极大的减少重建图像的伪影并提高重建图像质量；另一方面， 该特征提取器无需训练，可以节约对特征提取器训练过程，提高超分网络的训练效率。
在一些实施例中，该上采样处理为双三次插值处理。
在一些实施例中，上采样处理也可以为最近邻插值、双线性插值、转置卷积等，具体可以根据实际情况确定，此处不做限定。
本公开实施例中，基于双三次插值处理得到的上采样图像通过低分辨率图像的本身内容提高图像的分辨率，并没有引入更多信息，如此通过特征提取器对上采样图像进行特征提取得到的特征信息更接近低分辨率图像的特征信息，如此可以计算得到比较准确地对比损失函数值，进而可以基于对比损失函数值正确地调整超分网络的参数，使得通过调整后的超分网络对该低分辨率图像进行重建得到的第二重建图像的特征更接近高分辨率图像的特征，同时更远离低分辨率图像的特征。
在步骤S1102中，基于第一重建图像的特征信息、该高分辨率图像的特征信息和该上采样图像的特征信息，计算对比损失函数，得到基于第一重建图像的第一对比损失函数值。
在一些实施例中，该对比损失函数为第一差异项与第二差异项的比值，第一差异项用于表征第一重建图像与该高分辨率图像的差异，第二差异项用于表征该上采样图像与该高分辨率图像的差异。
可以理解，基于该对比损失函数计算得到的对比损失函数值越小，基于该超分网络得到的重建图像的效果越好，也就是重建图像的特征越接近高分辨率图像的特征，越远离低分辨率图像的特征。
本公开实施例中，该对比损失函数为第一差异项与第二差异项的比值，可以使对超分网络的训练过程更加稳定，重建图像的效果更好。
在一些实施例中，第一差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第二差异项为该上采样图像的特征信息与该高分辨率图像的特征信息的L1距离。
L1距离为曼哈顿距离(Manhattan Distance)，即两个图像的特征信息中的对应像素点的像素值的差值的绝对值之和，公式为：
其中，dL1为图像1的特征信息与图像2的特征信息的L1距离，为图像1的特征信息中的第P个像素点的像素值，为图像2的特征信息中的第P个像素点的像素 值，K为图像1的特征信息和图像2的特征信息中的像素点个数。
相应地，对比损失函数的公式为：
其中，anchor表示第一重建图像，φ(anchor)表示第一重建图像的特征信息，positive表示高分辨率图像，φ(positive)表示高分辨率图像的特征信息，negative表示上采样图像，φ(negative)表示上采样图像的特征信息；L1(φ(anchor),φ(positive))表示第一差异项，L1(φ(positive),φ(negative))表示第二差异项。
本公开实施例中，第一差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第二差异项为该上采样图像的特征信息与该高分辨率图像的特征信息的L1距离，可以使对超分网络的训练过程更加稳定，重建图像的效果更好。
本公开实施例中，设计了一种高频特征提取器，以及上述对比损失函数，这种特征提取器能够提取出精细准确的高频特征，基于这种高频特征的上述对比损失函数能够在训练过程中提供更加精确的监督信息，正确有效的引导对超分网络的训练过程，极大的减少重建图像的伪影并提高重建图像质量。
在一些实施例中，第一差异项可以为第一重建图像的特征信息与该高分辨率图像的特征信息的L2距离；第二差异项可以为该上采样图像的特征信息与该高分辨率图像的特征信息的L2距离。
其中，L2距离为欧氏距离，即两个图像的特征信息中的对应像素点的像素值的差值的平方和再开根号，公式为：
其中，dL2为图像1的特征信息与图像2的特征信息的L2距离，为图像1的特征信息中的第P个像素点的像素值，为图像2的特征信息中的第P个像素点的像素值，K为图像1的特征信息和图像2的特征信息中的像素点个数。
相应地，对比损失函数的公式为：
其中，anchor表示第一重建图像，φ(anchor)表示第一重建图像的特征信息，positive表示高分辨率图像，φ(positive)表示高分辨率图像的特征信息，negative表示上采样图像，φ(negative)表示上采样图像的特征信息；L2(φ(anchor),φ(positive))表示第一差异项， L2(φ(positive),φ(negative))表示第二差异项。
本公开实施例中，第一差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L2距离；第二差异项为该上采样图像的特征信息与该高分辨率图像的特征信息的L2距离，可以使对超分网络的训练过程更加稳定，重建图像的效果更好。
在一些实施例中，第一差异项还可以为其他表征第一重建图像与该高分辨率图像的差异的公式，具体可以根据实际情况确定，此处不做限定。第二差异项还可以为其他表征该高分辨率图像与该低分辨率图像的差异的公式，具体可以根据实际情况确定，
在一些实施例中，该对比损失函数为第三差异项与第四差异项的比值，第三差异项用于表征第一重建图像与该高分辨率图像的差异，第四差异项用于表征第一重建图像与该上采样图像的差异。
可以理解，基于该对比损失函数计算得到的对比损失函数值越小，基于该超分网络得到的重建图像的效果越好，也就是重建图像的特征越接近高分辨率图像的特征，越远离低分辨率图像的特征。
本公开实施例中，该对比损失函数为第三差异项与第四差异项的比值，可以使对超分网络的训练过程更加稳定，重建图像的效果更好。
在一些实施例中，第三差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第四差异项为第一重建图像的特征信息与该上采样图像的特征信息的L1距离。
其中，对L1距离的描述可以参考上述对L1距离的相关描述，此处不再赘述。
相应地，对比损失函数的公式为：
其中，anchor表示第一重建图像，φ(anchor)表示第一重建图像的特征信息，positive表示高分辨率图像，φ(positive)表示高分辨率图像的特征信息，negative表示上采样图像，φ(negative)表示上采样图像的特征信息；L1(φ(anchor),φ(positive))表示第三差异项，L1(φ(anchor),φ(negative))表示第四差异项。
本公开实施例中，第三差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第四差异项为第一重建图像的特征信息与该上采样图像的特征信息的L1距离，可以使对超分网络的训练过程更加稳定，重建图像的效果更好。
在一些实施例中，第三差异项可以为第一重建图像的特征信息与该高分辨率图像 的特征信息的L2距离；第四差异项可以为第一重建图像的特征信息与该上采样图像的特征信息的L2距离。
其中，对L2距离的描述可以参考上述对L2距离的相关描述，此处不再赘述。
相应地，对比损失函数的公式为：
其中，anchor表示第一重建图像，φ(anchor)表示第一重建图像的特征信息，positive表示高分辨率图像，φ(positive)表示高分辨率图像的特征信息，negative表示上采样图像，φ(negative)表示上采样图像的特征信息；L2(φ(anchor),φ(positive))表示第三差异项，L2(φ(anchor),φ(negative))表示第四差异项。
本公开实施例中，第三差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L2距离；第四差异项为第一重建图像的特征信息与该上采样图像的特征信息的L2距离，可以使对超分网络的训练过程更加稳定，重建图像的效果更好。
在一些实施例中，第三差异项还可以为其他表征第一重建图像与该高分辨率图像的差异的公式，具体可以根据实际情况确定，此处不做限定。第四差异项还可以为其他表征第一重建图像与该低分辨率图像的差异的公式，具体可以根据实际情况确定，
在一些实施例中，该对比损失函数还可以为其他地公式，具体可以根据实际情况确定，此处不做限定。
在步骤S1103中，基于第一对比损失函数值，调整该超分网络的参数，以便通过该调整后的超分网络对该低分辨率图像进行重建得到第二重建图像，且基于第二重建图像的第二对比损失函数值小于第一对比损失函数值。
其中，获得第二对比损失函数值的过程可以参考上述步骤S1101至步骤S1102中获得第一对比损失函数值的过程，此处不再赘述。
可以理解，基于第二重建图像的第二对比损失函数值小于第一对比损失函数值，从而第二重建图像与该高分辨率图像的差异，小于第一重建图像与该高分辨率图像的差异，且第二重建图像与低分辨率图像的差异，大于第一重建图像与低分辨率图像的差异，也就是说，相比于第一重建图像，第二重建图像的特征更接近高分辨率图像的特征，更远离低分辨率图像的特征。说明基于第一对比损失函数值对超分网络进行调整后，使得调整后的超分网络的重建图像效果更好。
可以理解，基于第二重建图像重新执行上述步骤S1101至步骤S1102(在执行过 程中，用第二重建图像替换第一重建图像)得到的基于第二重建图像的对比损失函数值，若基于第二重建图像的对比损失函数值小于或等于预设阈值，则超分网络训练完成，否则重新执行上述步骤S1103，得到新的调整后的超分网络，然后重复执行上述步骤S1101至步骤S1103，直至得到的对比损失函数值小于或等于预设阈值，超分网络训练完成，否则继续训练。
本公开实施例提供的技术方案与现有技术相比具有如下优点：本公开实施例中，通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，该特征提取器包括至少一个卷积激活模块，该卷积激活模块由卷积层和激活函数组成；基于第一重建图像的特征信息、该高分辨率图像的特征信息和该上采样图像的特征信息，计算对比损失函数，得到基于第一重建图像的第一对比损失函数值；基于第一对比损失函数值，调整该超分网络的参数，基于调整后的超分网络对低分辨率图像进行重建得到第二重建图像，且基于第二重建图像的第二对比损失函数值小于第一对比损失函数值。由于该特征提取器包括至少一个卷积激活模块，该特征提取器中不存在改变分辨率的算子，因此特征提取器在对第一重建图像、高分辨率图像和上采样图像进行特征提取的过程中一直保持了高分辨率，没有做下采样，所以得到的第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息均没有特征损失，如此，可以提取出精细准确的图像边缘和高频特征，进而使得基于该特征提取器提取到的第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息得到的对比损失函数值，可以正确地调整超分网络的参数，进而通过调整后的超分网络重建得到的第二重建图像不会产生较多的伪影，也就不会因伪影导致第二重建图像失真，而且可以提升第二重建图像的主观质量和客观指标峰值信噪比。
本公开实施例提供了一种图像超分装置，图12为该图像超分装置的结构示意图，如图12所示，该图像超分装置120包括：特征提取模组121，特征处理模组122，图像重建模组123。
特征提取模组121用于对待超分图像进行特征提取，获取第一图像特征。
特征处理模组122用于通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征，其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行 相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
图像重建模组123用于根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。
在一些实施例中，所述特征处理模块还包括：通道数调节单元；
所述通道数调节单元用于在所述注意力单元对所述特征融合单元的输出特征进行处理之前，将所述特征融合单元的输出特征调节为与所述第一图像特征通道数相同的特征。
在一些实施例中，所述特征处理模组102，具体用于通过多级串接的所述特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
在一些实施例中，所述第一激活函数层的激活函数为线性整流函数ReLU函数。
在一些实施例中，所述通道数调节单元，包括：第二卷积层；
其中，所述第二卷积层的卷积核的尺寸为1*1。
在一些实施例中，所述特征提取模组101，具体用于通过第三卷积层对所述待超分图像进行卷积处理，以获取所述第一图像特征。
在一些实施例中，所述图像重建模组103，具体用于通过第四卷积层对所述第二图像特征进行处理，获取第三图像特征；对所述第一图像特征和所述第三图像特征进行相加融合，获取第四图像特征；通过第五卷积层对所述第四图像特征进行处理，获取第五图像特征；对所述第五图像特征进行上采样获取第六图像特征；根据所述第六图像特征生成所述待超分图像对应的超分图像。
本实施例提供的图像超分装置可以执行上述方法实施例提供的图像超分方法，其实现原理与技术效果类似，此处不再赘述。
本公开还提供一种超分网络参数调整装置。图14为本公开实施例示出的一种超分网络参数调整装置的结构框图，如图14所示，包括：特征提取模块141，计算模块142和调整模块143；该特征提取模块141，用于通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，该特征提取器包括至少一个卷积激活模块，该卷积激活模块由卷积层和激活函数组成，第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，该高分辨率图像与该低分辨率图像为该超分网络的训练样本图像，该高分辨率图像与该低分辨率图像为不同分辨率下相同视觉内容的图像，该上采样图像为对该低分辨率图像进行上采样处理得到 的图像，第一重建图像、该高分辨率图像与该上采样图像具有相同的分辨率；该计算模块142，用于基于该特征提取模块提取的第一重建图像的特征信息、该高分辨率图像的特征信息和该上采样图像的特征信息，计算对比损失函数，得到基于第一重建图像的第一对比损失函数值；该调整模块143，用于基于该计算模块得到的第一对比损失函数值，调整该超分网络的参数，以便通过该调整后的超分网络对该低分辨率图像进行重建得到第二重建图像，且基于第二重建图像的第二对比损失函数值小于第一对比损失函数值。
在一些实施例中，该至少一个卷积激活模块中的每个卷积激活模块的卷积层相同。
在一些实施例中，该至少一个卷积激活模块串联连接。
在一些实施例中，该上采样处理为双三次插值处理。
在一些实施例中，该对比损失函数为第一差异项与第二差异项的比值，第一差异项用于表征第一重建图像与该高分辨率图像的差异，第二差异项用于表征该上采样图像与该高分辨率图像的差异。
在一些实施例中，第一差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第二差异项为该上采样图像的特征信息与该高分辨率图像的特征信息的L1距离。
在一些实施例中，该对比损失函数为第三差异项与第四差异项的比值，第三差异项用于表征第一重建图像与该高分辨率图像的差异，第四差异项用于表征第一重建图像与该上采样图像的差异。
在一些实施例中，第三差异项为第一重建图像的特征信息与该高分辨率图像的特征信息的L1距离；第四差异项为第一重建图像的特征信息与该上采样图像的特征信息的L1距离。
本公开实施例中，各模块可以实现上述方法实施例提供的超分网络参数调整方法，且能达到相同的技术效果，为避免重复，这里不再赘述。
本公开实施例还提供了一种电子设备。图15为本公开实施例提供的电子设备的结构示意图，如图15所示，本实施例提供的电子设备包括：存储器151和处理器152，所述存储器151用于存储计算机程序；所述处理器152用于在调用计算机程序时执行上述实施例提供的图像超分方法和/或超分网络参数调整方法。
图16为本公开另一些实施例提供的一种电子设备的结构示意图，用于对实现本公开实施例中任意超分网络参数调整方法的电子设备进行示例性说明，不应理解为对 本公开实施例的具体限定。
如图16所示，电子设备160可以包括处理器(例如中央处理器、图形处理器等)161，其可以根据存储在只读存储器(ROM)162中的程序或者从存储装置168加载到随机访问存储器(RAM)163中的程序而执行各种适当的动作和处理。在RAM 163中，还存储有电子设备160操作所需的各种程序和数据。处理器161、ROM 162以及RAM 163通过总线164彼此相连。输入/输出(I/O)接口165也连接至总线164。
通常，以下装置可以连接至I/O接口165：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置166；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置167；包括例如磁带、硬盘等的存储装置168；以及通信装置169。通信装置169可以允许电子设备160与其他设备进行无线或有线通信以交换数据。虽然示出了具有各种装置的电子设备160，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置169从网络上被下载和安装，或者从存储装置168被安装，或者从ROM 162被安装。在该计算机程序被处理器161执行时，可以执行本公开实施例提供的任意超分网络参数调整方法中限定的功能。
本公开实施例还提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，当计算机程序被处理器执行时，使得所述计算设备实现上述实施例提供的图像超分方法和/或超分网络参数调整方法。
本公开实施例还提供了一种计算机程序产品，包括：指令，当所述指令被处理器执行时，使得所述处理器实现上述实施例提供的图像超分方法和/或超分网络参数调整方法。
本公开实施例还提供了一种计算机程序，包括：指令，当所述指令被处理器执行时，使得所述处理器实现上述实施例提供的图像超分方法和/或超分网络参数调整方法。
本领域技术人员应明白，本公开的实施例可提供为方法、系统、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码 的计算机可用存储介质上实施的计算机程序产品的形式。
处理器可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
存储器可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。存储器是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动存储介质。存储介质可以由任何方法或技术来实现信息存储，信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。根据本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
在一些实施方式中，客户端、服务端可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
本公开还提供以下方法。
(1)一种图像超分方法，包括：
在超分网络中，对待超分图像进行特征提取，获取第一图像特征；
通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征；其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细 化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
根据所述第一图像特征和所述第二图像特征，生成所述待超分图像对应的超分图像。
(2)根据(1)所述的图像超分方法，其中，所述特征处理模块还包括：通道数调节单元，其中，所述通道数调节单元用于在所述注意力单元对所述特征融合单元的输出特征进行处理之前，将所述特征融合单元的输出特征调节为与所述第一图像特征通道数相同的特征。
(3)根据(2)所述的图像超分方法，其中，所述通道数调节单元包括：第二卷积层，其中，所述第二卷积层的卷积核的尺寸为1*1。
(4)根据(1)-(3)任一项所述的图像超分方法，其中，所述通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征包括：
通过多级串接的所述特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
(5)根据(1)-(4)任一项所述的图像超分方法，其中，所述第一激活函数层的激活函数为线性整流函数ReLU函数。
(6)根据(1)-(5)任一项所述的图像超分方法，其中，所述对待超分图像进行特征提取，获取第一图像特征包括：
通过第三卷积层对所述待超分图像进行处理，以获取所述第一图像特征。
(7)根据(1)-(6)任一项所述的图像超分方法，其中，所述根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像包括：
通过第四卷积层对所述第二图像特征进行处理，获取第三图像特征；
对所述第一图像特征和所述第三图像特征进行相加融合，获取第四图像特征；
通过第五卷积层对所述第四图像特征进行处理，获取第五图像特征；
对所述第五图像特征进行上采样获取第六图像特征；
根据所述第六图像特征生成所述待超分图像对应的超分图像。
(8)根据(1)-(7)任一项所述的图像超分方法，还包括：
通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，其中，所述特征提取器包括至少一个卷积激活模块，所述卷 积激活模块由卷积层和激活函数组成，所述第一重建图像为通过所述超分网络对低分辨率图像进行重建得到的超分图像，所述高分辨率图像与所述低分辨率图像为所述超分网络的训练样本图像，所述高分辨率图像与所述低分辨率图像为不同分辨率下相同视觉内容的图像，所述上采样图像为对所述低分辨率图像进行上采样处理得到的图像，所述第一重建图像、所述高分辨率图像与所述上采样图像具有相同的分辨率；
基于所述第一重建图像的特征信息、所述高分辨率图像的特征信息和所述上采样图像的特征信息，计算对比损失函数，得到基于所述第一重建图像的第一对比损失函数值；
基于所述第一对比损失函数值，调整所述超分网络的参数，以便通过所述调整后的超分网络对所述低分辨率图像进行重建得到第二重建图像，且基于所述第二重建图像的第二对比损失函数值小于所述第一对比损失函数值。
(9)根据(8)所述的图像超分方法，其中，所述至少一个卷积激活模块中的每个卷积激活模块的卷积层相同。
(10)根据(9)所述的图像超分方法，其中，所述至少一个卷积激活模块串联连接。
(11)根据(8)-(10)任一项所述的图像超分方法，其中，所述上采样处理为双三次插值处理。
(12)根据(8)-(11)中任一项所述的图像超分方法，其中，所述对比损失函数为第一差异项与第二差异项的比值，所述第一差异项用于表征所述第一重建图像与所述高分辨率图像的差异，所述第二差异项用于表征所述上采样图像与所述高分辨率图像的差异。
(13)根据(12)所述的图像超分方法，其中，所述第一差异项为所述第一重建图像的特征信息与所述高分辨率图像的特征信息的L1距离；
所述第二差异项为所述上采样图像的特征信息与所述高分辨率图像的特征信息的L1距离。
(14)根据(8)至(13)中任一项所述的图像超分方法，其中，所述对比损失函数为第三差异项与第四差异项的比值，所述第三差异项用于表征所述第一重建图像与所述高分辨率图像的差异，所述第四差异项用于表征所述第一重建图像与所述上采样图像的差异。
(15)根据(14)所述的图像超分方法，其中，所述第三差异项为所述第一重建 图像的特征信息与所述高分辨率图像的特征信息的L1距离；
所述第四差异项为所述第一重建图像的特征信息与所述上采样图像的特征信息的L1距离。
本公开还提供以下装置。
(16)一种图像超分装置，包括：
特征提取模组，用于在超分网络中对待超分图像进行特征提取，获取第一图像特征；
特征处理模组，用于通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征，其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
图像重建模组，用于根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。
(17)根据(16)所述的图像超分装置，还包括：特征提取模块，计算模块和调整模块，其中：
所述特征提取模块，用于通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，所述特征提取器包括至少一个卷积激活模块，所述卷积激活模块由卷积层和激活函数组成，第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，所述高分辨率图像与所述低分辨率图像为所述超分网络的训练样本图像，所述高分辨率图像与所述低分辨率图像为不同分辨率下相同视觉内容的图像，所述上采样图像为对所述低分辨率图像进行上采样处理得到的图像，第一重建图像、所述高分辨率图像与所述上采样图像具有相同的分辨率；
所述计算模块，用于基于所述特征提取模块提取的第一重建图像的特征信息、所述高分辨率图像的特征信息和所述上采样图像的特征信息，计算对比损失函数，得到基于第一重建图像的第一对比损失函数值；
所述调整模块，用于基于所述计算模块得到的第一对比损失函数值，调整所述超分网络的参数，以便通过所述调整后的超分网络对所述低分辨率图像进行重建得到第二重建图像，且基于第二重建图像的第二对比损失函数值小于第一对比损失函数值。
(18)一种电子设备，其中，包括：存储器和处理器，存储器用于存储计算机程序；处理器用于在运行计算机程序时执行(1)至(15)中任一项所述的图像超分方法。
(19)一种计算机可读存储介质，其中，其上存储有计算机程序，计算机程序被处理器执行(1)至(15)中任一项所述的图像超分方法。
(20)一种计算机程序产品，其中，所述计算机程序产品上存储有计算机程序，所述计算机程序被处理器执行(1)至(15)中任一项所述的图像超分方法。
(21)一种计算机程序，包括：指令，所述指令被处理器执行(1)至(15)中任一项所述的图像超分方法。最后应说明的是：以上各实施例仅用以说明本公开的技术方案，而非对其限制；尽管参照前述各实施例对本公开进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本公开各实施例技术方案的范围。

Claims (20)

  1. 一种图像超分方法，包括：
    对待超分图像进行特征提取，获取第一图像特征；
    通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征；其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
    根据所述第一图像特征和所述第二图像特征，生成所述待超分图像对应的超分图像。
  2. 根据权利要求1所述的图像超分方法，其中，所述特征处理模块还包括：通道数调节单元，其中，所述通道数调节单元用于在所述注意力单元对所述特征融合单元的输出特征进行处理之前，将所述特征融合单元的输出特征调节为与所述第一图像特征通道数相同的特征。
  3. 根据权利要求2所述的图像超分方法，其中，所述通道数调节单元包括：第二卷积层，其中，所述第二卷积层的卷积核的尺寸为1*1。
  4. 根据权利要求1-3任一项所述的图像超分方法，其中，所述通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征包括：
    通过多级串接的所述特征处理模块对所述第一图像特征进行处理，获取所述第二图像特征。
  5. 根据权利要求1-4任一项所述的图像超分方法，其中，所述第一激活函数层的激活函数为线性整流函数ReLU函数。
  6. 根据权利要求1-5任一项所述的图像超分方法，其中，所述对待超分图像进行特征提取，获取第一图像特征包括：
    通过第三卷积层对所述待超分图像进行处理，以获取所述第一图像特征。
  7. 根据权利要求1-6任一项所述的图像超分方法，其中，所述根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像包括：
    通过第四卷积层对所述第二图像特征进行处理，获取第三图像特征；
    对所述第一图像特征和所述第三图像特征进行相加融合，获取第四图像特征；
    通过第五卷积层对所述第四图像特征进行处理，获取第五图像特征；
    对所述第五图像特征进行上采样获取第六图像特征；
    根据所述第六图像特征生成所述待超分图像对应的超分图像。
  8. 一种超分网络参数调整方法，包括：
    通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，其中，所述特征提取器包括至少一个卷积激活模块，所述卷积激活模块由卷积层和激活函数组成，所述第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，所述高分辨率图像与所述低分辨率图像为所述超分网络的训练样本图像，所述高分辨率图像与所述低分辨率图像为不同分辨率下相同视觉内容的图像，所述上采样图像为对所述低分辨率图像进行上采样处理得到的图像，所述第一重建图像、所述高分辨率图像与所述上采样图像具有相同的分辨率；
    基于所述第一重建图像的特征信息、所述高分辨率图像的特征信息和所述上采样图像的特征信息，计算对比损失函数，得到基于所述第一重建图像的第一对比损失函数值；
    基于所述第一对比损失函数值，调整所述超分网络的参数，以便通过所述调整后的超分网络对所述低分辨率图像进行重建得到第二重建图像，且基于所述第二重建图像的第二对比损失函数值小于所述第一对比损失函数值。
  9. 根据权利要求8所述的超分网络参数调整方法，其中，所述至少一个卷积激活模块中的每个卷积激活模块的卷积层相同。
  10. 根据权利要求9所述的超分网络参数调整方法，其中，所述至少一个卷积激活模块串联连接。
  11. 根据权利要求8-10任一项所述的超分网络参数调整方法，其中，所述上采样处理为双三次插值处理。
  12. 根据权利要求8至11中任一项所述的超分网络参数调整方法，其中，所述对比损失函数为第一差异项与第二差异项的比值，所述第一差异项用于表征所述第一重建图像与所述高分辨率图像的差异，所述第二差异项用于表征所述上采样图像与所述高分辨率图像的差异。
  13. 根据权利要求12所述的超分网络参数调整方法，其中，所述第一差异项为所述第一重建图像的特征信息与所述高分辨率图像的特征信息的L1距离；
    所述第二差异项为所述上采样图像的特征信息与所述高分辨率图像的特征信息 的L1距离。
  14. 根据权利要求8至13中任一项所述的超分网络参数调整方法，其中，所述对比损失函数为第三差异项与第四差异项的比值，所述第三差异项用于表征所述第一重建图像与所述高分辨率图像的差异，所述第四差异项用于表征所述第一重建图像与所述上采样图像的差异。
  15. 根据权利要求14所述的超分网络参数调整方法，其中，所述第三差异项为所述第一重建图像的特征信息与所述高分辨率图像的特征信息的L1距离；
    所述第四差异项为所述第一重建图像的特征信息与所述上采样图像的特征信息的L1距离。
  16. 一种图像超分装置，包括：
    特征提取模组，用于对待超分图像进行特征提取，获取第一图像特征；
    特征处理模组，用于通过特征处理模块对所述第一图像特征进行处理，获取第二图像特征，其中，所述特征处理模块包括：由至少一组串接的第一卷积层和第一激活函数层组成的特征细化单元，用于对所述特征处理模块的输入特征进行处理，特征融合单元，用于对所述特征处理模块的输入特征和所述特征细化单元的输出特征进行相加融合，注意力单元，用于对所述特征融合单元的输出特征进行处理，获取所述特征处理模块的输出特征；
    图像重建模组，用于根据所述第一图像特征和所述第二图像特征生成所述待超分图像对应的超分图像。
  17. 一种超分网络参数调整装置，其中，包括：特征提取模块，计算模块和调整模块；
    所述特征提取模块，用于通过特征提取器分别提取第一重建图像的特征信息、高分辨率图像的特征信息和上采样图像的特征信息，所述特征提取器包括至少一个卷积激活模块，所述卷积激活模块由卷积层和激活函数组成，所述第一重建图像为通过超分网络对低分辨率图像进行重建得到的超分图像，所述高分辨率图像与所述低分辨率图像为所述超分网络的训练样本图像，所述高分辨率图像与所述低分辨率图像为不同分辨率下相同视觉内容的图像，所述上采样图像为对所述低分辨率图像进行上采样处理得到的图像，所述第一重建图像、所述高分辨率图像与所述上采样图像具有相同的分辨率；
    所述计算模块，用于基于所述特征提取模块提取的所述第一重建图像的特征信息、 所述高分辨率图像的特征信息和所述上采样图像的特征信息，计算对比损失函数，得到基于所述第一重建图像的第一对比损失函数值；
    所述调整模块，用于基于所述计算模块得到的所述第一对比损失函数值，调整所述超分网络的参数，以便通过所述调整后的超分网络对所述低分辨率图像进行重建得到第二重建图像，且基于所述第二重建图像的第二对比损失函数值小于所述第一对比损失函数值。
  18. 一种电子设备，其中，包括：存储器和处理器，存储器用于存储计算机程序；处理器用于在调用计算机程序时执行权利要求1至7中任一项所述的图像超分方法和/或权利要求8至15任一项所述的超分网络参数调整方法。
  19. 一种计算机可读存储介质，其中，其上存储有计算机程序，计算机程序被处理器执行时实现权利要求1至7中任一项所述的图像超分方法和/或权利要求8至15任一项所述的超分网络参数调整方法。
  20. 一种计算机程序产品，其中，所述计算机程序产品上存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1至7中任一项所述的图像超分方法和/或权利要求8至15任一项所述的超分网络参数调整方法。
PCT/CN2023/093880 2022-05-13 2023-05-12 图像超分方法、超分网络参数调整方法、相关装置及介质 Ceased WO2023217270A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/858,296 US20250272786A1 (en) 2022-05-13 2023-05-12 Image super-resolution method, super-resolution network parameter adjustment method, related apparatus, and medium

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210524546.1 2022-05-13
CN202210524706.2A CN117114981A (zh) 2022-05-13 2022-05-13 超分网络参数调整方法、装置、设备、介质及程序产品
CN202210524546.1A CN117114980A (zh) 2022-05-13 2022-05-13 一种图像超分方法及装置
CN202210524706.2 2022-05-13

Publications (1)

Publication Number Publication Date
WO2023217270A1 true WO2023217270A1 (zh) 2023-11-16

Family

ID=88729763

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/093880 Ceased WO2023217270A1 (zh) 2022-05-13 2023-05-12 图像超分方法、超分网络参数调整方法、相关装置及介质

Country Status (2)

Country Link
US (1) US20250272786A1 (zh)
WO (1) WO2023217270A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111047515A (zh) * 2019-12-29 2020-04-21 兰州理工大学 一种基于注意力机制的空洞卷积神经网络图像超分辨率重建方法
WO2021073493A1 (zh) * 2019-10-18 2021-04-22 京东方科技集团股份有限公司 图像处理方法及装置、神经网络的训练方法、合并神经网络模型的图像处理方法、合并神经网络模型的构建方法、神经网络处理器及存储介质
CN113344794A (zh) * 2021-08-04 2021-09-03 腾讯科技（深圳）有限公司 一种图像处理方法、装置、计算机设备及存储介质
CN114022362A (zh) * 2021-11-17 2022-02-08 南京邮电大学 一种基于金字塔注意力机制和对称网络的图像超分辨率方法
CN114463183A (zh) * 2022-02-14 2022-05-10 中南大学 基于频域和空域的图像超分辨率方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021073493A1 (zh) * 2019-10-18 2021-04-22 京东方科技集团股份有限公司 图像处理方法及装置、神经网络的训练方法、合并神经网络模型的图像处理方法、合并神经网络模型的构建方法、神经网络处理器及存储介质
CN111047515A (zh) * 2019-12-29 2020-04-21 兰州理工大学 一种基于注意力机制的空洞卷积神经网络图像超分辨率重建方法
CN113344794A (zh) * 2021-08-04 2021-09-03 腾讯科技（深圳）有限公司 一种图像处理方法、装置、计算机设备及存储介质
CN114022362A (zh) * 2021-11-17 2022-02-08 南京邮电大学 一种基于金字塔注意力机制和对称网络的图像超分辨率方法
CN114463183A (zh) * 2022-02-14 2022-05-10 中南大学 基于频域和空域的图像超分辨率方法

Also Published As

Publication number Publication date
US20250272786A1 (en) 2025-08-28

Similar Documents

Publication Publication Date Title
US12511713B2 (en) Image super-resolution reconstructing
Sun et al. Shufflemixer: An efficient convnet for image super-resolution
CN112488923B (zh) 图像超分辨率重建方法、装置、存储介质及电子设备
CN111429347A (zh) 图像超分辨率重建方法、装置及计算机可读存储介质
CN108074215A (zh) 图像升频系统及其训练方法、以及图像升频方法
CN113628115B (zh) 图像重建的处理方法、装置、电子设备和存储介质
CN110852948A (zh) 基于特征相关性的图像超分辨方法、存储介质及终端设备
Esmaeilzehi et al. UPDResNN: A deep light-weight image upsampling and deblurring residual neural network
CN115409855B (zh) 图像处理方法、装置、电子设备和存储介质
CN115526773A (zh) 图像重建方法及装置、设备、存储介质
CN111798385A (zh) 图像处理方法及装置、计算机可读介质和电子设备
WO2021218414A1 (zh) 视频增强方法及装置、电子设备、存储介质
CN116188254A (zh) 基于傅里叶域的超分图像处理方法、装置、设备及介质
US20240320793A1 (en) Method and apparatus with super-sampling
Liu et al. Gradient prior dilated convolution network for remote sensing image super-resolution
Zhang et al. Network architecture for single image super‐resolution: A comprehensive review and comparison
CN114066722B (zh) 用于获取图像的方法、装置和电子设备
CN114640796B (zh) 视频处理方法、装置、电子设备及存储介质
US20250217925A1 (en) Distributed image scaling
WO2023217270A1 (zh) 图像超分方法、超分网络参数调整方法、相关装置及介质
CN117196959B (zh) 基于自注意力的红外图像超分辨率方法、装置及可读介质
CN117114981A (zh) 超分网络参数调整方法、装置、设备、介质及程序产品
WO2023093838A1 (zh) 超分图像处理方法、装置、设备及介质
CN113905177A (zh) 视频生成方法、装置、设备及存储介质
CN115760658A (zh) 图像处理方法、装置、存储介质及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23803038

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18858296

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 06-03-2025)

WWP Wipo information: published in national office

Ref document number: 18858296

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 23803038

Country of ref document: EP

Kind code of ref document: A1