WO2020238558A1

WO2020238558A1 - Image super-resolution method and system

Info

Publication number: WO2020238558A1
Application number: PCT/CN2020/088215
Authority: WO
Inventors: 夏树涛; 戴涛; 李清; 林栋�; 汪漪
Original assignee: 鹏城实验室; 清华大学深圳国际研究生院
Priority date: 2019-05-24
Filing date: 2020-04-30
Publication date: 2020-12-03
Also published as: CN110175953A; CN110175953B

Abstract

The present invention provides an image super-resolution method and system. The method comprises: take an image to be processed as the input of a convolutional neural network super-resolution model, the convolutional neural network super-resolution model being composed of four execution modules which are connected in sequence; a first execution module processes said image to obtain a first processing image; a second execution module processes the first processing image and outputs a second processing image; a third execution module processes the second processing image and outputs a third processing image; a fourth execution module processes the third processing image and outputs a super-resolution image. Based on the present invention, the convolutional neural network super-resolution model sets the weighted features for said image, the important features in said image are determined by learning the weighted features, and the super-resolution processing is carried out according to the important features, so that the feature expression capability of the convolutional neural network super-resolution model is improved, and the detail quality of the super-resolution image obtained after the super-resolution processing is greatly improved.

Description

An image super-resolution method and system

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on May 24, 2019, the application number is 201910439532.8, and the invention title is "an image super-resolution method and system", the entire content of which is incorporated into this application by reference in.

Technical field

The present invention relates to the field of artificial intelligence technology, in particular to an image super-resolution method and system.

Background technique

Image super-resolution is a very important preprocessing link in computer vision systems such as video surveillance, logistics, and face recognition. Image super-resolution can reconstruct super-resolution images from low-resolution images to meet user browsing requirements. Thanks to the powerful nonlinear expression capabilities of Convolutional Neural Networks (CNN), the super-resolution model of Convolutional Neural Networks can be used to achieve super-resolution processing of low-resolution images to obtain super-resolution images.

At present, the super-resolution processing performance of the existing convolutional neural network super-division model is often only related to the depth or width of the convolutional neural network structure. However, the super-resolution processing performance improvement effect brought by increasing the depth or width of the convolutional neural network structure is very limited, and cannot meet the high standard requirements of users for super-resolution images. Therefore, the super-resolution images generated by the existing convolutional neural network super-division model still have many shortcomings.

Summary of the invention

In view of this, the embodiments of the present invention provide an image super-resolution method and system to solve the problem that the image details of the super-resolution image generated through the existing convolutional neural network super-resolution model are not obvious.

In order to achieve the foregoing objective, the embodiments of the present invention provide the following technical solutions:

The first aspect of the embodiments of the present invention discloses an image super-resolution method. The image super-resolution method includes:

The image to be processed is used as the input of the pre-built convolutional neural network super-division model, the convolutional neural network super-division model is composed of four successively connected execution modules, and the second execution module is superimposed and embedded in the second-order channel attention The residual module composition of the module;

The image to be processed is processed through the first execution module in the convolutional neural network super-division model, and the first processed image is obtained as the input of the second execution module. The size of the image to be processed is the same;

Using the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the input of the third execution module in the convolutional neural network superdivision model;

Based on the third execution module and preset scale factors of the input image and output image that are set in advance, the second processed image is processed to obtain a third processed image whose size meets the preset scale factor, and the The third processed image is used as the input of the fourth execution module in the convolutional neural network super-division model;

Performing a mapping process on the third processed image via a fourth execution module, and output a super-resolution image corresponding to the image to be processed.

Optionally, in the above-mentioned image super-resolution method, the pre-built process of the convolutional neural network super-division model includes:

Constructing a training set, the training set including a low-resolution image and a high-resolution image corresponding to the low-resolution image;

Input the low-resolution image into a preset convolutional neural network model to perform feature extraction, feature amplification, and feature mapping to obtain a processed image;

Based on the low resolution image, the high resolution image corresponding to the low resolution image, and the processed image, the preset loss function and optimization algorithm are used to train the preset convolutional neural network model until The preset convolutional neural network model outputs a high-resolution image corresponding to the low-resolution image, and it is determined that the currently trained convolutional neural network model is a convolutional neural network super-division model;

Wherein, the convolutional neural network super-division model is composed of four execution modules connected in sequence, and the second execution module is composed of a residual module superimposed and embedded in a second-order channel attention module.

Optionally, in the foregoing image super-resolution method, the process of superimposing and embedding the residual module of the second-order channel attention module to form the second execution module includes:

Embed the preset second-order channel attention module into the residual module to obtain a residual module with weighted features;

Determine the number of residual modules with weighted features required to construct the second execution module;

Stack each of the residual modules with weighted features in turn to obtain the second execution module.

Optionally, in the above-mentioned image super-resolution method, embedding the second-order channel attention module into the residual module to obtain the residual module with weighted features includes:

The first convolutional layer, the activation layer, the second convolutional layer, the first residual unit, the second-order channel attention module, and the second residual unit are sequentially connected in order to obtain the residual module with weighted features.

Optionally, in the foregoing image super-resolution method, the process of presetting the second-order channel attention module includes:

Obtain the first feature of any layer in the convolutional layer of the convolutional neural network based on the first processed image as input;

Performing mapping processing on the first feature according to the matrix reorganization method to obtain the second feature;

Based on the transposition of the first feature and the second feature, a sample variance matrix is calculated;

Normalizing the sample variance matrix to obtain a covariance matrix;

Based on the covariance matrix, a row mean vector based on the depth dimension is calculated;

Dimensionality reduction learning and dimensionality increase learning are sequentially performed on the row mean vector based on the depth dimension to obtain the first weight;

Normalizing the first weight to obtain a second weight;

Obtaining a weighted feature based on the first feature and the second weight;

The second-order channel attention module is constructed using the weighted features.

The second aspect of the embodiments of the present invention discloses an image super-resolution system, which includes:

The input unit is used to use the image to be processed as the input of the pre-built convolutional neural network super-division model, the convolutional neural network super-division model is composed of four sequentially connected execution modules, and the second execution module is superimposed and embedded The residual module composition of the second-order channel attention module;

The first execution unit is configured to process the to-be-processed image via the first execution module in the convolutional neural network super-division model, and obtain the first processed image as the input of the second execution module. A size of the processed image is the same as the size of the image to be processed;

The second execution unit is configured to use the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the first in the convolutional neural network super-division model 3. Input of execution module;

The third execution unit is configured to process the second processed image based on the third execution module and the preset scale factor of the preset input image and output image size to obtain a second processed image whose size meets the preset scale factor Three processing images, using the third processing image as the input of the fourth execution module in the convolutional neural network super-division model;

The fourth execution unit is configured to perform mapping processing on the third processed image via the fourth execution module, and output a super-resolution image corresponding to the image to be processed.

Optionally, in the foregoing image super-resolution system, the input unit includes:

A constructing subunit for constructing a training set, the training set including a low-resolution image and a high-resolution image corresponding to the low-resolution image;

The processing subunit is used to input the low-resolution image into a preset convolutional neural network model for feature extraction, feature amplification, and feature mapping, to obtain a processed image;

The training subunit is used to train the preset volume based on the low-resolution image, the high-resolution image corresponding to the low-resolution image, and the processed image, using a preset loss function and optimization algorithm Convolutional neural network model until the preset convolutional neural network model outputs a high-resolution image corresponding to the low-resolution image, and determining that the currently trained convolutional neural network model is a convolutional neural network superdivision model; Wherein, the convolutional neural network super-division model is composed of four execution modules connected in sequence, and the second execution module is composed of a residual module superimposed and embedded in a second-order channel attention module.

Optionally, in the above image super-resolution system, it further includes:

The first construction unit is used to embed the preset second-order channel attention module into the residual module to obtain the residual module with weighted characteristics, and determine the weighted characteristic of the residual module required to construct the second execution module The number of modules is sequentially stacked to obtain the second execution module.

Optionally, in the above-mentioned image super-resolution system, the second-order channel attention module is embedded in the residual module to obtain the first building unit of the residual module with weighted characteristics, which is specifically configured to sequentially connect the first The convolutional layer, the activation layer, the second convolutional layer, the second-order channel attention module, and the residual are used to obtain the residual module with weighted features.

Optionally, in the above-mentioned image super-resolution system, it further includes: a second construction unit, and the second construction unit includes:

The acquiring subunit is configured to acquire the first feature of any layer in the convolutional layer of the convolutional neural network based on the first processed image as input;

The mapping subunit is used to perform mapping processing on the first feature according to the matrix reorganization method to obtain the second feature;

The feature calculation subunit is configured to calculate a sample variance matrix based on the transposition of the first feature and the second feature;

The matrix normalization subunit is used to normalize the sample variance matrix to obtain a covariance matrix;

The matrix calculation subunit is configured to calculate a row mean vector based on the depth dimension based on the covariance matrix;

The learning subunit is used to sequentially perform dimensionality reduction learning and dimensionality increase learning on the row mean vector based on the depth dimension to obtain the first weight;

The weight normalization subunit is used to normalize the first weight to obtain the second weight;

The feature weighting subunit is used to obtain a weighted feature based on the first feature and the second weight;

The construction subunit is used to construct a second-order channel attention module using the weighted feature.

Based on the image super-resolution method and system provided by the above-mentioned embodiments of the present invention, the image to be processed is used as the input of the pre-built convolutional neural network super-division model, and the convolutional neural network super-division model consists of four sequentially connected The second execution module is composed of a residual module that is sequentially superimposed and embedded in a second-order channel attention module; the first execution module in the convolutional neural network super-division model processes the image to be processed, Obtain a first processed image as the input of the second execution module; use the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the convolution The input of the third execution module of the neural network hyperdivision model; based on the third execution module and the preset scale factor of the preset input image and output image size, the second processed image is up-sampled to obtain The third processed image whose size meets the preset scale factor is used as the input of the fourth execution module in the convolutional neural network super-division model; the third processed image is processed through the fourth execution module. The processed image is subjected to mapping processing, and a super-resolution image corresponding to the image to be processed is output. Based on the embodiment of the present invention, the convolutional neural network super-division model sets weighted features for the image to be processed, and determines important features in the image to be processed by learning the weighted features, and performs super-resolution based on the important features Processing, thereby improving the feature expression ability of the super-division model of the convolutional neural network, so that the detail quality of the super-resolution image obtained after the super-resolution processing is greatly improved.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work.

FIG. 1 is a schematic flowchart of an image super-resolution method provided by an embodiment of the present invention;

2 is a schematic flowchart of a method for constructing a convolutional neural network super-division model provided by an embodiment of the present invention;

3 is a schematic flowchart of a method for constructing a second execution module according to an embodiment of the present invention;

4 is a schematic structural diagram of a residual module with weighted features provided by an embodiment of the present invention;

5 is a schematic flowchart of a method for constructing a second-order channel attention module according to an embodiment of the present invention;

6 is a schematic structural diagram of an image super-resolution system provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of another image super-resolution system provided by an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of another image super-resolution system provided by an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of another image super-resolution system provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

In this application, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes no Other elements clearly listed, or also include elements inherent to this process, method, article or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment including the element.

As shown in FIG. 1, it is a schematic flowchart of an image super-resolution method provided by an embodiment of the present invention. The method includes the following steps:

S101: Use an image to be processed as an input of a pre-built convolutional neural network super-division model.

In S101, the convolutional neural network super-division model is composed of four execution modules connected in sequence, and the execution modules include a first execution module, a second execution module, a third execution module, and a fourth execution module. Wherein, the second execution module is composed of a residual module that is sequentially superimposed and embedded in a second-order channel attention module.

It should be noted that the second-order channel attention module utilizes the second-order statistical characteristics of the features of the input image in the convolutional neural network superdivision model, so that the convolutional neural network superdivision model adaptively learns features The importance of better focus on using useful features to improve the feature expression ability of convolutional neural networks, thereby improving the processing effect of image super-resolution.

S102: Process the image to be processed via the first execution module in the convolutional neural network super-division model, and obtain the first processed image as the input of the second execution module.

In S102, the size of the first processed image is the same as the size of the image to be processed.

It should be noted that the first execution module includes a convolution layer, and the first execution module performs a convolution operation on the image to be processed based on the convolution layer to obtain the first processed image.

S103: Use the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the input of the third execution module in the convolutional neural network super-division model.

In S103, the second execution module uses its own multiple residual modules with second-order channel attention modules to weight the features corresponding to the first processed image multiple times, thereby determining the first One processes the more important features in the image, and obtains a second processed image that contains weighted features.

It should be noted that, because the second processed image contains weighted features, in the subsequent processing of the third execution module and the fourth execution module, through the learning of the weighted features, the importance of the image to be processed is determined Features, and super-resolution processing is performed according to important features, thereby improving the feature expression ability of the convolutional neural network super-division model.

It should be noted that the size of the second processed image is the same as the size of the image to be processed.

S104: Process the second processed image based on the third execution module and the preset scale factors of the input image and output image that are preset to obtain a third processed image whose size meets the preset scale factor. As the input of the fourth execution module in the convolutional neural network super division model.

In S104, the second processed image is input to the third execution module, and the third execution module amplifies the second processed image according to a preset scale factor, and sends it to the fourth execution module The third processed image is output.

It should be noted that the third execution module includes a convolutional layer, and based on the convolutional layer, the third execution module performs feature amplification on the second processed image to obtain the first image whose size meets the preset scale factor. Three processing images.

It should be noted that the specific value of the preset scale factor can be set by a technician according to actual conditions.

S105: Perform mapping processing on the third processed image via the fourth execution module, and output a super-resolution image corresponding to the image to be processed.

In S105, the size of the super-resolution image is the same as the size of the image to be processed.

It should be noted that the fourth execution module includes a convolution layer, and the fourth execution module maps the third processed image based on the convolution layer to obtain and output the super-resolution image.

In the embodiment of the present invention, the image to be processed is used as the input of the pre-built convolutional neural network superdivision model. The convolutional neural network superdivision model is composed of four execution modules connected in sequence, and the second execution module is The residual module of the second-order channel attention module is superimposed and embedded; the image to be processed is processed through the first execution module in the convolutional neural network super-division model to obtain the first processed image as the second Input of the execution module; using the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the third execution module of the convolutional neural network super-division model Based on the third execution module, and the preset scale factor of the input image and output image preset in advance, the second processed image is processed to obtain a third processed image whose size meets the preset scale factor, The third processed image is used as the input of the fourth execution module in the convolutional neural network super-division model; the third processed image is mapped through the fourth execution module, and the output corresponds to the to-be-processed Super-resolution image of the image. Based on the embodiment of the present invention, the convolutional neural network super-division model sets weighted features for the image to be processed, and determines important features in the image to be processed by learning the weighted features, and performs super-resolution based on the important features Processing, thereby improving the feature expression ability of the super-division model of the convolutional neural network, so that the detail quality of the super-resolution image obtained after the super-resolution processing is greatly improved.

Preferably, the above-mentioned convolutional neural network super-division model shown in FIG. 1, in the specific construction process, refer to FIG. 2, which is a schematic flowchart of a method for constructing a convolutional neural network super-division model provided by an embodiment of the present invention. The method includes the following steps:

S201: Construct a training set.

In S201, the training set includes a low-resolution image and a high-resolution image corresponding to the low-resolution image.

S202: Input the low-resolution image into a preset convolutional neural network model to perform feature extraction, feature amplification, and feature mapping, to obtain a processed image.

In S202, in the preset convolutional neural network model, perform a convolution operation on the low-resolution image; then, perform a feature on the low-resolution image after the convolution operation through the convolutional neural network structure Extraction; secondly, the feature in the low-resolution image is enlarged through an up-sampling model; finally, based on the high-resolution image corresponding to the low-resolution image, the enlarged low-resolution image is The feature is mapped to obtain the processed image.

S203: Based on the low-resolution image, the high-resolution image corresponding to the low-resolution image, and the processed image, use the preset loss function and optimization algorithm to train the preset convolutional neural network model until the preset convolution The neural network model outputs a high-resolution image corresponding to the low-resolution image, and it is determined that the currently trained convolutional neural network model is a convolutional neural network superdivision model.

In S203, the convolutional neural network super-division model is composed of four execution modules connected in sequence, wherein the second execution module is composed of a residual module superimposed and embedded in a second-order channel attention module.

It should be noted that, in the embodiment of the present invention, the loss function includes but is not limited to L1-norm, and the optimization algorithm includes but is not limited to a stochastic gradient descent algorithm.

In the embodiment of the present invention, by constructing a training set, the training set includes a low-resolution image and a high-resolution image corresponding to the low-resolution image; the low-resolution image is input to a preset convolutional nerve The network model performs feature extraction, feature magnification, and feature mapping to obtain processed images; based on low-resolution images, high-resolution images corresponding to low-resolution images, and processed images, using preset loss functions and The optimization algorithm trains the preset convolutional neural network model until the preset convolutional neural network model outputs a high-resolution image corresponding to the low-resolution image, and determines that the currently trained convolutional neural network model is a convolutional neural network Network super-division model. Based on the embodiment of the present invention, a convolutional neural network superdivision model can be effectively constructed.

Preferably, as shown in FIG. 1 above, the second execution module is composed of a residual module superimposed and embedded in the second-order channel attention module. In the specific construction process, refer to FIG. 3, which is an example provided by the embodiment of the present invention. A schematic flow chart of a method for constructing a second execution module, the method includes the following steps:

S301: Embedding a preset second-order channel attention module into the residual module to obtain a residual module with weighted features.

In S301, the residual module includes two convolutional layers, one activation layer and two residual units.

Preferably, the first convolutional layer, the activation layer, the second convolutional layer, the first residual unit, the second-order channel attention module, and the second residual unit are sequentially connected in order to obtain the weighted feature Residual module.

In a specific implementation, the specific structure of the residual module with weighted features can refer to FIG. 4.

S302: Determine the number of residual modules with weighted features required to construct the second execution module.

In S302, the greater the number of residual modules with weighted features, the better the processing effect of the second execution module.

It should be noted that the number of residual modules with weighted features required by the second execution module can be set by a technician according to actual conditions, and is not limited in the embodiment of the present invention.

S303: Stack each residual module with weighted features in sequence to obtain a second execution module.

In S303, a convolutional neural network structure can be constructed by stacking multiple residual modules. In other words, the second execution module is the convolutional neural network structure. Since the convolutional neural network structure is formed by stacking each of the residual modules with weighted features, the convolutional neural network structure can set weighted features for the input image. It can be seen from this that the second execution module can generate a second processed image including a weighted feature based on the first processed image.

In the embodiment of the present invention, the second-order channel attention module is embedded in the residual module to obtain the residual module with weighted characteristics; the number of residual modules with weighted characteristics required to construct the second execution module is determined ; Sequentially stack each of the residual modules with weighted features to obtain the second execution module. Based on the embodiment of the present invention, a second execution module with a second-order channel attention mechanism can be effectively constructed.

Preferably, in S301 shown in FIG. 3, the process of the preset second-order channel attention module is in specific implementation. Referring to FIG. 5, it is a method for constructing second-order channel attention provided by the embodiment of the present invention. Schematic diagram of the flow of the module method, the method includes the following steps:

S501: Obtain the first feature of any layer in the convolutional layer of the convolutional neural network based on the first processed image as the input.

In S501, the first feature is specifically a H×W×C feature map, and the H×W×C feature map is marked as x, where H is the height of the convolutional layer, and W is the convolution The width of the layer, C is the depth of the convolutional layer.

S502: Perform mapping processing on the first feature according to the matrix reorganization method to obtain the second feature.

In S502, the H×W×C feature map x is mapped to the (H*W)×C feature X according to the matrix reorganization method, and X is output, and X is the second feature.

S503: Calculate the sample variance matrix based on the transposition of the first feature and the second feature.

In S503, the sample variance matrix Σ is calculated according to formula (1).

among them,

I refers to the identity matrix of (H*W)×(H*W), and 1 refers to a matrix with all 1 elements.

S504: Normalize the sample variance matrix to obtain a covariance matrix.

In S504, according to formula (2), the sample variance matrix Σ is normalized to obtain the covariance matrix Y.

Y＝Σ ^0.5 ＝UΛ ^0.5 U ^T (2)

Among them, Λ=diag(λ ₁ ,...,λ _C ), U refers to an orthogonal matrix, and Λ refers to a diagonal matrix whose elements are eigenvalues λ _i , and each eigenvalue in the diagonal matrix λ _{i is} sorted in descending order, λ is a positive integer, and i refers to the number of columns of the diagonal matrix.

S505: Based on the covariance matrix, calculate the row mean vector based on the depth dimension.

In S505, the mean value of the elements in the jth row of the covariance matrix Y is calculated according to formula (3), and the C-dimensional row mean vector is determined according to the mean value of the elements in the jth row.

Wherein, j refers to the number of rows of the diagonal matrix.

S506: Perform dimensionality reduction learning and dimensionality increase learning sequentially on the row mean vector based on the depth dimension to obtain the first weight.

In S506, the C-dimensional row mean vector is used as the input of the preset first fully connected network, and 1×1×C/r is output to obtain the output result m. The output result m is used as the input of the preset second fully connected network, and 1×1×C is output to obtain the first weight.

S507: Perform normalization processing on the first weight to obtain the second weight.

In S507, a Sigmoid function is used to perform normalization calculation on the first weight to obtain the second weight.

S508: Obtain a weighted feature based on the first feature and the second weight.

In S508, the first feature and the second weight are multiplied to obtain the weighted feature.

S509: Construct a second-order channel attention module using weighted features.

In S509, embed the weighted feature into a convolutional neural network to generate the second-order channel attention module.

In the embodiment of the present invention, the first feature of any layer in the convolutional layer of the convolutional neural network is obtained based on the first processed image as input; the first feature is mapped according to the matrix reorganization method to obtain the second Feature; based on the first feature and the second feature, calculate the sample variance matrix; normalize the sample variance matrix to obtain the covariance matrix; based on the covariance matrix, calculate the row based on the depth dimension Mean vector; performing dimensionality reduction learning and dimensionalization learning on the row mean vector based on the depth dimension to obtain a first weight; normalizing the first weight to obtain a second weight; based on the first feature And the second weight to obtain a weighted feature, and use the weighted feature to construct a second-order channel attention module. Based on the embodiment of the present invention, the second-order channel attention module can be effectively constructed.

Based on the above-mentioned image super-resolution method provided by the embodiment of the present invention, the embodiment of the present invention also provides an image super-resolution system. As shown in FIG. 6, it is the image super-resolution system provided by the embodiment of the present invention. Structure diagram, the system includes:

The input unit 100 is configured to use the image to be processed as the input of the pre-built convolutional neural network super-division model, the convolutional neural network super-division model is composed of four sequentially connected execution modules, and the second execution module is superimposed and The residual module is composed of the second-order channel attention module embedded.

The first execution unit 200 is configured to process the image to be processed via the first execution module in the convolutional neural network super-division model to obtain the first processed image as the input of the second execution module, and The size of the first processed image is the same as the size of the image to be processed.

The second execution unit 300 is configured to use the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the convolutional neural network super-division model The input of the third execution module.

The third execution unit 400 is configured to process the second processed image based on the third execution module and preset scale factors of the preset input image and output image to obtain a size meeting the preset scale factor The third processed image is used as the input of the fourth execution module in the convolutional neural network super-division model.

The fourth execution unit 500 is configured to perform mapping processing on the third processed image via the fourth execution module, and output a super-resolution image corresponding to the image to be processed.

In the embodiment of the present invention, the image to be processed is used as the input of the pre-built convolutional neural network superdivision model. The convolutional neural network superdivision model is composed of four execution modules connected in sequence, and the second execution module is The residual module of the second-order channel attention module is superimposed and embedded; the image to be processed is processed through the first execution module in the convolutional neural network super-division model to obtain the first processed image as the second Input to the execution module; use the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the third execution module of the convolutional neural network super-division model Based on the third execution module, and the preset scale factor of the input image and output image preset in advance, the second processed image is processed to obtain a third processed image whose size meets the preset scale factor, The third processed image is used as the input of the fourth execution module in the convolutional neural network super-division model; the third processed image is mapped through the fourth execution module, and the output corresponds to the to-be-processed Super-resolution image of the image. Based on the embodiment of the present invention, the convolutional neural network super-division model sets weighted features for the image to be processed, and determines important features in the image to be processed by learning the weighted features, and performs super-resolution based on the important features Processing, thereby improving the feature expression ability of the super-division model of the convolutional neural network, so that the detail quality of the super-resolution image obtained after the super-resolution processing is greatly improved.

Preferably, in conjunction with FIG. 6, referring to FIG. 7, it is a schematic structural diagram of another image super-resolution system provided by an embodiment of the present invention, and the input unit 100 includes:

The construction subunit 101 is configured to construct a training set, the training set including a low-resolution image and a high-resolution image corresponding to the low-resolution image.

The processing subunit 102 is configured to input the low-resolution image into a preset convolutional neural network model for feature extraction, feature amplification, and feature mapping, to obtain a processed image.

The training subunit 103 is configured to train the preset loss function and optimization algorithm based on the low resolution image, the high resolution image corresponding to the low resolution image, and the processed image. Convolutional neural network model, until the preset convolutional neural network model outputs a high-resolution image corresponding to the low-resolution image, and it is determined that the currently trained convolutional neural network model is a convolutional neural network super-division model ; Wherein, the convolutional neural network super-division model is composed of four sequentially connected execution modules, and the second execution module is composed of a residual module superimposed and embedded in a second-order channel attention module.

Preferably, referring to FIG. 6, referring to FIG. 8, it is a schematic structural diagram of another image super-resolution system provided by an embodiment of the present invention, and the system further includes:

The first construction unit 600 is used to embed the second-order channel attention module into the residual module to obtain a residual module with weighted features, and determine the number of residual modules with weighted features required to construct the second execution module , Stacking each of the residual modules with weighted features in turn to obtain the second execution module.

Preferably, the first construction unit 600 is specifically configured to sequentially connect the first convolutional layer, the activation layer, the second convolutional layer, the second-order channel attention module, and the residual in order to obtain the residual weighted feature. Poor module.

Preferably, referring to FIG. 8 and FIG. 9, it is a schematic structural diagram of another image super-resolution system according to an embodiment of the present invention. The system further includes: a second construction unit 700.

The second construction unit 700 includes:

The obtaining subunit 701 is configured to obtain the first feature of any layer in the convolutional layer of the convolutional neural network based on the first processed image as input.

The mapping subunit 702 is configured to perform a mapping process on the first feature according to a matrix reorganization method to obtain a second feature.

The feature calculation subunit 703 is configured to calculate a sample variance matrix based on the transposition of the first feature and the second feature.

The matrix normalization subunit 704 is configured to perform normalization processing on the sample variance matrix to obtain a covariance matrix.

The matrix calculation subunit 705 is configured to calculate a row mean vector based on the depth dimension based on the covariance matrix.

The learning subunit 706 is configured to sequentially perform dimensionality reduction learning and dimensionality increase learning on the row mean vector based on the depth dimension to obtain the first weight.

The weight normalization subunit 707 is configured to perform normalization processing on the first weight to obtain the second weight.

The feature weighting subunit 708 is configured to obtain weighted features based on the first feature and the second weight.

The construction subunit 709 is used to construct a second-order channel attention module using the weighted features.

In the embodiment of the present invention, the first feature of any layer in the convolutional layer of the convolutional neural network is obtained based on the first processed image as input; the first feature is mapped according to the matrix reorganization method to obtain the second Feature; based on the transposition of the first feature and the second feature, the sample variance matrix is calculated; the sample variance matrix is normalized to obtain the covariance matrix; based on the covariance matrix, the calculation is based on the depth Dimensional row mean vector; performing dimensionality reduction learning and dimensionalization learning on the depth dimension-based row mean vector to obtain a first weight; normalizing the first weight to obtain a second weight; based on the The first feature and the second weight are used to obtain a weighted feature, and the weighted feature is used to construct a second-order channel attention module. Based on the embodiment of the present invention, the second-order channel attention module can be effectively constructed.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. The system and system embodiments described above are merely illustrative, where the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, namely It can be located in one place, or it can be distributed to multiple network model units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement it without creative work.

Professionals may further realize that the units and algorithm steps of the examples described in the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of both, in order to clearly illustrate the possibilities of hardware and software. Interchangeability. In the above description, the composition and steps of each example have been generally described in accordance with the function. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the present invention.

The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined in this document can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this document, but should conform to the widest scope consistent with the principles and novel features disclosed in this document.

Claims

An image super-resolution method, characterized in that the method includes:

The image to be processed is used as the input of the pre-built convolutional neural network super-division model, the convolutional neural network super-division model is composed of four successively connected execution modules, and the second execution module is superimposed and embedded in the second-order channel attention The residual module composition of the module;

The image to be processed is processed through the first execution module in the convolutional neural network super-division model, and the first processed image is obtained as the input of the second execution module. The size of the image to be processed is the same;

Using the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the input of the third execution module in the convolutional neural network superdivision model;

Based on the third execution module and preset scale factors of the input image and output image that are set in advance, the second processed image is processed to obtain a third processed image whose size meets the preset scale factor, and the The third processed image is used as the input of the fourth execution module in the convolutional neural network super-division model;

Performing a mapping process on the third processed image via a fourth execution module, and output a super-resolution image corresponding to the image to be processed.
The method according to claim 1, wherein the process of constructing the pre-built convolutional neural network super-division model comprises:

Constructing a training set, the training set including a low-resolution image and a high-resolution image corresponding to the low-resolution image;

Input the low-resolution image into a preset convolutional neural network model to perform feature extraction, feature amplification, and feature mapping to obtain a processed image;

Based on the low resolution image, the high resolution image corresponding to the low resolution image, and the processed image, the preset loss function and optimization algorithm are used to train the preset convolutional neural network model until The preset convolutional neural network model outputs a high-resolution image corresponding to the low-resolution image, and it is determined that the currently trained convolutional neural network model is a convolutional neural network super-division model;

Wherein, the convolutional neural network super-division model is composed of four execution modules connected in sequence, and the second execution module is composed of a residual module superimposed and embedded in a second-order channel attention module.
The method according to claim 1, wherein the process of superimposing and embedding the residual module of the second-order channel attention module to form the second execution module comprises:

Embed the preset second-order channel attention module into the residual module to obtain a residual module with weighted features;

Determine the number of residual modules with weighted features required to construct the second execution module;

Stack each of the residual modules with weighted features in turn to obtain the second execution module.
The method according to claim 3, wherein the embedding the second-order channel attention module into the residual module to obtain the residual module with weighted features comprises:

The first convolutional layer, the activation layer, the second convolutional layer, the first residual unit, the second-order channel attention module, and the second residual unit are sequentially connected in order to obtain the residual module with weighted features.
The method according to claim 4, wherein the process of presetting the second-order channel attention module comprises:

Obtain the first feature of any layer in the convolutional layer of the convolutional neural network based on the first processed image as input;

Performing mapping processing on the first feature according to the matrix reorganization method to obtain the second feature;

Based on the transposition of the first feature and the second feature, a sample variance matrix is calculated;

Normalizing the sample variance matrix to obtain a covariance matrix;

Based on the covariance matrix, a row mean vector based on the depth dimension is calculated;

Dimensionality reduction learning and dimensionality increase learning are sequentially performed on the row mean vector based on the depth dimension to obtain the first weight;

Normalizing the first weight to obtain a second weight;

Obtaining a weighted feature based on the first feature and the second weight;

The second-order channel attention module is constructed using the weighted features.
An image super-resolution system, characterized in that it comprises:

The input unit is used to use the image to be processed as the input of the pre-built convolutional neural network super-division model, the convolutional neural network super-division model is composed of four sequentially connected execution modules, and the second execution module is superimposed and embedded The residual module composition of the second-order channel attention module;

The first execution unit is configured to process the to-be-processed image via the first execution module in the convolutional neural network super-division model, and obtain the first processed image as the input of the second execution module. A size of the processed image is the same as the size of the image to be processed;

The second execution unit is configured to use the second execution module to perform feature extraction and feature processing on the first processed image, and output a second processed image containing weighted features as the first in the convolutional neural network super-division model 3. Input of execution module;

The third execution unit is configured to process the second processed image based on the third execution module and the preset scale factor of the preset input image and output image size to obtain a second processed image whose size meets the preset scale factor Three processing images, using the third processing image as the input of the fourth execution module in the convolutional neural network super-division model;

The fourth execution unit is configured to perform mapping processing on the third processed image via the fourth execution module, and output a super-resolution image corresponding to the image to be processed.
The system according to claim 6, wherein the input unit comprises:

A constructing subunit for constructing a training set, the training set including a low-resolution image and a high-resolution image corresponding to the low-resolution image;

The processing subunit is used to input the low-resolution image into a preset convolutional neural network model for feature extraction, feature amplification, and feature mapping, to obtain a processed image;

The training subunit is used to train the preset volume based on the low-resolution image, the high-resolution image corresponding to the low-resolution image, and the processed image, using a preset loss function and optimization algorithm Convolutional neural network model until the preset convolutional neural network model outputs a high-resolution image corresponding to the low-resolution image, and determining that the currently trained convolutional neural network model is a convolutional neural network superdivision model; Wherein, the convolutional neural network super-division model is composed of four execution modules connected in sequence, and the second execution module is composed of a residual module superimposed and embedded in a second-order channel attention module.
The system according to claim 6, further comprising:

The first construction unit is used to embed the preset second-order channel attention module into the residual module to obtain the residual module with weighted characteristics, and determine the weighted characteristic of the residual module required to construct the second execution module The number of modules is sequentially stacked to obtain the second execution module.
The system according to claim 8, wherein the first building unit of the second-order channel attention module embedded in the residual module to obtain the residual module with weighted characteristics is specifically configured to sequentially connect the second A convolutional layer, an activation layer, a second convolutional layer, a second-order channel attention module, and residuals are used to obtain the residual module with weighted features.
The system according to claim 9, further comprising: a second construction unit, the second construction unit comprising:

The acquiring subunit is configured to acquire the first feature of any layer in the convolutional layer of the convolutional neural network based on the first processed image as input;

The mapping subunit is used to perform mapping processing on the first feature according to the matrix reorganization method to obtain the second feature;

The feature calculation subunit is configured to calculate a sample variance matrix based on the transposition of the first feature and the second feature;

The matrix normalization subunit is used to normalize the sample variance matrix to obtain a covariance matrix;

The matrix calculation subunit is configured to calculate a row mean vector based on the depth dimension based on the covariance matrix;

The learning subunit is used to sequentially perform dimensionality reduction learning and dimensionality increase learning on the row mean vector based on the depth dimension to obtain the first weight;

The weight normalization subunit is used to normalize the first weight to obtain the second weight;

The feature weighting subunit is used to obtain a weighted feature based on the first feature and the second weight;

The construction subunit is used to construct a second-order channel attention module using the weighted feature.