WO2019192588A1 - 图像超分方法及装置 - Google Patents

图像超分方法及装置 Download PDF

Info

Publication number
WO2019192588A1
WO2019192588A1 PCT/CN2019/081445 CN2019081445W WO2019192588A1 WO 2019192588 A1 WO2019192588 A1 WO 2019192588A1 CN 2019081445 W CN2019081445 W CN 2019081445W WO 2019192588 A1 WO2019192588 A1 WO 2019192588A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
map
input feature
processing
resolution image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/081445
Other languages
English (en)
French (fr)
Inventor
姚春凤
冯柏岚
颜波
李可
巴热巴合提亚尔
钱莉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2019192588A1 publication Critical patent/WO2019192588A1/zh
Priority to US17/029,554 priority Critical patent/US11593916B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4084—Scaling of whole images or parts thereof, e.g. expanding or contracting in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/48—Extraction of image or video features by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the embodiments of the present invention relate to the field of communications technologies, and in particular, to an image over-method and apparatus.
  • Image super-resolution reconstruction referred to as image super resolution, also known as image super-score (SR), refers to the use of image processing methods to reconstruct low-resolution images (LR) into high-resolution images (high) Resolution, HR) is an image processing technology that can effectively improve the sharpness of images. It is of great significance for video surveillance, satellite imagery, camera photography, HDTV, medical imaging and other fields. For example, in the field of video surveillance, high-definition video provides more favorable support for road traffic supervision, and also effectively improves the level of social security. However, in practical situations, many applications cannot directly obtain high resolution due to various considerations. Image or video, and image over-score technology can effectively improve the problem.
  • the neural network structure used therein has multiple direct network branches, that is, the output of a layer of neural network directly skips several layers of neural networks and directly connects to the processed signals, and the directly connected network branches do not need to undergo intermediate convolution, etc. Operation, which makes the output layer calculate the loss function, and when the gradient is returned, the gradient can be directly transmitted to several layers before the direct connection branch, thereby reducing the gradient disappearance of the deep neural network during training and improving the deep nerve. The convergence of the network.
  • VDSR can get better super-score performance by deepening the network depth, but this brings the disadvantages of large parameter and super-time-consuming.
  • VDSR can solve the convergence problem to some extent based on the neural network structure of multiple direct network branches, with the increase of the number of convolution layers in the network, there is still a convergence problem after a certain number, which affects the over-score. The effect is to reduce the image quality.
  • the embodiment of the present application discloses an image over-score method and device, which can improve image over-score quality and obtain better image over-score effect.
  • an embodiment of the present application provides an image over-score method, including: receiving a low-resolution image; pre-processing the low-resolution image to obtain a vertical gradient map, a horizontal gradient map, and a luminance map; a vertical gradient map, the horizontal gradient map and the luminance map respectively as information of three different dimensions to form a feature map to be input; performing size conversion on the to-be-input feature map to obtain an input feature map; and inputting the characteristic map Performing nonlinear transformation processing to obtain a nonlinearly transformed input feature map, performing weighting processing on the input feature map and the nonlinearly transformed input feature map to obtain an output feature map; and performing size conversion on the output feature map A residual map is obtained; combining the low resolution image and the residual map to obtain a high resolution image.
  • the embodiment of the present application is an image super-division method based on a deep neural network, where the deep neural network includes an input layer, at least one hidden layer, and an output layer; Receiving a low resolution image; the at least one hidden layer may be used to preprocess the low resolution image to obtain a vertical gradient map, a horizontal gradient map, and a luminance map; and the vertical gradient map
  • the horizontal gradient map and the brightness map are respectively used as information of three different dimensions to form a feature map to be input; and the input feature map is subjected to size conversion to obtain an input feature map; and the input feature map is nonlinear Transforming the input feature map obtained by the nonlinear transformation, and weighting the input feature map and the nonlinearly transformed input feature map to obtain an output feature map; and performing size conversion on the output feature map a residual map; and combining the low resolution image and the residual map to obtain a high resolution image; the output layer can be used to output the high Resolution image.
  • the input feature map and the non-linearly transformed input feature map are weighted to obtain an output feature map; and one of the objects of the weighting process: “the input feature map” is Skip several hidden layers in the deep neural network, that is, directly obtained from the output of the previous step without processing, that is, directly obtained from the direct connected branch in the deep neural network, the direct connected branch can well overcome the depth The problem of convergence of the neural network, thereby improving the image super-segment effect based on the deep neural network, and, due to the weighting processing method adopted herein, the input feature map and the nonlinearly transformed input feature map are correspondingly The ratios are added and added, so that the image quality of the reconstructed high-resolution image is greatly improved.
  • the embodiment of the present application combines the low resolution image and the residual image to obtain a high resolution image, that is, the low resolution image is directly obtained by the direct connection branch, so that the embodiment of the present application further It overcomes the convergence problem of deep neural network and finally gets better image super-segment effect.
  • the method before performing the weighting process on the input feature map and the non-linearly transformed input feature map, the method further includes:
  • the set of weight values is used for weighting the input feature map and the nonlinearly transformed input feature map.
  • the set of weight values is obtained according to the input feature map, including:
  • the horizontal gradient map and the luminance map are respectively used as information of three different dimensions to form a feature map to be input, including:
  • the vertical gradient map as second dimension information
  • the luminance map as the third dimensional information
  • the third dimensional information The information constitutes the feature map to be input.
  • the performing a non-linear transformation process on the input feature map to obtain a nonlinearly transformed input feature map includes:
  • This possible implementation reduces the amount of parameters at the front end by performing a de-fitting operation on the input feature map.
  • the input parameters of the convolution operation are reduced.
  • the calculation of the convolution operation is reduced, which can reduce the time of over-division and increase the speed of image over-score.
  • the combining the low resolution image and the residual image to obtain a high resolution image comprises:
  • Pixel values of each pixel in the low resolution image are added to pixel values of corresponding pixels in the residual map to obtain pixel values of corresponding pixels in the high resolution image.
  • the combining the low resolution image and the residual image to obtain a high resolution image comprises:
  • a pixel value of each pixel in the low resolution image and a pixel value of a corresponding pixel in the residual map are weighted and summed to obtain a pixel value of a corresponding pixel in the high resolution image.
  • the coefficients used for the weighted summation are obtained from the trained deep neural network, the well-developed deep neural network can make the output high-resolution image have the best image quality. Therefore, this possible implementation manner High-resolution images that can be over-outputted with images are of better quality.
  • the input feature map and the non-linearly transformed input feature map are weighted, including: inputting the input feature map and the nonlinearly transformed input according to weight information
  • the feature map is weighted, and the weight information is obtained by the trained deep neural network; wherein the weight information includes a weight value or a set of weight values; the set of weight values may be a weight map or a The weight vector is used to represent.
  • the performing weighting processing on the input feature map and the non-linearly transformed input feature map according to the weight information includes:
  • the weight value is one, weighting each first pixel value in the input feature map and each second pixel value in the nonlinearly transformed input feature map by using the weight value And obtaining a third pixel value of the output feature map, the first pixel value is any one of the input feature maps, and the relationship between the second pixel value and the first pixel value is one-to-one correspondence And the third pixel value has a one-to-one correspondence with the first pixel value.
  • the input feature map and the nonlinearly transformed input feature map can be given different weight values; the configuration of each weight value enables the
  • the output feature map has better expressive power, resulting in better image quality for the final output high resolution image.
  • each weight value in the set of weight values is respectively used in each of the first pixel values in the input feature map and the nonlinearly transformed input feature map.
  • a pixel value is a one-to-one correspondence, and the third pixel value has a one-to-one correspondence with the first pixel value.
  • the well-developed deep neural network can make the output high-resolution image have the best image quality, and therefore, the trained deep neural network is obtained.
  • the individual weight values can make the output feature map have better expressive ability. Further, when the weight value is a group, different local features in the input feature map can be more targeted. The different weight values make the output feature map have more possibilities, so that the image over-score method obtains a higher quality high-resolution map.
  • an image over-score method including:
  • Receiving a low resolution image preprocessing the low resolution image to obtain a vertical gradient map, a horizontal gradient map, and a luminance map; and the vertical gradient map, the horizontal gradient map, and the luminance map as three Information of different dimensions to form a feature map to be input; performing size conversion on the feature map to be input to obtain an input feature map; weighting the input feature map according to a weighted processing flow in series to obtain an output feature map;
  • the series weighting process includes n processing steps, and the n processing links are connected in series;
  • the serial weighting processing flow includes n sets of weight values, and the n sets of weight values are used for the n processing links and one-to-one correspondence with the n processing links, wherein n is a positive integer greater than or equal to 2; performing size conversion on the output feature map to obtain a residual map; and the low-resolution image and the The residual map is combined to obtain a high resolution image.
  • the embodiment of the present application is an image super-division method based on a deep neural network, where the deep neural network includes an input layer, at least one hidden layer, and an output layer; Receiving a low resolution image; the at least one hidden layer may be used to preprocess the low resolution image to obtain a vertical gradient map, a horizontal gradient map, and a luminance map; and the vertical gradient map
  • the horizontal gradient map and the brightness map are respectively used as information of three different dimensions to form a feature map to be input; and the size conversion of the feature map to be input is obtained to obtain an input feature map; and according to the weighting processing flow in series
  • the series weighting processing flow includes n processing links, and the n processing links are connected in series;
  • the serial weighting processing flow includes n groups a weight value, the n sets of weight values are respectively used for the n processing links and are in one-to-one correspondence with the n processing links, where n is
  • the input feature map is weighted according to a serial weighting process to obtain an output feature map
  • the serial weighting process includes n processing links, and the n processing links Connect in series.
  • the key difference between the image super-division method provided by the embodiment of the present application and the first aspect is that: in the first aspect, the input feature map and the non-linear transformed input feature graph are weighted once, that is, the output feature map is output, and The next step is processed.
  • the series weighting process includes n processing steps, wherein each processing step has a round of weighting processing, that is, the first processing step.
  • the output is not directly output as the output feature map, but is input as a second processing link for the second processing step to perform a weighting process again until the last processing link in the series weighting processing flow is weighted. Processing, using the output as the output feature map, for further processing of the output feature map in the next step.
  • the series-weighted processing flow can make the reconstructed high-resolution image quality more than the high-resolution image quality obtained by the method provided by the first aspect, that is, the image super-segment effect is more it is good.
  • the input feature map is weighted according to the weighted processing flow in series to obtain the output feature map, including:
  • the process of processing one of the n processing steps includes: performing an upper channel processing and a lower channel processing on the input feature map of the processing link i, where the upper channel processing includes: according to the processing link The input feature map of i obtains a set of weight values of the processing link i; the lower channel processing includes performing nonlinear transformation processing on the input feature map of the processing link i, and performing nonlinear transformation after processing by the lower channel The input feature map of the processing link i; the input feature map of the processing link i and the input feature map of the nonlinearly transformed processing link i according to a set of weight values of the processing link i The weighting process is performed to obtain an output feature map of the processing link i; wherein the processing link i is any one of the n processing links; wherein, when the processing link i is the serialized weighting processing flow In the first link, the input feature map of the processing link i is the input feature map; when the processing link i is the last one of the series weighting processing flows At the time of the link,
  • the processing of the upper channel includes: performing a convolution operation on an input feature map of the current processing link and obtaining an upper channel convolution result, and convolving the upper channel according to an S-type function.
  • the result is processed to obtain a set of weight values for the current processing session.
  • the processing of the lower channel specifically includes: de-fitting the input feature map of the current processing link, obtaining a de-fitting result, and performing a convolution operation on the de-fitting result The nonlinear transformation result is obtained.
  • the input feature map is weighted according to the weighted processing flow in series to obtain the output feature map, including:
  • the processing of one of the n processing steps includes: performing nonlinear transformation processing on the input feature map of the processing link i, and obtaining an input characteristic map of the processing link i after the nonlinear transformation; Performing weighting processing on the input feature map of the processing link i and the input feature map of the nonlinearly transformed processing link i by the set of weight values corresponding to the processing link i to obtain the processing link i
  • An output feature map wherein the processing link i is any one of the n processing links, 1 ⁇ i ⁇ n; wherein, when the processing link i is the first in the series weighting processing flow
  • the input feature map of the processing link i is the input feature map; when the processing link i is the last link in the series weighting process flow, the output feature of the processing link i
  • the figure shows the output feature map.
  • the method further includes:
  • the obtaining a set of weight values of the processing link i according to the input feature map of the processing link i includes:
  • the input feature map of the processing link i is subjected to nonlinear transformation processing, and the input feature map of the processing link i obtained by the nonlinear transformation includes:
  • the n is equal to three.
  • the combining the low resolution image and the residual image to obtain a high resolution image comprises:
  • Pixel values of each pixel in the low resolution image are added to pixel values of corresponding pixels in the residual map to obtain pixel values of corresponding pixels in the high resolution image.
  • the combining the low resolution image and the residual image to obtain a high resolution image comprises:
  • a pixel value of each pixel in the low resolution image and a pixel value of a corresponding pixel in the residual map are weighted and summed to obtain a pixel value of a corresponding pixel in the high resolution image.
  • the training process of the deep neural network is mainly the training process for the connection parameters of the layers of the neural network, the embodiment of the present application will be before the start of the embodiment
  • the weight coefficient obtained by the trained deep neural network can make the output high-resolution image have the best image quality. Therefore, the high resolution of the image super-output can be achieved by this possible implementation. Rate images have better quality.
  • each of the n sets of weight values includes at least one weight value (wherein at least one includes one and at least two cases), wherein a set of weight values
  • the processing link corresponding to the weight value of the group performs weighting processing on the link according to the one weight value; when a set of weight values includes at least two weight values, the processing link corresponding to the weight value of the group is according to the At least two weight values are used to perform weighting processing on the link.
  • different weight values may be assigned to different parts of the input feature map input to the link, so that the feature map output by the processing link corresponding to the at least two weight values is more
  • the richness also makes the feature map output by this link have more possibilities, so that the image over-subscription method can obtain a higher-quality high-resolution image.
  • an image super-sorting apparatus comprising means for performing the method of the first aspect or any of the possible implementations of the first aspect.
  • an image super-sorting apparatus comprising means for performing the method of the second aspect or any of the possible implementations of the second aspect.
  • an image super-sorting device comprising a storage medium and a central processing unit, the storage medium being a non-volatile storage medium, wherein the storage medium stores a computer executable program, the central A processor is coupled to the non-volatile storage medium and executes the computer executable program to implement the method of the first aspect or any of the possible implementations of the first aspect.
  • an image super-sorting device comprising a storage medium and a central processing unit, the storage medium being a non-volatile storage medium, wherein the storage medium stores a computer executable program, the central A processor is coupled to the non-volatile storage medium and executes the computer-executable program to implement the method of the second aspect or any of the possible implementations of the second aspect.
  • a chip comprising a processor and a data interface, the processor reading the instructions stored on the memory through the data interface, performing the first aspect or any possible implementation of the first aspect The method in the way.
  • the chip may further include a memory, where the memory stores an instruction, the processor is configured to execute an instruction stored on the memory, when the instruction is executed, The processor is for performing the method of the first aspect or any of the possible implementations of the first aspect.
  • a chip comprising a processor and a data interface, the processor reading an instruction stored on the memory through the data interface, performing any possible implementation of the second aspect or the second aspect The method in the way.
  • the chip may further include a memory, where the memory stores an instruction, the processor is configured to execute an instruction stored on the memory, when the instruction is executed, The processor is for performing the method of any of the possible implementations of the second aspect or the second aspect.
  • a ninth aspect a computer readable storage medium storing program code for device execution, the program code comprising any of the possible implementations for performing the first aspect or the first aspect The instructions in the method.
  • a tenth aspect a computer readable storage medium storing program code for device execution, the program code comprising any of the possible implementations for performing the second aspect or the second aspect The instructions in the method.
  • FIG. 1 is a schematic structural diagram of an electronic device 100 for neural network calculation according to an embodiment of the present application
  • FIG. 2 is a schematic flowchart of an image over-score method provided by an embodiment of the present application.
  • FIG. 2a is a schematic structural view of a road unit provided by an embodiment of the present application.
  • 2b is a schematic structural view of a road unit provided by an embodiment of the present application.
  • FIG. 3 is a schematic flowchart of an image over-score method provided by an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of an image over-score device according to an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of another image super-score device according to an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of another electronic device for neural network calculation according to an embodiment of the present application.
  • Image super-resolution reconstruction (referred to as image super-resolution, also known as image over-score)
  • Image Super Resolution refers to a technique of recovering a low resolution (LR) image from a high resolution (HR) image by a computer using an image processing method.
  • High-resolution images mean that the image has a high pixel density and can provide more image detail, which often plays a key role in the application.
  • Image super-resolution methods can be divided into two categories: the first is the traditional image super-resolution method, such as the use of data operations to achieve LR to HR recovery; the second is based on learning image super-resolution method, that is, the use The machine learning method implements LR to HR recovery.
  • the reconstructed image super-resolution method can statistically estimate the high-resolution image of the maximum posterior probability by using a frequency domain algorithm or a spatial domain algorithm.
  • the learning-based image super-resolution method can include two phases, a training phase and a test phase.
  • the training phase the initial super-resolution model and training set are first established.
  • the training set may include a plurality of low resolution images and a high resolution image corresponding to each low resolution image. Learning the correspondence between the high-resolution image and the low-resolution image through the low-resolution image in the training set and its corresponding high-resolution image, thereby correcting the value of the parameter in the initial super-resolution model, so that The error convergence between the high-resolution image and the reconstructed image finally determines the newly created super-resolution model obtained after training.
  • super-resolution reconstruction of the image can be guided by the newly created super-resolution model.
  • the method for acquiring a pair of training images may be obtained by first obtaining a high-resolution image and then the high-resolution image.
  • the fuzzy function is processed to obtain a low resolution image corresponding thereto, wherein the blur function is irreversible.
  • the initial super-resolution model may be an experimentally determined model and may be non-linear.
  • the super resolution model can be a convolutional neural network.
  • the neural network may be composed of neural units, and the neural unit may refer to an arithmetic unit having an input of x s and an intercept 1 , and the output of the arithmetic unit may be:
  • W s is the weight of x s
  • b is the offset of the neural unit.
  • f is the activation function of the neural unit for introducing nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as an input to the next layer of convolution.
  • the activation function can be a sigmoid function.
  • a neural network is a network formed by joining together a plurality of the above-described single neural units, that is, the output of one neural unit may be an input of another neural unit.
  • the input to each neural unit can be linked to the local accepted domain of the previous layer to extract features of the local accepted domain, which can be an area composed of several neural units.
  • Deep Neural Network can be understood as a neural network with many layers of hidden layers. There are no special metrics for "many”. We often say the essence of multi-layer neural networks and deep neural networks. It is the same thing. From the DNN according to the location of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in the middle is the hidden layer. The layers are fully connected, that is, any one of the i-th layers must be connected to any one of the i+1th layers. Although DNN looks complicated, it is not complicated in terms of the work of each layer.
  • the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as
  • the superscript 3 represents the number of layers in which the coefficient W is located, and the subscript corresponds to the output of the third layer index 2 and the input second layer index 4.
  • the coefficient of the kth neuron from the L-1th layer to the jth neuron in the Lth layer is defined as Note that the input layer has no W parameters.
  • more hidden layers make the network more capable of portraying complex situations in the real world. In theory, the more complex the model, the higher the “capacity”, which means it can perform more complex learning tasks.
  • CNN Convosutionas Neuras Network
  • the convolutional neural network consists of a feature extractor consisting of a convolutional layer and a sub-sampling layer.
  • the feature decimator can be thought of as a filter, and the convolution process can be thought of as convolution with an input image or a convolution feature map using a trainable filter.
  • a convolutional layer is a layer of neurons in a convolutional neural network that convolves an input signal. In the convolutional layer of a convolutional neural network, a neuron can be connected only to a portion of the adjacent layer neurons.
  • a convolutional layer usually contains several feature planes, each of which can be composed of a number of rectangularly arranged neural units.
  • the neural units of the same feature plane share weights, and the weights shared here are convolution kernels. Sharing weight can be understood as the way in which image information is extracted regardless of location.
  • the implicit principle is that the statistical information of a certain part of the image is the same as other parts. That means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learned image information.
  • multiple convolution kernels can be used to extract different image information. Generally, the larger the number of convolution kernels, the richer the image information reflected by the convolution operation.
  • the convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can be reasonably weighted by learning during the training process of the convolutional neural network.
  • the direct benefit of shared weighting is to reduce the connections between the layers of the convolutional neural network while reducing the risk of overfitting.
  • the convolutional neural network can use the error back propagation (BP) algorithm to correct the parameters of the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model is smaller and smaller.
  • BP error back propagation
  • the input signal is forwarded until the output produces an error loss, and the parameters in the initial super-resolution model are updated by the back propagation error loss information, thereby converging the error loss.
  • the backpropagation algorithm is a backpropagation motion dominated by error loss, aiming to obtain parameters of the optimal super-resolution model, such as weight matrix.
  • the pixel value of the image can be a red, green, and blue (RGB) color value, and the pixel value can be a long integer representing the color.
  • the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component.
  • Blue represents the blue component
  • Green represents the green component
  • Red represents the red component.
  • the pixel value can be a grayscale value.
  • Image features mainly include image color features, texture features, shape features, and spatial relationship features.
  • the color feature is a global feature that describes the surface properties of the scene corresponding to the image or image area; the general color feature is based on the characteristics of the pixel, at which point all pixels belonging to the image or image area have their own contributions. Since the color is insensitive to changes in the direction, size, etc. of the image or image area, the color feature does not capture the local features of the object in the image well.
  • the texture feature is also a global feature. It also describes the surface properties of the scene corresponding to the image or image area. However, since the texture is only a feature of the surface of the object and does not fully reflect the essential properties of the object, only the texture feature is used. Unable to get high level image content. Unlike color features, texture features are not pixel-based features, and they require statistical calculations in regions that contain multiple pixels.
  • contour feature There are two types of representation methods for shape features, one is contour feature and the other is region feature.
  • the contour feature of the image is mainly for the outer boundary of the object, while the region feature of the image is related to the entire shape region.
  • the spatial relationship feature refers to the mutual spatial position or relative direction relationship between multiple objects segmented in the image. These relationships can also be divided into connection/adjacency relationship, overlap/overlap relationship, and inclusion/inclusion relationship.
  • spatial location information can be divided into two categories: relative spatial location information and absolute spatial location information. The former relationship emphasizes the relative situation between the targets, such as the above-mentioned relationship, and the latter relationship emphasizes the distance between the targets and the orientation.
  • image features may be used as some examples of features in the image, and the images may also have other features, such as higher-level features: semantic features, which are not expanded here.
  • the relationship between the vertical gradient map, the horizontal gradient map, and the luminance map to be mentioned in the embodiments of the present application may be: the vertical gradient map and the horizontal gradient map in the embodiment of the present application describe adjacent pixels.
  • the transformation between the horizontal transformation is represented by a horizontal gradient map
  • the vertical transformation is represented by a vertical gradient map, which includes both texture features and shape features.
  • the brightness map contains color features.
  • the vertical gradient map, the horizontal gradient map, and the luminance map in the embodiments of the present application can be considered as a comprehensive embodiment of some of the above various image features.
  • a deeper deep neural network that is, a deeper neural network with more layers may be used.
  • the more layers of the deep neural network the larger the corresponding parameter quantity, and the larger the calculation amount, which makes the super-division time-consuming; and when the depth of the deep neural network reaches a certain level, it will inevitably lead to The problem of difficulty in convergence.
  • the embodiment of the present application provides an image super-division method, which uses a highway network (Highway Networks) and a residual network in combination with a deep neural network structure.
  • the resolution image is reconstructed to obtain a high resolution image.
  • the road network and the residual network can be considered as special deep neural networks. Specifically, it can be understood as a special connection between multiple hidden layers in the deep neural network. Some or all of the steps in the image over-sub-method of the embodiment of the present application can be understood as being built on the special deep neural network structure constructed by the road network and the residual network, and implemented by the special deep neural network.
  • the image super-score method provided by the embodiment of the present application can be understood to be based on a trained deep neural network, wherein the trained deep neural network may include a residual network and a road network, which are used in the embodiment of the present application. Residual networks and road networks can be understood as trained, and can also be called trained neural networks. For the sake of brevity, they will not be described again.
  • the residual network can be: in addition to layer-by-layer connection between multiple hidden layers in a deep neural network, for example, the first layer of the hidden layer is connected to the second layer of the hidden layer, and the second layer is connected by the hidden layer.
  • the third layer of the hidden layer, the third layer of the hidden layer is connected to the fourth layer of the hidden layer (this is a neural network data operation path, also can be called the neural network transmission), the residual network is more straight
  • this direct branch is directly connected from the first layer of the hidden layer to the fourth layer of the hidden layer, that is, the processing of the second layer and the third layer of the hidden layer is skipped, and the first layer of the hidden layer is The data is transferred directly to the layer 4 hidden layer for operation.
  • the road network may be: in addition to the above-mentioned operation path and direct branch, the deep neural network includes a weight acquisition branch, and the branch introduces a transform gate to obtain a weight value and outputs
  • the weight value T is used for the operations of the above arithmetic path and the direct connection branch.
  • the weighted value T obtained by using the weight acquisition branch, the input feature map obtained through the direct connection branch, and the processed input feature map obtained by processing the input feature map through the layer-by-layer connected operation path are processed.
  • a weighting process is performed to obtain an output feature map.
  • the required information, and further, the output high-resolution image has better image quality; further, the embodiment of the present application adopts the idea of the residual network to directly skip the hidden layer by directly connecting the input feature map to the direct connection branch to a layer, and the combination of the layer and the processed residual map to obtain a high-resolution image, the design of the direct-connected branch further enhances the convergence effect based on the above-mentioned lifting convergence effect, so that the present application is implemented
  • the deep neural network in the example has better convergence ability, which leads to higher image super-segment efficiency and better image super-segment effect for the image hyper-segmentation method based on the deep neural network.
  • the image over-score provided by the embodiment of the present application is excessive.
  • the method is based on a deep neural network, and specifically, may be performed according to a convolutional neural network.
  • the convolutional neural network can be understood as a special deep neural network, that is, the convolutional neural network (CNN) is a deep neural network with a convolution structure.
  • the convolutional neural network may be pre-trained. Since the convolutional neural network is used for image over-score in the embodiment of the present application, the convolutional neural network for image over-score is also referred to as super Subnet.
  • the work of each layer in the deep neural network can be mathematically expressed.
  • the work of each layer in the deep neural network can be understood as the transformation of the input space to the output space through the operation of five input spaces (the set of input vectors) (ie the row space to the column of the matrix) Space), these five operations include: 1, up / down dimension; 2, zoom in / zoom out; 3, rotation; 4, translation; 5, "bend".
  • 1, 2, 3 are operated by Finished
  • 4 operations by Complete the operation of 5 is implemented by ⁇ ().
  • the reason why the word "space” is used here is because the classified object is not a single thing, but a kind of thing, and space refers to a collection of all individuals of such things.
  • W is a weight vector
  • each value in the vector represents a weight value of one of the neurons in the layer neural network.
  • This vector W determines the spatial transformation of the input space to the output space described above, ie the weight W of each layer controls how the space is transformed.
  • the purpose of training a deep neural network is to finally obtain a weight matrix (a weight matrix formed by vectors W of many layers) of all layers of the trained neural network. Therefore, the training process of the neural network is essentially the way to learn to control the spatial transformation, and more specifically the learning weight matrix.
  • the output of the deep neural network is as close as possible to the value that really wants to be predicted, it is possible to update each layer of the nerve by comparing the predicted value of the current network with the truly desired target value, and then based on the difference between the two.
  • the weight vector of the network (of course, there will usually be an initialization process before the first update, which is a pre-configured parameter for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it The prediction is lower and constant adjustment until the neural network can predict the true desired target value.
  • the system configures parameters for the super-sub-network, and the parameter is an initialized parameter, which is obtained without learning.
  • the parameters configured by the system are essentially initialized matrix vectors for configuring the structure of the network and the connection between the layers of the neural network.
  • a training data set needs to be constructed, where the training data set includes a number of pairs of low resolution images and high resolution images, ie, each pair contains a low resolution image and a high resolution image corresponding to the low resolution image.
  • Pairs of low-resolution images and high-resolution images are input into the above-described configured super-division network, and are calculated by weight vectors of layers in the super-division network, and the reconstructed high-resolution images are output.
  • Calculating the loss function by outputting the high-resolution image and the high-resolution image included in the training data set, calculating according to the back propagation algorithm of the neural network, updating the parameters of each layer that need to be learned in the super-network, and obtaining the updated The parameters of each layer, that is, the weight vector W.
  • the weight matrix formed by the updated weight vector W of each layer (since each weight vector in the matrix has been updated, the connection relationship of each layer in the network is updated with the update of the weight vector) is included in the training data set.
  • Several pairs of low resolution images and high resolution images continue to be trained until the difference between the reconstructed high resolution image of the super-segment network output and the high resolution image included in the training data set, or loss of the loss function is satisfied Pre-set conditions, then output the trained weight matrix to complete the training.
  • the trained super-sub-network can be used to perform the image super-division method of the embodiment of the present application. This involves the application process of the super-sub-network, that is, the low-resolution image is input into the trained super-sub-network to obtain high-resolution. image. This process will be expanded in detail later, and will not be introduced here.
  • the high-resolution image obtained by image over-score can be quality-evaluated according to an image evaluation method (for example, an evaluation function), and the above-mentioned trained super-division network can be performed according to the evaluation result.
  • an image evaluation method for example, an evaluation function
  • the above-mentioned trained super-division network can be performed according to the evaluation result.
  • the trained super-sub-network it is possible to continuously and iteratively improve the poor-quality super-sub-images, thereby obtaining a better super-sub-network, and outputting better-quality high-resolution images for image over-score applications.
  • the fine-tuning training of the super-sub-network can be performed, so that the super-division network can continuously expand and train the new data in the application, so that the image over-score effect is better.
  • the image over-sub-method provided by the embodiment of the present application may be executed on a server, executed in the cloud, or on a terminal (such as a mobile phone, a computer, a camera, or an autonomous vehicle).
  • the data calculation used in the embodiment of the present application may be a hardware structure of a conventional processor + memory, or may be a hardware structure of a processor + a neural network processor + a memory, or may be a neural network.
  • the processing structure of the processor + memory is not limited in this application. The following is exemplified by the processing structure of the processor + neural network processor + memory. As shown in Figure 1:
  • the electronic device 100 includes a processor 102 and a memory 103.
  • the processor 102 may be a general name of a plurality of processors, and may include only a traditional processor, such as a CPU (Central Processing Unit), and may also include a conventional processor and a neural network processor (here Refers to a processor that can be used for neural network operations, and is not limited to a dedicated neural network processor.
  • the common neural network processors may include, but are not limited to, the following: GPU (Graphics Processing Unit), NPU (Neural-network Processing Uint, neural network processor), TPU (Tensor Processing Unit).
  • the processor 102 may include a CPU, and the processor 102 may include a CPU and a GPU.
  • the processor 102 may include a CPU, a GPU, and an NPU, or the processor 102 may include a CPU and a GPU.
  • the NPU and the TPU may also be a combination of any one or two of a CPU and a GPU, an NPU, and a TPU; wherein the number of CPUs, GPUs, NPUs, or TPUs included in the above example may be one or Multiple, it needs to be determined according to the calculation amount and calculation type corresponding to the specific application scenario of the neural network.
  • the embodiment of the present application is not limited.
  • the electronic device 100 can be a printed circuit board (PCB), an integrated circuit (IC), or a communication device such as a mobile phone, a portable computer, or the like.
  • the processor 102 and the memory 103 are disposed on the PCB as two discrete devices; when the processor 102 is composed of multiple processors, the multiple processors may also be separate devices.
  • the PCB board is set.
  • the processor 102, the memory 103 are integrated or packaged on an IC. It can be understood that the above two parts can be integrated or separated as needed. For example, when the processor 102 is composed of multiple processors, the multiple processors can be integrated or packaged in an application-specific integrated circuit (ASIC). And then placed on the PCB together with the memory 103.
  • ASIC application-specific integrated circuit
  • the bus 101 is connected to a plurality of processors in the processor 102 and the bus 101 of the memory 103.
  • the bus 101 may be an AXI (Advanced eXtensible Interface) bus protocol, or other bus protocol.
  • the processor 102 When the processor 102 includes a CPU, the CPU is configured to execute software programs and/or instructions stored in the memory 103 to perform various functions of the electronic device 100, which may be based on an X86 architecture, based on an ARM architecture, and based on a cortex. -A architecture CPU.
  • the memory 103 may include a volatile memory such as a random access memory (RAM), and may also include a non-volatile memory such as a flash memory, a hard disk or a solid state drive (SSD). It may also be a combination of the above types of memories.
  • the memory 103 is used to store related instructions and data.
  • the neural network processor may be a separate hardware chip, or may be a computing circuit integrated with the processor 102 in a hardware chip, such as IP (Intellectual Property , intellectual property) nuclear.
  • the one or more hardware chips may be an ASIC, a Programmable Logic Device (PLD), or a combination of the above types of hardware chips, wherein the PLD may be a Field Programmable Gate Array (FPGA). , Generic Array Logic (GAL), etc.
  • the electronic device 100 further includes a display device 104, an input device 105, a radio frequency circuit 106, and a communication module 107.
  • the display device 104 is configured to output visible text, graphics, video, and any combination thereof.
  • the display device 104 may be an LCD (Liquid Crystal Display) or an LPD (Lighting Power Density); Commands and information may be input to the electronic device 100 through the input device 105, such as image data or sound data to be recognized (in the embodiment of the present application, may be an image to be over-reconstructed), wherein the input device 105 may be a mouse, a keyboard, a scanner or a camera, etc.;
  • the RF circuit 106 is configured to receive and transmit electromagnetic waves, convert the electrical signals into electromagnetic waves, or convert the electromagnetic waves into electrical signals, and communicate with the communication network or other communication device through the electromagnetic waves;
  • the communication module 107 Communication data for processing communication data, such as electromagnetic waves received by the RF circuit 106 and converted into electrical signals.
  • the CPU and the memory 103 are connected via a bus 101.
  • the CPU acquires input data, input parameters and instructions for the neural network operation from the memory 103 via the bus 101, wherein the input data includes neural network data to be calculated, such as image data to be recognized, sound data; and input parameters include The parameters calculated by the neural network, such as offline trained neural network parameters; the instructions include information indicating a neural network layer, and the CPU configures a computing mode of the corresponding neural network layer according to the instructions, and the computing modes may include multiply, accumulate, and open A square operation, or a combination of these operations.
  • the CPU can also process multiple neural network layer calculations, such as a Convolution layer, a Full Connected layer, a Pooling layer, and a Normalize layer. Network layer.
  • the neural network processor may be integrated with a conventional processor (such as a CPU) in a hardware chip, or may be separate.
  • a conventional processor such as a CPU
  • the hardware chip, the bus 101 in the electronic device 100 communicatively connects the CPU, the memory 103, and the neural network processor to form a data path.
  • the CPU controls the neural network processor to initiate neural network operations via the bus 101; when the neural network processor completes the neural network calculation, the processor 101 is notified via the bus 101 that the calculation is complete.
  • the neural network processor acquires input data, input parameters and instructions for neural network calculation from the memory 103 via the bus 101, wherein the input data includes neural network data to be calculated, such as image data to be identified, sound data; input parameters include Parameters for neural network calculation, such as offline trained neural network parameters; the instructions include information indicating a neural network layer, and the neural network processor configures a calculation mode of the corresponding neural network layer according to the instructions, and the calculation modes may include multiplication, Accumulate, square, etc., or a combination of these operations. Alternatively, the neural network processor can handle computations of various neural network layers, such as Convolution, Full Connected, Pooling, Normalize, etc. Network layer.
  • the CPU is used to process the neural network alone and the CPU and the neural network processor are used for the neural network operation.
  • the difference is that only when the CPU performs the neural network operation, the interaction between the CPU and the neural network processor is lost, that is, The CPU does not need to control the neural network processor to start or end the calculation because it is the subject of the calculation itself; but because the CPU and the neural network processor are different in design, they are necessary for large-scale parallel operations in the neural network.
  • the computational efficiency is lower than that of the neural network processor. Therefore, the combination of the CPU and the neural network processor will be more common in the neural network calculation, but the embodiment of the present application does not limit this.
  • a neural network may include multiple neural network layers, and the same neural network layer may appear in a neural network multiple times.
  • a neural network layer required to identify an image includes: a convolutional layer, a sigmoid layer, a pooling layer, The sigmoid layer, the pooling layer and the fully connected layer, the above neural network layer comprises a 2-layer sigmoid layer and a 2-layer pool layer.
  • the processor 102 includes a neural network processor, in the neural network calculation, the neural network processor sequentially calculates each layer of the neural network layer, wherein the calculation result of the current neural network layer can be used as the next layer of the neural network layer. Calculated input.
  • the input data of the convolutional layer is the input data of the input parameters and the image, and the calculation result obtained by calculating the convolutional layer is used for the input data of the sigmoid layer.
  • the neural network processor first reads the input data to be processed from the memory 103 through the bus 101 during a complete calculation process on the layer of the neural network. Enter parameters and instructions. Since the calculation of different neural network layers requires different modes of calculation of the data, the input parameters and instructions read are related to the type of the specific neural network layer. The neural network processor calculates the input data and the input parameters according to the corresponding instructions, and the obtained calculation result is written into the memory 103 through the bus 101, and these calculation results are used as data required for the subsequent calculation of the neural network layer.
  • the neural network processor reads the input data and the input parameters from the memory 103 again through the bus 101, wherein the read input data may include the calculation result of the upper layer of the neural network layer or has not passed yet. Calculated data.
  • the internal operation structure of the neural network processor is briefly described below.
  • the most widely used ones are currently used. It is a GPU, so the following is mainly based on the GPU, and the GPU is compared with the CPU to explain the structure and operation characteristics of the GPU.
  • GPUs also known as display cores, vision processors, and display chips, make a microprocessor specifically designed for image operations. Similar to the CPU, except that the GPU is designed to perform complex mathematical and geometric calculations. For example, GPU-accelerated computing can provide extraordinary application performance because it transfers the application-intensive portion of the workload to the GPU while still running the rest of the program code by the CPU. Because the CPU is composed of several core components optimized for sequential serial processing, the GPU has a massively parallel computation consisting of thousands of more efficient cores designed for simultaneous processing of multiple tasks. Architecture. The computational complexity that a single core can handle in a GPU is much lower than the computational complexity that a single core in a CPU can handle.
  • GPUs and CPUs are very different is that they are designed for two different application scenarios because of their different design goals. For example, CPUs require a high degree of versatility to handle a variety of different data types, while at the same time requiring logical decisions, thus introducing a large number of branch jumps and interrupts. These make the internal structure of the CPU extremely complicated.
  • the GPU is faced with a highly unified, non-dependent large-scale data and a pure computing environment that does not need to be interrupted.
  • the primary goal of GPU processing is computation and data throughput, while the primary purpose of the internal transistors of the CPU is to reduce processing latency and keep the pipeline busy, which in turn determines the GPU's superior performance in terms of intensive computing compared to the CPU. All in all, the CPU and GPU are different in design because the tasks that were originally used for processing are different. Some tasks are similar to the ones that the GPU originally solves, so you can use the GPU to calculate.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • FIG. 2 is a schematic flowchart diagram of an image over-score method according to an embodiment of the present application. As shown in FIG. 2, the image oversubscription method includes, but is not limited to, the following steps S201-S207.
  • the low-resolution image received in step 201 is obtained by performing correlation processing on the original image that is originally obtained.
  • This correlation processing method is usually a bicubic interpolation method.
  • bicubic interpolation also known as bicubic interpolation, is a method for "interpolating" or increasing the number/density of "Pixel" in an image. Interpolation techniques are often used to increase graphics data so that it can increase print area and/or resolution when it is printed or otherwise output.
  • the purpose of this correlation process is to make the low resolution image larger than the original image and of the same size as the resulting high resolution image.
  • the vertical gradient map represents a vertical transformation between adjacent pixels in an image
  • a horizontal gradient map represents a lateral transformation between adjacent pixels in the image
  • a luminance map represents an image. Brightness information.
  • the pre-processing manner of step 202 is: performing different feature extraction from the low-resolution image to obtain different features, so that the three different feature maps are formed by different features (vertical Gradient map, horizontal gradient map, and luminance map), when the feature extraction is performed by using a convolution operation, the low resolution image may be separately convoluted by using three different convolution kernels to obtain the above vertical Gradient map, horizontal gradient map and brightness map.
  • the vertical gradient map, the horizontal gradient map, and the luminance map are respectively used as information of three different dimensions to form a feature map to be input.
  • the vertical gradient map, the horizontal gradient map, and the luminance map are respectively used as information of three different dimensions to form a feature map to be input, and the vertical gradient map, the horizontal gradient map, and the luminance map may be spliced, for example,
  • the vertical gradient map, the horizontal gradient map, and the luminance map are both X*Y two-dimensional maps
  • the to-be-input feature map obtained by the splicing is 3*X*Y, that is, the depth Z of the to-be-input feature map is 3.
  • S204 Perform size conversion on the to-be-input feature map to obtain an input feature map.
  • the size of the image may change.
  • the feature map to be input needs to be converted into a size (also referred to as dimensional conversion, the same later, The manner of the input feature is obtained.
  • the manner of the size conversion is usually a convolution operation, and of course, it may be another manner, which is not limited in the embodiment of the present application.
  • S205 Perform nonlinear transformation processing on the input feature map to obtain a nonlinearly transformed input feature map, and perform weighting processing on the input feature map and the nonlinearly transformed input feature map to obtain an output feature map.
  • the input feature map and the non-linearly transformed input feature map are weighted.
  • the weighting process may be performed by using a road network. .
  • the road network has been introduced in the foregoing.
  • the weight acquisition branch introduces a transform gate and a carry gate.
  • the transmission gate is simply referred to as T, and the portable gate is abbreviated as C.
  • one weight acquisition branch is used to obtain the weight value according to the input feature map and output the weight value T
  • one branch is used according to at least one hidden layer (for example, implied here)
  • the layer is n-layer) nonlinearly transforms the input signal
  • the other branch skips the n-layer hidden layer directly to a layer.
  • the input feature map used for performing the weighting process is directly obtained according to the direct connection branch, and the input feature map after the nonlinear transformation is nonlinearly transformed by the n-layer hidden layer. of.
  • Two probabilityes T and C are respectively assigned to the nonlinearly transformed input feature map and the input feature map, and further weighted to obtain an output feature map, wherein T is assigned to the nonlinear transformed feature map, C is assigned to the input feature map.
  • the size of the output feature map may change due to the processing in the middle of several steps.
  • the output feature map needs to be size-converted, so that the residual obtained after the conversion is obtained.
  • the size of the difference map is the same as the size of the low resolution image, thereby providing a basis for implementation of the combination of the low resolution image and the residual map.
  • the manner of the size conversion is usually a convolution operation, and of course, it may be another manner, which is not limited by the embodiment of the present application.
  • a residual network is used, that is, in addition to the processing branch of the residual map obtained through the series of processes, there is a direct branch, which is directly on the convolutional neural network.
  • the output of the direct connection branch in the embodiment of the present application is the low resolution image in step 207.
  • two feature maps are combined, one is a residual map processed by a multi-layer network (the multi-layer network here includes but is not limited to a road network, and the included road network is used for processing step 205), The other is a low-resolution image directly obtained by the directly connected branch, combining the residual map with the low-resolution image, essentially the pixel in the residual map and the low-resolution image.
  • the pixels in the combination are combined to obtain a high-resolution image, and the process of image over-score is completed.
  • Embodiment 1 may be an image super-division method based on a deep neural network, where the deep neural network includes an input layer, at least one hidden layer, and an output layer; the input layer may be used for A low resolution image is received; the at least one hidden layer can be used to process S202 through S207, and the output layer can be used to output the high resolution image.
  • the above-mentioned road network and residual network can be understood as being formed by a certain connection relationship between layers of neural networks in the deep neural network. Related parts of the following embodiments and implementations are similar to those herein, and are not described again.
  • step 205 since the output feature map in step 205 is obtained by weighting the input feature map and the nonlinearly transformed input feature map, on the one hand, the processing of step 205 includes directly connecting the directly connected branch. Entering the feature map, the direct link branch overcomes the gradient disappearance problem of the deep neural network to a certain extent, and thus brings better convergence effect. On the other hand, since the input feature map and the nonlinearly transformed input feature map are weighted in step 205, the input feature map and the nonlinearly transformed input feature map are respectively assigned differently. The weight values T and C, which can lead to better super-division effects.
  • the high resolution image is obtained by combining the low resolution image and the residual image in step 207, that is, the low resolution image is directly obtained by the direct connection branch, so that the embodiment of the present application further overcomes The gradient disappearance problem of deep neural network and the convergence performance of deep neural network are further improved, and finally a better image super-segment effect is obtained.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • the second embodiment includes all the steps of the first embodiment, and further includes S205a on the basis of the first embodiment, the S205a before the step 205, the S205a includes:
  • the set of weight values is used for weighting the input feature map and the nonlinearly transformed input feature map.
  • the obtaining, according to the input feature map, the set of weight values includes: performing a convolution operation on the input feature map and obtaining a convolution result, and processing the convolution result according to an S-type function, A weight value T is obtained, which is used to perform weighting processing on the input feature map and the nonlinearly transformed input feature map.
  • the road network includes a branch for acquiring the weight value T.
  • the branch may obtain the weight value T according to the trained deep neural network and output the weight value T.
  • the specific weight acquisition method may be the method described in S205a of the second embodiment, that is, the input feature map is convoluted and A convolution result is obtained, and the convolution result is processed according to an sigmoid function to obtain a weight value T.
  • the S-type function here is also called the Sigmoid function. Due to its single addition and the inverse of the inverse function, the Sigmoid function is often used as a threshold function of the neural network to map variables to 0,1.
  • the input feature map in the step 205 is subjected to nonlinear transformation processing to obtain a nonlinearly transformed input.
  • the feature map includes: performing a de-fitting operation on the input feature map, obtaining a de-fitting result, and performing a convolution operation on the de-fitting result to obtain the nonlinearly transformed input feature map.
  • FIG. 2a may be used to indicate a single road unit following a road network structure, where the road unit is There are three branches (see the previous introduction to the road network, not repeated here), as shown in Figure 2a, the upper channel branch, which is used to obtain the weight value T, as shown in Figure 2a, the lower channel branch, It is used to perform nonlinear transformation processing on the output signal according to at least one hidden layer (for example, the hidden layer is n layer), as shown in the direct connection branch shown in FIG.
  • the hidden layer for example, the hidden layer is n layer
  • this branch skips the n implied Layer processing
  • the signal x input to the road network unit is directly transmitted to the aggregation layer in FIG. 2a
  • the aggregation layer is used to input the feature map according to the weight value T obtained by the upper channel branch (in FIG. 2a, input)
  • the x representation) and the non-linearly transformed input feature map are weighted, and the weighting process described herein is a weighted summation.
  • the weight value T is assigned to the nonlinearly transformed input feature map
  • (1-T) is assigned to the input feature map.
  • Output T*y+(1-T)*x; where Output is the output of the aggregation layer in FIG. 2a, that is, the output feature map described in the embodiment of the present application, and y is a nonlinear transformation. After the input feature map, x is the input feature map.
  • the upper channel branch may include a convolution layer and a Sigmiod layer, specifically by the convolution layer 1 -1 and the Sigmiod layer are formed in series.
  • the lower channel branch may include a Dropout layer, a convolution layer, and a ReLU layer, and may be sequentially connected by a Dropout layer, a convolution layer 2-1, a ReLU1, a convolution layer 2-2, a ReLU2, and a convolution layer 2-3. form.
  • the ReLU is a Rectified Linear Unit.
  • the ReLU function is often used as an activation function of a neuron.
  • the ReLU function is essentially a piecewise linear function, which can change all negative values to 0 and positive values. This operation is called unilateral inhibition, and it is because of this unilateral inhibition that neurons in the neural network are sparsely activated.
  • the sparse neural network realized by ReLU can better mine related features and fit the training data.
  • the processing may be corresponding to the corresponding explanation of the foregoing step 205 in the first embodiment, that is, a branch is used to perform nonlinear transformation processing on the input signal according to at least one layer of the hidden layer.
  • the non- The linear transformation process includes the processing of the Dropout layer and the processing of the convolutional layer, where the Dropout layer is used to perform the de-fitting operation.
  • the de-fitting operation is performed on the input feature map to obtain a de-fitting result, and the de-fitting result is subjected to a convolution operation, and the embodiment of the present application first performs a de-fitting operation on the input feature map, thereby
  • the process of further reducing the parameter amount in the front end since the input parameter of the convolution operation is reduced, the calculation amount of the convolution operation is reduced, the over-consumption time can be reduced, and the image over-score can be improved. speed.
  • the step 207 may further include at least two specific implementation manners as follows:
  • S2072 Weighting and summing pixel values of each pixel in the low-resolution image and pixel values of corresponding pixels in the residual image to obtain pixel values of corresponding pixels in the high-resolution image.
  • step 2071 the combination of the low resolution image and the residual map is to add pixel values of corresponding positions in the two graphs, and The obtained result is a pixel value of a position in the high resolution image corresponding to the corresponding position of the low resolution image and the residual map.
  • step 2072 the combination of the two images is not only additive, but weighted summation using the weight values.
  • step 2072 since the coefficients used for the weighted summation are obtained through deep neural network learning (also called neural network training), in the process of learning the weight value by the deep neural network, the weight coefficient obtained by learning can make the output
  • the high resolution image has the best image quality, so that the high resolution image of the image super-output can be made to have better quality by step 2072.
  • Embodiment 3 is a diagrammatic representation of Embodiment 3
  • FIG. 3 is a schematic flowchart diagram of an image over-score method according to an embodiment of the present application. As shown in FIG. 3, the image oversubscription method includes, but is not limited to, the following steps S301-S307.
  • the vertical gradient map, the horizontal gradient map, and the luminance map are respectively used as information of three different dimensions to form a feature map to be input.
  • S304 Perform size conversion on the to-be-input feature map to obtain an input feature map.
  • S305 Perform weighting processing on the input feature map according to the weighting process in series to obtain an output feature map, where the serial weighting process includes n processing links, and the n processing links are connected in series;
  • the tandem weighting process includes n sets of weight values, and the n sets of weight values are respectively used for the n processing links and are in one-to-one correspondence with the n processing links, where n is greater than or equal to 2 A positive integer.
  • the steps S301 to S304 are the same as the steps S201 to S204 of the first embodiment, the steps 306 and the step 206 are the same, and the steps 307 and 207 are the same.
  • the steps S301 to S304 are the same as the steps S201 to S204 of the first embodiment, the steps 306 and the step 206 are the same, and the steps 307 and 207 are the same.
  • the steps S301 to S304 are the same as the steps S201 to S204 of the first embodiment, the steps 306 and the step 206 are the same, and the steps 307 and 207 are the same.
  • the steps S301 to S304 are the same as the steps S201 to S204 of the first embodiment, the steps 306 and the step 206 are the same, and the steps 307 and 207 are the same.
  • the steps S301 to S304 are the same as the steps S201 to S204 of the first embodiment, the steps 306 and the step 206 are the same, and the steps 307 and 207 are the same.
  • the key different from the first embodiment is the step 305.
  • step 205 adopts a road network, and introduces a transmission gate T and a carrying door C, which essentially acquires a weight value T by using a branch of the road network, and according to the weight value T
  • the weighted ratio of the input characteristic map after the nonlinear transformation obtained by the nonlinear transform processing is determined.
  • step 305 and step 205 are similar, except that step 205 ends the processing of the road network after a round of weighting processing, and outputs the output feature map to perform the processing of step 206.
  • the step 307 performs a plurality of rounds of weighting processing, that is, the tandem weighting processing flow described in S307.
  • the tandem weighting process includes n processing steps, and each processing link can be understood as a road network processing process.
  • a complete unit of a complete road network processing process can include three branches as described in step 205. I will not repeat them here.
  • step 307 includes multiple road network processing processes, and each road network processing process is connected in series, and each road network processing process has a weight acquiring branch of the road network, and passes the The weight acquisition branch obtains the weight value of the road network, and the weight value obtained in the process of the road network is used by the road network, that is, the n group weight values described in step 305 are used for the n Processing step and one-to-one correspondence with the n processing links", in a single road network processing process, that is, a single processing link among the above-mentioned n processing links, the processing flow is the same as the process of step 205, but There is a difference between the input and output.
  • the first processing step in the series process its input is the output of step 304, that is, the input feature map, but its output is not directly output as the output feature map, but through further roads.
  • Network processing that is, entering the next processing step, continuing the related operations of the road network described above, and the obtained output will be made again.
  • For the input of the next road network enter the next processing step until the last link in the series process is processed, and its output is taken as the output of step 305, that is, the output feature map, as the input of the next step 306. .
  • Embodiment 3 may be an image super-division method based on a deep neural network, where the deep neural network includes an input layer, at least one hidden layer, and an output layer; the input layer may be used for A low resolution image is received; the at least one hidden layer can be used to process S302 through S307, and the output layer can be used to output the high resolution image.
  • the above-mentioned road network and residual network can be understood as being formed by a certain connection relationship between layers of neural networks in the deep neural network. The relevant parts of the following embodiments are similar to those herein, and will not be described again.
  • the n is equal to 3. That is, the weighting process in series in step 305 includes three processing steps.
  • the processing of the current processing link includes: performing an upper channel processing and a lower channel processing on the input feature maps of the current processing link, where the upper channel processing includes: according to the current processing The input feature map of the link acquires a set of weight values of the current processing link; the lower channel processing includes nonlinear transformation processing, and the nonlinear transformation result is obtained through the processing of the lower channel; and the one-to-one correspondence with the current processing link a set of weight values for weighting the input feature map of the current processing link and the nonlinear transform result to obtain an output feature map of the current processing link; wherein the current processing link is the n processes Any one of the links; wherein, when the current processing link is the first link in the series of weighting processing flows, the input feature map of the current processing link is the input feature map; When the current processing link is the last link in the serialized weighting processing flow, the output of the current processing link The picture shows the intrinsic output characteristics of FIG.
  • the upper channel acquisition weight value may be understood as the branch for obtaining the weight value T among the three branches of the road network in step 205.
  • the specific operations of the branch may include:
  • the lower channel processing may be understood as the non-linearity of the input signal according to at least one hidden layer (for example, the hidden layer is n layer) in the three branches of the road network in step 205.
  • the specific operations of the branch may include:
  • this implementation of the embodiment of the present application first performs a de-fitting operation on the input feature map, thereby reducing the parameter amount at the front end, in the process of further performing the convolution operation on the de-fitting result, due to the input of the convolution operation
  • the parameter is reduced, so that the calculation amount of the convolution operation is reduced, the over-consumption time can be reduced, and the speed of image over-score can be improved.
  • the processing of one of the n processing steps includes: performing nonlinear transformation processing on the input feature map of the processing link i, and obtaining an input characteristic map of the processing link i after the nonlinear transformation; Performing weighting processing on the input feature map of the processing link i and the input feature map of the nonlinearly transformed processing link i by the set of weight values corresponding to the processing link i to obtain the processing link i
  • An output feature map wherein the processing link i is any one of the n processing links, 1 ⁇ i ⁇ n; wherein, when the processing link i is the first in the series weighting processing flow
  • the input feature map of the processing link i is the input feature map; when the processing link i is the last link in the series weighting process flow, the output feature of the processing link i
  • the figure shows the output feature map.
  • the method further includes:
  • the obtaining a set of weight values of the processing link i according to the input feature map of the processing link i includes:
  • step 307 may specifically include at least two specific implementation manners as follows:
  • S3072 Weighting and summing pixel values of each pixel in the low-resolution image and pixel values of corresponding pixels in the residual image to obtain pixel values of corresponding pixels in the high-resolution image.
  • step 3071 and step 2071 are similar, step 3072 is similar to step 2072, and the difference between step 3071 and step 3072 is the same as that of step 2071 and step 2072, so please refer to the related description at step 2071 and step 2072, and no longer here. Narration.
  • Embodiment 4 is a diagrammatic representation of Embodiment 4:
  • FIG. 4 is a schematic structural diagram of an image over-sub-device 400 according to an embodiment of the present application.
  • the image super-segment device includes, but is not limited to, the following modules 401-406.
  • a receiving module 401 configured to receive a low resolution image
  • a pre-processing module 402 configured to preprocess the low-resolution image to obtain a vertical gradient map, a horizontal gradient map, and a luminance map; and the vertical gradient map, the horizontal gradient map, and the luminance map are respectively used as Information of three different dimensions to constitute a feature map to be input;
  • a first size conversion module 403, configured to perform size conversion on the to-be-input feature map to obtain an input feature map
  • the highway module 404 is configured to perform nonlinear transformation processing on the input feature map to obtain a nonlinearly transformed input feature map, and perform weighting processing on the input feature map and the nonlinearly transformed input feature map to obtain Output feature map;
  • a second size conversion module 405, configured to perform size conversion on the output feature map to obtain a residual map
  • the super-division module 406 is configured to combine the low-resolution image and the residual map to obtain a high-resolution image.
  • the device 400 may further include:
  • a weight obtaining module configured to obtain a set of weight values according to the input feature map; the set of weight values is used to perform weighting processing on the input feature map and the nonlinearly transformed input feature map.
  • the weight obtaining module may be configured to perform a convolution operation on the input feature map and obtain a convolution result, and process the convolution result according to an S-type function to obtain a weight value T, where The weight value T is used by the highway module to perform weighting processing on the input feature map and the nonlinearly transformed input feature map.
  • the weight acquisition module can be included in the above-mentioned highway module 404 as an implementation carrier of the weight acquisition branch, that is, the weight acquisition module can be a sub-module of the highway module 404 for the highway module 404. Get and output the weight value T.
  • the details of the related processing are the same as those of the first embodiment, and are not described here.
  • the road module 404 is specifically configured to:
  • the process is another branch of the above description of the road network (for the non-intrusion of the input signal according to at least one hidden layer (for example, the hidden layer is n layer)
  • the processing procedure, the specific processing manner and the technical effect of the linear transformation processing are the same as those of the first embodiment or the second embodiment, or the corresponding processing manners on the basis of the possible implementation manners, and are not described herein again.
  • the super-segment module 406 can be specifically used to:
  • Pixel values of each pixel in the low resolution image are added to pixel values of corresponding pixels in the residual map to obtain pixel values of corresponding pixels in the high resolution image.
  • the super-segment module 406 can be specifically used to:
  • a pixel value of each pixel in the low resolution image and a pixel value of a corresponding pixel in the residual map are weighted and summed to obtain a pixel value of a corresponding pixel in the high resolution image.
  • Embodiment 5 is a diagrammatic representation of Embodiment 5:
  • FIG. 5 is a schematic structural diagram of an image over-score device 500 according to an embodiment of the present application.
  • the image super-segment device includes but is not limited to the following modules 501-506:
  • the receiving module 501 is configured to receive a low resolution image.
  • a pre-processing module 502 configured to perform pre-processing on the low-resolution image to obtain a vertical gradient map, a horizontal gradient map, and a luminance map; and the vertical gradient map, the horizontal gradient map, and the luminance map are respectively used as Three different dimensions of information to form a feature map to be entered.
  • the first size conversion module 503 is configured to perform size conversion on the to-be-input feature map to obtain an input feature map.
  • the highway module 504 is configured to perform weighting processing on the input feature map according to the weighting process flow in series to obtain an output feature map, where the serial weighting process includes n road units, and the n road units are Connected in series; the series weighting process includes n sets of weight values, and the n sets of weight values are used for the n road units and are in one-to-one correspondence with the n road units, where n is greater than Or a positive integer equal to 2.
  • n is greater than Or a positive integer equal to 2.
  • more possible road units are omitted between the road unit 2 and the road unit n for the sake of brevity.
  • the second size conversion module 505 is configured to perform size conversion on the output feature map to obtain a residual map.
  • the super-division module 506 is configured to combine the low-resolution image and the residual image to obtain a high-resolution image.
  • the n is equal to 3. That is, the highway module 504 includes three highway units connected in series.
  • the first road unit is specifically configured to: perform an upper channel processing and a lower channel processing on the input feature maps of the first highway unit, where the upper channel processing includes: according to the An input characteristic map of a highway unit obtains a set of weight values of the first road unit; the lower channel processing includes a nonlinear transformation process, and the nonlinear transformation result is obtained through the lower channel processing; according to the first road The input feature map of the first road unit and the non-linear transformation result are weighted by a set of weight values corresponding to the unit to obtain an output feature map of the first road unit; wherein the first road The unit is any one of the n road units; wherein, when the first road unit is the first one of the series of weighting processing flows, the input characteristics of the first road unit The figure is the input feature map; when the first road unit is the last link in the series weighting process flow, the first road unit loses Wherein the output characteristic graph of FIG.
  • the upper channel acquisition weight value may be understood as the branch for obtaining the weight value T among the three branches of the road network in step 205.
  • the specific operations of the branch may include:
  • the lower channel processing may be understood as the non-linearity of the input signal according to at least one hidden layer (for example, the hidden layer is n layer) in the three branches of the road network in step 205.
  • the specific operations of the branch may include:
  • this implementation of the embodiment of the present application first performs a de-fitting operation on the input feature map, thereby reducing the parameter amount at the front end, in the process of further performing the convolution operation on the de-fitting result, due to the input of the convolution operation
  • the parameter is reduced, so that the calculation amount of the convolution operation is reduced, the over-consumption time can be reduced, and the speed of image over-score can be improved.
  • one of the road units i is specifically used for:
  • the apparatus further includes a weight acquisition module, configured to obtain a set of weight values of the highway unit i according to the input feature map of the highway unit i; the highway unit A set of weight values of i is used to weight the input feature map of the highway unit i and the input feature map of the nonlinearly transformed highway unit i.
  • a weight acquisition module configured to obtain a set of weight values of the highway unit i according to the input feature map of the highway unit i; the highway unit A set of weight values of i is used to weight the input feature map of the highway unit i and the input feature map of the nonlinearly transformed highway unit i.
  • the weight obtaining module is specifically configured to: perform a convolution operation on the input feature map of the highway unit i and obtain a convolution result, and process the convolution result according to an S-type function to obtain a A set of weight values for the highway unit i.
  • the super-segment module 506 can be specifically configured to:
  • the color of the resolution image is the color of the high resolution image.
  • the color of the low resolution image is used as the color of the high resolution image.
  • FIG. 1 When FIG. 1 is used to perform image over-score provided by the embodiment of the present application, the memory 103 can be used to store the super-segment model described above.
  • the communication module 107 can be used to communicate with other devices, for example, can be used to receive low resolution images that need to be reconstructed.
  • the processor 102 may be one or more central processing units (CPUs).
  • the processor 102 may be a single-core CPU or a multi-core CPU; the processor 102 may also A neural network processor, such as the GPU, NPU or TPU described above, is used for high-density parallelizable massive operations in neural networks.
  • the processor 102 is configured to read the program code stored in the memory 103 and perform the following operations:
  • the vertical gradient map, the horizontal gradient map, and the luminance map are respectively taken as information of three different dimensions to constitute a feature map to be input;
  • the processor 102 before performing weighting on the input feature map and the non-linearly transformed input feature map, the processor 102 further performs:
  • the processor 102 performs nonlinear transformation processing on the input feature map to obtain a nonlinearly transformed input feature map, including:
  • the processor 102 combines the low resolution image and the residual image to obtain a high resolution image, including:
  • Pixel values of each pixel in the low resolution image are added to pixel values of corresponding pixels in the residual map to obtain pixel values of corresponding pixels in the high resolution image.
  • a pixel value of each pixel in the low resolution image and a pixel value of a corresponding pixel in the residual map are weighted and summed to obtain a pixel value of a corresponding pixel in the high resolution image.
  • the above image super-sub-device may be an image recognition device such as a face recognition device.
  • the image super-sub-device described above may also be an image enhancement device or the like.
  • the processor 102 in the image super-sub-device is used to read the program code stored in the memory 103, and performs the following operations:
  • the vertical gradient map, the horizontal gradient map, and the luminance map are respectively used as information of three different dimensions to form a feature map to be input;
  • the tandem weighting process includes n sets of weight values, and the n sets of weight values are respectively used for the n processing links and are in one-to-one correspondence with the n processing links, where n is a positive integer greater than or equal to 2. ;
  • the processor 102 is configured to: perform an upper channel processing and a lower channel processing on the input feature maps of the current processing link in a current processing step of the n processing links.
  • the processing of the upper channel includes: obtaining a set of weight values of the current processing link according to the input feature map of the current processing link; the processing of the lower channel includes nonlinear transformation processing, and obtaining nonlinearity after processing by the lower channel Transforming the result; performing weighting processing on the input feature map of the current processing link and the nonlinear transformation result according to a set of weight values corresponding to the current processing link, to obtain an output feature map of the current processing link;
  • the current processing link is any one of the n processing links; wherein, when the current processing link is the first link in the serialized weighting processing flow, the current processing link
  • the input feature map is the input feature map; when the current processing link is the last one of the series of weighting processing flows When the section of the currently processed part of the output characteristic graph of the output characteristics of FIG.
  • n is equal to 3.
  • the processor 102 is specifically configured to: perform a convolution operation on an input feature map of the current processing link, obtain an upper channel convolution result, and compare the upper channel volume according to an S-type function.
  • the product result is processed to obtain a set of weight values for the current processing link.
  • the processor 102 is specifically configured to: perform de-fitting on an input feature map of the current processing link, obtain a de-fitting result, and perform a convolution operation on the de-fitting result.
  • the nonlinear transformation result is obtained.
  • the processor 102 is specifically configured to:
  • Pixel values of each pixel in the low resolution image are added to pixel values of corresponding pixels in the residual map to obtain pixel values of corresponding pixels in the high resolution image.
  • a pixel value of each pixel in the low resolution image and a pixel value of a corresponding pixel in the residual map are weighted and summed to obtain a pixel value of a corresponding pixel in the high resolution image.
  • the above image super-sub-device may be an image recognition device such as a face recognition device.
  • the image super-sub-device described above may also be an image enhancement device or the like.
  • the embodiment of the invention further provides a chip system, the chip system comprising at least one processing module, a memory and an interface circuit, the memory, the interface circuit and the at least one processing module being interconnected by a line, the at least one memory
  • An instruction is stored therein; when the instruction is executed by the processing module, the method flow shown in FIG. 2 or FIG. 3 is implemented.
  • the processing module includes at least a central processing unit, and may further include a central processing unit and a neural network processor.
  • the central processing unit may be one or more, and the neural network processor may be one or more. And the type of the neural network processor is not limited.
  • the processing module includes a central processing unit and a neural network processor, when the instructions are executed by the processing module, steps 205 and 207 in FIG.
  • steps 305 and 307 in FIG. 3 are implemented according to the operation of the neural network processor in the processing module, and other steps may be implemented by the central processing unit or by the neural network processor. Of course, the above steps are implemented. 205, 207; 305, 307 can also be implemented by a central processing unit, which is not limited herein.
  • FIG. 6 is a schematic structural diagram of another electronic device 600 for neural network calculation, including an IO (Input and Output) port 610, a storage medium 620, and a PLD (Programmable Logic Device). ) 630.
  • the electronic device 600 reads configuration data output by other devices through the IO port 610.
  • the electronic device 600 reads the laid-out design netlist from the PC through a JTAG (Joint Test Action Group) interface.
  • the storage medium 620 can include RAM or ROM, or a combination of both, for storing the read configuration data; the PLD 630 can be an FPGA or other programmable logic device.
  • the storage medium 620 and the PLD 630 can perform bidirectional data transmission.
  • the PLD 630 reads the configuration data from the storage medium 620, and configures a plurality of internal logic blocks according to the configuration data, for example, configuring a connection manner of the plurality of lookup tables therein to form a neural network in the processor 102.
  • the storage medium 620 can also store data for calculations, such as input data and input parameters, and store the calculation results of the PLD 630.
  • the embodiment of the invention further provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when it runs on the processor, the method flow shown in FIG. 2 or FIG. 3 is implemented.
  • the embodiment of the invention further provides a computer program product, wherein the method flow shown in FIG. 2 or FIG. 3 is implemented when the computer program product runs on a processor.
  • the computer program product includes one or more computer instructions.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in or transmitted by a computer readable storage medium.
  • the computer instructions can be from a website site, computer, server or data center to another website site by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) Transfer from a computer, server, or data center.
  • the computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a Solid State Disk (SSD)) or the like.
  • the program can be stored in a computer readable storage medium, when the program is executed
  • the flow of the method embodiments as described above may be included.
  • the foregoing storage medium includes various media that can store program codes, such as a ROM or a random access memory RAM, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种图像超分方法及设备，该方法包括：接收低分辨率图像（S101）；对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图（S102）；将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图（S103）；对所述待输入特征图进行尺寸转换得到输入特征图（S104）；对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图（S105）；对所述输出特征图进行尺寸转换得到残差图（S106）；将所述低分辨率图像和所述残差图结合得到高分辨率图像（S107）。可以应用在人工智能中的计算机视觉技术领域，以提高超分得到的高分辨率图像的质量。

Description

图像超分方法及装置 技术领域
本发明实施例涉及通信技术领域，尤其涉及一种图像超分方法及装置。
背景技术
图像超分辨率重建，简称图像超分辨率(super resolution)，也称图像超分(SR)，是指利用图像处理方法将低分辨率图像(low resolution,LR)重建为高分辨率图像(high resolution，HR)的一种图像处理技术，可有效提升图像的清晰度，对于视频监控、卫星图像、相机拍照、高清电视、医学图像等各领域具有重要意义。例如，在视频监控领域，高清视频为道路交通监管提供了更为有利的支持，同时也有效提高了社会安全保障水平，然而在实际情况下许多应用出于各种考虑没办法直接获取高分辨率图像或视频，而图像超分技术可有效改善该问题。
目前已经提出了许多图像超分方法，早期的图像超分方法是基于插值的，如双三次插值和兰索斯重采样，由于每个像素从LR图像到HR图像的映射上有许多解，并且这类方法仅使用LR图像的信息，因此不能很好地重构出HR图像。近年来，随着CNN(Convolutional Neural Networks，卷积神经网络)在计算机视觉领域的应用，出现了许多基于CNN的超分方法，例如VDSR(Very Deep Convolutional Networks Super Resolution，很深的超分卷积网络)。其采用的神经网络结构中具有多个直连网络分支，即某一层神经网络的输出直接跳过若干层神经网络直连到处理后的信号中，此直连网络分支无需经过中间卷积等操作，这使得输出层在损失函数计算，回传梯度的时候，可以将梯度通过直连分支直接传递到若干层之前，从而减少了深度神经网络在训练过程中的梯度消失问题，提高了深度神经网络的收敛性。
为了获得更好的超分效果，VDSR通过加深网络深度获得更好的超分性能，但这带来了参数量大，超分耗时的缺点。并且，虽然VDSR基于多个直连网络分支的神经网络结构能在一定程度上解决收敛的问题，但随着网络中卷积层数的增加，到一定数量后仍然存在收敛问题，从而影响超分效果，降低图像超分质量。
发明内容
本申请实施例公开了一种图像超分方法及设备，可以提高图像超分质量，获得更好的图像超分效果。
第一方面，本申请实施例提供一种图像超分方法，包括：接收低分辨率图像；对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；对所述待输入特征图进行尺寸转换得到输入特征图；对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图；对所述输出特征图进行尺寸转换得到残差图；将所述低分辨率图像和所述残差图结合得到高分辨率图像。
值得说明的是，第一方面提供的本申请实施例是基于深度神经网络的图像超分方法，所述深度神经网络包括输入层，至少一层隐含层，输出层；所述输入层可以用于接收低分辨率图像；所述至少一层隐含层可以用于对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；并将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不 同维度的信息以构成待输入特征图；并对所述待输入特征图进行尺寸转换得到输入特征图；并对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，并对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图；并对所述输出特征图进行尺寸转换得到残差图；并将所述低分辨率图像和所述残差图结合得到高分辨率图像；所述输出层可以用于输出所述高分辨率图像。
在本申请实施例中，对所述输入特征图以及所述非线性变换后的输入特征图采用加权处理得到输出特征图；由于该加权处理的对象之一：“所述输入特征图”，是跳过深度神经网络中的若干隐含层，即未经处理直接由上一个步骤的输出得到的，即根据深度神经网络中的直连分支直接得到的，该直连分支能很好的克服深度神经网络的收敛性问题，从而提升基于该深度神经网络的图像超分效果，并且，由于此处采用的加权处理方式使所述输入特征图和所述非线性变换后的输入特征图按相应的比例取值并相加，从而对最后重建得到的高分辨率图像的图像质量有较大提升。进一步的，由于本申请实施例将所述低分辨率图像和所述残差图结合得到高分辨率图像，即其中的低分辨率图像是由直连分支直接得到的，使得本申请实施例进一步克服了深度神经网络的收敛性问题，最终得到更好的图像超分效果。
在一种可能的实现方式中，在所述对所述输入特征图以及非线性变换后的输入特征图进行加权处理之前，所述方法还包括：
根据所述输入特征图得到一组权重值；所述一组权重值用于对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理。
在一种可能的实现方式中，根据所述输入特征图得到所述一组权重值，包括：
对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T，所述权重值T用于对所述输入特征图以及非线性变换后的输入特征图进行加权处理。
在一种可能的实现方式中，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图，包括：
将所述水平梯度图作为第一维信息，所述垂直梯度图作为第二维信息，所述亮度图作为第三维信息，所述第一维信息和所述第二维信息以及所述第三维信息构成所述待输入特征图。
在一种可能的实现方式中，所述对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，包括：
对所述输入特征图进行去拟合操作，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换后的输入特征图。
这种可能的实现方式通过对输入特征图先进行去拟合操作，从而在前端减少参数量，在进一步对去拟合结果进行卷积操作的过程中，由于卷积操作的输入参数减少，使得卷积操作的计算量降低，可以减少超分耗时，提高图像超分的速度。
在一种可能的实现方式中，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
在一种可能的实现方式中，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行 加权求和，得到所述高分辨率图像中的相应像素的像素值。
由于加权求和采用的系数是根据训练好的深度神经网络得到的，通过训练好的深度神经网络是可以使得输出的高分辨率图像具有最佳的图像质量的，因此，这种可能的实现方式可以使图像超分输出的高分辨率图像具有更好的质量。
在一种可能的实现方式中，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，包括：根据权重信息对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，所述权重信息是通过训练好的深度神经网络的得到的；其中所述权重信息包括一个权重值或者一组权重值；所述一组权重值可以用一个权重图或一个权重向量来表示。
在一种可能的实现方式中，所述根据权重信息对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，包括：
当所述权重值为一个时，利用所述权重值对所述输入特征图中的每个第一像素值和所述非线性变换后的输入特征图中的每个第二像素值进行加权求和得到所述输出特征图的第三像素值，所述第一像素值为所述输入特征图中任意一个像素值，所述第二像素值与所述第一像素值为一一对应的关系，所述第三像素值与所述第一像素值为一一对应的关系。
由于该权重信息是根据训练好的深度神经网络得到的，针对所述输入特征图和所述非线性变换后的输入特征图，能赋予其不同的权重值；各个权重值的配置能使得所述输出特征图具有更优秀的表达能力，使最终输出的高分辨率图像具有更好的图像质量。
当所述权重值为一组时，利用所述一组权重值中的各权重值分别对所述输入特征图中的每个第一像素值和所述非线性变换后的输入特征图中的每个第二像素值进行加权求和得到所述输出特征图的第三像素值，所述第一像素值为所述输入特征图中任意一个像素值，所述第二像素值与所述第一像素值为一一对应的关系，所述第三像素值与所述第一像素值为一一对应的关系。
由于该权重信息是通过训练好的深度神经网络得到的，通过训练好的深度神经网络是可以使得输出的高分辨率图像具有最佳的图像质量的，因此，通过该训练好的深度神经网络得到的各个权重值能使得所述输出特征图具有更优秀的表达能力，更进一步的，当所述权重值为一组时，针对所述输入特征图中不同的局部特征可以更有针对性的赋予其不同的权重值，使得输出特征图具有更多种可能性，从而使所述的图像超分方法得到更高质量的高分辨率图。
第二方面，本申请实施例提供一种图像超分方法，包括：
接收低分辨率图像；对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；对所述待输入特征图进行尺寸转换得到输入特征图；根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个处理环节，所述n个处理环节以串联方式连接；所述串联的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个处理环节且与所述n个处理环节一一对应，其中n为大于或等于2的正整数；对所述输出特征图进行尺寸转换得到残差图；将所述低分辨率图像和所述残差图结合得到高分辨率图像。
值得说明的是，第一方面提供的本申请实施例是基于深度神经网络的图像超分方法，所述深度神经网络包括输入层，至少一层隐含层，输出层；所述输入层可以用于接收低分辨率图像；所述至少一层隐含层可以用于对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；并将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不 同维度的信息以构成待输入特征图；并对所述待输入特征图进行尺寸转换得到输入特征图；并根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个处理环节，所述n个处理环节以串联方式连接；所述串联的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个处理环节且与所述n个处理环节一一对应，其中n为大于或等于2的正整数；并对所述输出特征图进行尺寸转换得到残差图；将所述低分辨率图像和所述残差图结合得到高分辨率图像；所述输出层可以用于输出所述高分辨率图像。
在一种可能的实现方式中，根据串联的加权处理流程对所述输入特征图进行加权处理得到输出特征图，且所述串联的加权处理流程中包含n个处理环节，所述n个处理环节以串联方式连接。本申请实施例与第一方面提供的图像超分方法，关键区别在于：第一方面对所述输入特征图和非线性变换后的输入特征图进行一次加权处理即输出所述输出特征图，给下一步骤进行处理，在第二方面提供的图像超分方法中，串联的加权处理流程中包含n个处理环节，其中每个处理环节均有一轮加权处理的操作，即第一个处理环节的输出不直接作为所述输出特征图输出，而是作为第二个处理环节的输入，供第二个处理环节进行再一次的加权处理，直到该串联的加权处理流程中的最后一个处理环节完成加权处理，将其输出作为所述输出特征图，供下一步骤对该输出特征图进行进一步处理。该串联的加权处理流程相对第一方面的单轮加权处理而言，可以使重建得到的高分辨率图像质量比第一方面提供的方法得到的高分辨率图像质量更加，即图像超分效果更好。
在一种可能的实现方式中，所述根据串联的加权处理流程对所述输入特征图进行加权处理，得到所述输出特征图，包括：
在所述n个处理环节中的一个处理环节i的处理过程包括：将所述处理环节i的输入特征图分别进行上通道处理和下通道处理，所述上通道处理包括：根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值；所述下通道处理包括对所述处理环节i的输入特征图进行非线性变换处理，经过所述下通道处理得到非线性变换后的所述处理环节i的输入特征图；根据所述处理环节i的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理，得到所述处理环节i的输出特征图；其中，所述处理环节i为所述n个处理环节中的任意一个环节；其中，当所述处理环节i为所述串联的加权处理流程中的第一个环节时，所述处理环节i的输入特征图为所述输入特征图；当所述处理环节i为所述串联的加权处理流程中的最后一个环节时，所述处理环节i的输出特征图为所述输出特征图。
在一种可能的实现方式中，所述上通道处理包括：对所述当前处理环节的输入特征图进行卷积操作并得到上通道卷积结果，并根据S型函数对所述上通道卷积结果进行处理，得到所述当前处理环节的一组权重值。
在一种可能的实现方式中，所述下通道处理具体包括：对所述当前处理环节的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换结果。
在一种可能的实现方式中，所述根据串联的加权处理流程对所述输入特征图进行加权处理，得到所述输出特征图，包括：
在所述n个处理环节中的一个处理环节i的处理过程包括：对所述处理环节i的输入特征图进行非线性变换处理，得到非线性变换后的处理环节i的输入特征图；根据与所述处理环 节i一一对应的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理，得到所述处理环节i的输出特征图；其中，所述处理环节i为所述n个处理环节中的任意一个环节,1≤i≤n；其中，当所述处理环节i为所述串联的加权处理流程中的第一个环节时，所述处理环节i的输入特征图为所述输入特征图；当所述处理环节i为所述串联的加权处理流程中的最后一个环节时，所述处理环节i的输出特征图为所述输出特征图。
在一种可能的实现方式中，在所述根据与所述处理环节i一一对应的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理之前，所述方法还包括：
根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值；所述处理环节i的一组权重值用于对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理。
在一种可能的实现方式中，所述根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值，包括：
对所述处理环节i的输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到所述处理环节i的一组权重值。
在一种可能的实现方式中，所述对所述处理环节i的输入特征图进行非线性变换处理，得到非线性变换后的所述处理环节i的输入特征图包括：
对所述处理环节i的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到非线性变换后的所述处理环节i的输入特征图。
在一种可能的实现方式中，所述n等于3。
在一种可能的实现方式中，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
在一种可能的实现方式中，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
由于加权求和采用的系数是通过训练好的深度神经网络得到的(深度神经网络的训练过程主要是对各层神经网络的连接参数的训练过程，本申请实施例将在实施例开始之前对此进行简单介绍)，通过训练好的深度神经网络得到的权重系数可以使得输出的高分辨率图像具有最佳的图像质量的，因此，通过这种可能的实现方式可以使图像超分输出的高分辨率图像具有更好的质量。
在一种可能的实现方式中，所述n组权重值中的每一组权重值中，包括至少一个权重值(其中，至少一个包含一个和至少两个的情况)，其中当一组权重值中包含一个权重值时，该组权重值对应的处理环节根据该一个权重值进行该环节的加权处理；当一组权重值中包含至少两个权重值，该组权重值对应的处理环节根据该至少两个权重值进行该环节的加权处理，具体的，可以对输入该环节的输入特征图中的不同局部赋予不同的权重值，使该至少两个权重值对应的处理环节输出的特征图更加丰富，也使得这个环节输出的特征图具有更多种可能 性，从而使所述的图像超分方法得到更高质量的高分辨率图。
第三方面，提供了一种图像超分装置，所述装置包括用于执行所述第一方面或者第一方面的任一可能的实现方式中的方法的模块。
第四方面，提供了一种图像超分装置，所述装置包括用于执行所述第二方面或者第二方面的任一可能的实现方式中的方法的模块。
第五方面，提供一种图像超分装置，所述装置包括存储介质和中央处理器，所述存储介质可以是非易失性存储介质，所述存储介质中存储有计算机可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述计算机可执行程序以实现所述第一方面或者第一方面的任一可能的实现方式中的方法。
第六方面，提供一种图像超分装置，所述装置包括存储介质和中央处理器，所述存储介质可以是非易失性存储介质，所述存储介质中存储有计算机可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述计算机可执行程序以实现所述第二方面或者第二方面的任一可能的实现方式中的方法。
第七方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行第一方面或第一方面的任一可能的实现方式中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面或第一方面的任一可能的实现方式中的方法。
第八方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行第二方面或第二方面的任一可能的实现方式中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第二方面或第二方面的任一可能的实现方式中的方法。
第九方面，提供一种计算机可读存储介质，所述计算机可读介质存储用于设备执行的程序代码，所述程序代码包括用于执行第一方面或者第一方面的任一可能的实现方式中的方法的指令。
第十方面，提供一种计算机可读存储介质，所述计算机可读介质存储用于设备执行的程序代码，所述程序代码包括用于执行第二方面或者第二方面的任一可能的实现方式中的方法的指令。
附图说明
下面对本申请实施例用到的附图进行介绍。
图1是本申请实施例提供用于神经网络计算的电子设备100结构示意图；
图2是本申请实施例提供的一种图像超分方法的流程示意图；
图2a是本申请实施例提供的一种公路单元中的结构示意图；
图2b是本申请实施例提供的一种公路单元中的结构示意图；
图3是本申请实施例提供的一种图像超分方法的流程示意图；
图4是本申请实施例提供的一种图像超分装置的结构示意图；
图5是本申请实施例提供的另一种图像超分装置的结构示意图；
图6是本申请实施例提供的另一种用于神经网络计算的电子设备的结构示意图。
具体实施方式
首先，为了便于理解本申请实施例，对本申请实施例涉及的一些概念或术语进行解释。
(1)图像超分辨率重建(简称图像超分辨率，又称图像超分)
图像超分辨率(Image Super Resolution)是指：利用图像处理方法通过计算机将低分辨率(low resolution，LR)图像恢复出高分辨率(high resolution，HR)图像的技术。高分辨率图像即意味着图像具有高像素密度，可以提供更多的图像细节，这些细节往往在应用中起到关键作用。
图像超分辨率的方法可以分为两类：第一类是传统的图像超分辨率方法，例如利用数据运算实现LR到HR的恢复；第二类是基于学习的图像超分辨率方法，即利用机器学习的方法实现LR到HR的恢复。其中，基于重建的图像超分辨率方法可以通过频域算法或者空域算法来统计估算最大后验概率的高分辨率图像。基于学习的图像超分辨率方法可以包含训练阶段和测试阶段两个阶段。
在训练阶段，首先建立初始的超分辨率模型和训练集。训练集中可以包括多张低分辨率图像以及每张低分辨率图像对应的高分辨率图像。通过训练集中的低分辨率图像以及它对应的高分辨率图像，学习高分辨率图像和低分辨率图像之间的对应关系，进而修正初始的超分辨率模型中的参数的取值，来使高分辨率图像和重建图像之间误差收敛，最终确定训练后得到的新建的超分辨率模型。在测试阶段，可以根据新建的超分辨率模型来指导图像的超分辨率重建。
其中，获取训练集中图像对(低分辨率图像以及该低分辨率图像对应的高分辨率图像形成一个图像对)的方法可以是，先拿到一个高分辨率图像，然后将该高分辨率图像通过模糊函数处理以得到与之对应的低分辨率图像，其中，该模糊函数是不可逆的。
其中，初始的超分辨率模型可以是根据实验确定的模型，可以是非线性的。超分辨率模型可以是一种卷积神经网络。
(2)卷积神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2019081445-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入。激活函数可以是sigmoid函数。神经网络是将许多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(3)深度神经网络
深度神经网络(Deep Neural Network，DNN)，可以理解为具有很多层隐含层的神经网络，这里的“很多”并没有特别的度量标准，我们常说的多层神经网络和深度神经网络其本质上 是同一个东西。从DNN按不同层的位置划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2019081445-appb-000002
其中，
Figure PCTCN2019081445-appb-000003
是输入向量，
Figure PCTCN2019081445-appb-000004
是输出向量，
Figure PCTCN2019081445-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2019081445-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2019081445-appb-000007
由于DNN层数多，则系数W和偏移向量
Figure PCTCN2019081445-appb-000008
的数量也就是很多了。那么，具体的参数在DNN是如何定义的呢？首先我们来看看系数W的定义。以一个三层的DNN为例，如：第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2019081445-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。总结下，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2019081445-appb-000010
注意，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。
(4)卷积神经网络(Convosutionas Neuras Network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器。该特征抽取器可以看作是滤波器，卷积过程可以看作是使用一个可训练的滤波器与一个输入的图像或者卷积特征平面(feature map)做卷积。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。这其中隐含的原理是：图像的某一部分的统计信息与其他部分是一样的。即意味着在某一部分学习的图像信息也能用在另一部分上。所以对于图像上的所有位置，我们都能使用同样的学习得到的图像信息。在同一卷积层中，可以使用多个卷积核来提取不同的图像信息，一般地，卷积核数量越多，卷积操作反映的图像信息越丰富。
卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。(5)反向传播算法
卷积神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的超分辨率模型中参数的大小，使得超分辨率模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的超分辨率模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的超分辨率模型的参数，例如权重矩阵。
(6)像素值和图像特征
图像的像素值可以是一个红绿蓝(RGB)颜色值，像素值可以是表示颜色的长整数。例如，像素值为256*Red+100*Green+76Blue，其中，Blue代表蓝色分量，Green代表绿色分量，Red代表红色分量。各个颜色分量中，数值越小，亮度越低，数值越大，亮度越高。对于灰度图像来说，像素值可以是灰度值。
图像特征主要有图像的颜色特征、纹理特征、形状特征和空间关系特征等。
颜色特征是一种全局特征，描述了图像或图像区域所对应的景物的表面性质；一般颜色 特征是基于像素点的特征，此时所有属于图像或图像区域的像素都有各自的贡献。由于颜色对图像或图像区域的方向、大小等变化不敏感，所以颜色特征不能很好地捕捉图像中对象的局部特征。
纹理特征也是一种全局特征，它也描述了图像或图像区域所对应景物的表面性质；但由于纹理只是一种物体表面的特性，并不能完全反映出物体的本质属性，所以仅仅利用纹理特征是无法获得高层次图像内容的。与颜色特征不同，纹理特征不是基于像素点的特征，它需要在包含多个像素点的区域中进行统计计算。
形状特征有两类表示方法，一类是轮廓特征，另一类是区域特征，图像的轮廓特征主要针对物体的外边界，而图像的区域特征则关系到整个形状区域；
空间关系特征，是指图像中分割出来的多个目标之间的相互的空间位置或相对方向关系，这些关系也可分为连接/邻接关系、交叠/重叠关系和包含/包容关系等。通常空间位置信息可以分为两类：相对空间位置信息和绝对空间位置信息。前一种关系强调的是目标之间的相对情况，如上下左右关系等，后一种关系强调的是目标之间的距离大小以及方位。
需要说明的，上述列举的图像特征可以作为图像中具有的特征的一些举例，图像还可以具有其他特征，如更高层级的特征：语义特征，此处不再展开。
本申请实施例中即将提到的垂直梯度图、水平梯度图、亮度图与上述各图像特征的关系，具体可以是：本申请实施例中的垂直梯度图和水平梯度图描述的是相邻像素之间的变换，横向变换由水平梯度图表示，纵向变换由垂直梯度图表示，它们中均包含了纹理特征和形状特征。而亮度图则包含了颜色特征。可以认为本申请实施例中的垂直梯度图、水平梯度图和亮度图是上述多种图像特征中某些特征的综合体现。
在进行图像超分时，为了取的较好的图像超分效果，可以采用比较深的深度神经网络，即较多层的深度神经网络。但是深度神经网络的层数越多，其对应的参数量也越大，并导致更大的计算量，使得超分耗时较多；并且，当深度神经网络的深度达到一定程度时，必然导致收敛困难的问题。
为提高图像超分过程中经过图像超分处理得到的HR图像的质量，本申请实施例提供一种图像超分方法，采用公路网络(Highway Networks)和残差网络相结合深度神经网络结构对低分辨率图像进行重构得到高分辨率图像。
其中，需要说明的是，公路网络和残差网络可以认为是特殊的深度神经网络，具体的，可以理解为是深度神经网络中多个隐含层之间的特殊连接方式。本申请实施例的图像超分方法中的部分步骤或全部步骤可以理解为建立在这个由公路网络和残差网络构建成的特殊深度神经网络结构之上，由该特殊的深度神经网络实现。本申请实施例提供的图像超分方法可以理解为是基于训练好的深度神经网络进行的,其中该训练好的深度神经网络中可以包括残差网络和公路网络,在本申请实施例中使用的残差网络和公路网络可以被理解为经训练得到的,也可以称之为训练好的神经网络,为了简洁起见,后文不再赘述。
简单来说，残差网络可以是：深度神经网络中多个隐含层之间除了逐层相连之外，例如第1层隐含层连接第2层隐含层，第2层隐含层连接第3层隐含层，第3层隐含层连接第4层隐含层(这是一条神经网络的数据运算通路，也可以形象的称为神经网络传输)，残差网络还多了一条直连支路，这条直连支路从第1层隐含层直接连到第4层隐含层，即跳过第2层和第3层隐含层的处理，将第1层隐含层的数据直接传输给第4层隐含层进行运算。公路网络可以是：深度神经网络中除了有上面所述的运算通路和直连分支之外，还包括一条权重获 取分支，这条支路引入传输门(transform gate)进行权重值的获取，并输出权重值T供上面的运算通路和直连分支后续的运算使用。
在本申请实施例中，利用权重获取支路得到的权重值T,对通过直连分支得到的输入特征图和经过逐层相连的运算通路对输入特征图进行处理得到的经处理的输入特征图进行加权处理，得到输出特征图。通过该处理过程，不仅可以更好的克服深度神经网络的梯度消失问题使深度神经网络的收敛性更好，且通过加权处理得到的输出特征图能更好地描述出图像超分中高分辨率图像所需要的信息，并进一步使输出的高分辨率图像具有更好地图像质量；进一步的，本申请实施例采用残差网络的思想将输入特征图通过直连分支直接跳过若干隐含层到某一层，并在该层与经处理得到的残差图进行结合得到高分辨率图像，这种直连分支的设计在上述提升收敛性效果的基础上进一步提升了收敛效果，使得本申请实施例中的深度神经网络有更优秀的收敛能力，从而为基于该深度神经网络的图像超分方法带来更高的图像超分效率及更好的图像超分效果。
在介绍完本申请实施例涉及的一些概念或术语之后，其次，需要指出的是，根据本申请实施例提供的图像超分方法和装置，可以理解的是：本申请实施例提供的图像超分方法是基于深度神经网络进行的，具体的，可以是根据卷积神经网络进行的。根据上文的概念介绍可以知道，卷积神经网络可以理解为一种特殊的深度神经网络，即卷积神经网络(CNN)是一种带有卷积结构的深度神经网络。该卷积神经网络可以是预先训练好的，由于该卷积神经网络在本申请实施例中用于进行图像超分，因此后文也将该用于图像超分的卷积神经网络称为超分网络。
下面简单介绍本申请实施例的超分网络的训练过程。
在这之前，需要先对深度神经网络的训练过程有一个简单的了解：
首先，深度神经网络中的每一层的工作可以用数学表达式
Figure PCTCN2019081445-appb-000011
来描述：从物理层面深度神经网络中的每一层的工作可以理解为通过五种对输入空间(输入向量的集合)的操作，完成输入空间到输出空间的变换(即矩阵的行空间到列空间)，这五种操作包括：1、升维/降维；2、放大/缩小；3、旋转；4、平移；5、“弯曲”。其中1、2、3的操作由
Figure PCTCN2019081445-appb-000012
完成，4的操作由
Figure PCTCN2019081445-appb-000013
完成，5的操作则由α()来实现。这里之所以用“空间”二字来表述是因为被分类的对象并不是单个事物，而是一类事物，空间是指这类事物所有个体的集合。其中，W是权重向量，该向量中的每一个值表示该层神经网络中的一个神经元的权重值。该向量W决定着上文所述的输入空间到输出空间的空间变换，即每一层的权重W控制着如何变换空间。训练深度神经网络的目的，也就是最终得到训练好的神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。因此，神经网络的训练过程本质上就是学习控制空间变换的方式，更具体的就是学习权重矩阵。
接着，具体介绍深度神经网络的训练过程：
因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断的调整，直到神经网络能够预测出真正想要的目标值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss) 越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
回到本申请实施例的超分网络的训练过程，首先，系统会为该超分网络配置参数，该参数为初始化的参数，无需通过学习得到。该系统配置的参数本质上为初始化的矩阵向量，用于配置网络的结构和神经网络各层之间的连接关系。接着，需要构建训练数据集，这里的训练数据集包括若干对低分辨率图像和高分辨率图像，即每对中均包含一个低分辨率图像和对应该低分辨率图像的高分辨率图像。将成对的低分辨率图像和高分辨率图像输入上述配置好的超分网络，通过超分网络中各层的权重向量进行计算，输出重建的高分辨率图像。将输出的高分辨率图像与训练数据集包括的高分辨率图像计算损失函数，根据神经网络的反向传播算法进行计算，更新该超分网络中需要学习的各个层的参数，得到更新后的各个层的参数，即权重向量W。然后根据该更新后的各个层的权重向量W构成的权重矩阵(由于矩阵中各个权重向量已经更新，因此网络中各层的连接关系随着权重向量的更新而更新了)对训练数据集包含的若干对低分辨率图像和高分辨率图像继续进行训练，直到该超分网络输出的重建的高分辨率图像与训练数据集包括的高分辨率图像之间的差异，或者说损失函数的loss满足预设条件，则输出训练好的权重矩阵，完成训练。
训练好的超分网络可以用于执行本申请实施例的图像超分方法，这边涉及到超分网络的应用过程，即将低分辨率图像输入该训练好的超分网络即可得到高分辨率图像。后面将对这个过程进行详细的展开，此处不再对此介绍。
另外，根据该训练好的超分网络进行图像超分得到的高分辨率图像，可以根据图像评价方法(例如某种评价函数)进行质量评价，可以根据评价结果对上述已经训练好的超分网络进行更新，比如将评分较高的高分辨率图像及其对应的低分辨率图像作为正样本，将评分较低的高分辨率图像及其对应的低分辨率图像作为负样本，再次输入到上述已经训练好的超分网络中，由此可以不断的对质量差的超分图像进行迭代改进，从而得到更好的超分网络，进而为图像超分的应用输出质量更好的高分辨率图像。可选的，也可以在有更多新增训练数据集之后，对超分网络进行微调训练，使得超分网络能够在应用中不断对新数据进行扩充训练，使得图像超分的效果更好。
接着，对本申请实施例的硬件运行环境做简单介绍.
首先需要说明的是，本申请实施例提供的图像超分方法可以在服务器上被执行，也可以在云端被执行，还可以在终端(例如手机，电脑，摄像头，或者自动驾驶的车辆等)上被执行。更具体的，用于执行本申请实施例中的数据计算的，可以是传统的处理器+存储器的硬件结构，也可以是处理器+神经网络处理器+存储器的硬件结构，还可以是神经网络处理器+存储器的处理结构，本申请对此不作限定。以下以处理器+神经网络处理器+存储器的处理结构来进行举例。如图1所示：
图1所示的一种用于神经网络计算的电子设备100为例。电子设备100包括处理器102、存储器103。其中，所述处理器102可以是多个处理器的总称，其中可以只包括传统处理器，如CPU(Central Processing Unit，中央处理器)，还可以包括传统处理器和神经网络处理器(此处指可以用于神经网络运算的处理器，不限定为专用的神经网络处理器)；其中常见的神经网络处理器可以包括但不限于以下几种：GPU(Graphics Processing Unit，图形处理器)、NPU(Neural-network Processing Uint，神经网络处理器)、TPU(Tensor Processing Unit，张量处理器)。
例如，该处理器102中可以包括CPU，该处理器102中可以包括CPU和GPU，或，该 处理器102中可以包括CPU、GPU和NPU，或，该处理器102中可以包括CPU、GPU、NPU和TPU，也可以是CPU与GPU、NPU、TPU三者中任意一种或两种的组合；其中上面集中举例中所包括的CPU、GPU、NPU或TPU的数量可以是一个，也可以是多个，需要根据神经网络具体的应用场景对应的计算量和计算类型来决定。本申请实施例不做限定。
电子设备100可以为一个印制电路板(Printed Circuit Board，PCB)，集成电路(Integrated Circuit，IC)，或者一种通讯设备，例如移动电话、便携式电脑等。在一种实现方式中，处理器102、存储器103作为两个分立器件被设置于PCB板上；当处理器102是由多个处理器组成的，该多个处理器也可以分别作为独立的器件被设置的PCB板上。在另一种实现方式中，处理器102、存储器103被集成或封装于一个IC上。可以理解上述两部分可以根据需要进行集成或分立，例如当处理器102是由多个处理器组成时，可以将该多个处理器集成或封装于一个专用集成电路(Application-Specific Integrated Circuit，ASIC)上，再与存储器103一起设置于PCB板上。当处理器102是由多个处理器组成时，该多个处理器之间，以及它们与存储器103之间均可互相通信耦合，并存在高速数据传输连接，该高速数据传输连接可以通过分别通信连接处理器102中的多个处理器及存储器103的总线101来实现，总线101可以为AXI(Advanced eXtensible Interface)总线协议，或其他总线协议。
当处理器102包括CPU时，该CPU用于运行存储在存储器103中的软件程序和/或指令，以执行电子设备100的各种功能，上述CPU可以是基于X86架构、基于ARM架构和基于cortex-A架构的CPU。存储器103可以包括易失性存储器，例如随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器，例如闪存(flash memory)，硬盘或固态硬盘(Solid-State Drive，SSD)，也可以是上述种类的存储器的组合。该存储器103用于存储相关指令及数据。当处理器102中包含用于神经网络运算的处理器时，该神经网络处理器可以是单独的硬件芯片，也可以是与处理器102集成在一个硬件芯片中的计算电路，例如IP(Intellectual Property，知识产权)核。上述的一个或多个硬件芯片可以是ASIC、可编程逻辑器件(Programmable Logic Device，PLD)或上述种类的硬件芯片的组合，其中PLD可以是现场可编程逻辑门阵列(Field Programmable Gate Array，FPGA)、通用阵列逻辑(Generic Array Logic，GAL)等。
可选的，电子设备100还包括显示设备104、输入设备105、射频电路106和通信模块107。其中，显示设备104用于输出可视的文本、图形、视频及其任意的组合，显示设备104可以是LCD(Liquid Crystal Display，液晶显示器)或LPD(Lighting Power Density，发光聚合物显示器)；用户可以通过输入设备105输入命令和信息至电子设备100中，例如待识别的图像数据或声音数据(在本申请实施例中可以是待进行超分重建的图像)，其中输入设备105可以是鼠标、键盘、扫描仪或摄像头等；射频电路106用于接收和发送电磁波，将电信号变换成电磁波，或是将电磁波变换成电信号，并且通过电磁波与通信网络或其他通信设备进行通信；通信模块107用于处理通信数据，例如射频电路106接收的电磁波并变换成的电信号所表示的通信数据。
在一种实施方式中，当处理器102中包括CPU时：通过总线101连接所述CPU和所述存储器103。所述CPU通过总线101从存储器103中获取用于神经网络运算的输入数据、输入参数和指令，其中输入数据包括需要计算的神经网络数据，如待识别的图像数据、声音数据；输入参数包括用于神经网络计算的参数，如离线训练好的神经网络参数；指令包括指示神经网络层的信息，所述CPU根据指令配置对应的神经网络层的计算模式，这些计算模式可 以包括乘、累加、开方等运算，或者这些运算的组合。可选的，所述CPU也可以处理多种神经网络层的计算，如卷积(Convolution)层、全连接(Full Connected)层、池化(Pooling)层、归一化(Normalize)层等神经网络层。
在一种实施方式中，当所述处理器102包括传统处理器和神经网络处理器时，该神经网络处理器可以和传统处理器(如CPU)集成在一个硬件芯片中，也可以是单独的硬件芯片，电子设备100中的总线101将CPU、存储器103和神经网络处理器通信连接，形成数据通路。CPU通过总线101控制神经网络处理器启动神经网络运算；当神经网络处理器完成神经网络计算，则通过总线101通知处理器102计算结束。神经网络处理器通过总线101从存储器103中获取用于神经网络计算的输入数据、输入参数和指令，其中输入数据包括需要计算的神经网络数据，如待识别的图像数据、声音数据；输入参数包括用于神经网络计算的参数，如离线训练好的神经网络参数；指令包括指示神经网络层的信息，神经网络处理器根据指令，配置对应的神经网络层的计算模式，这些计算模式可以包括乘、累加、开方等运算，或者这些运算的组合。可选的，神经网络处理器可以处理多种神经网络层的计算，如卷积(Convolution)层、全连接(Full Connected)层、池化(Pooling)层、归一化(Normalize)层等神经网络层。
可以看出，单独利用CPU处理神经网络的运算和利用CPU加神经网络处理器进行神经网络运算，区别在于：仅有CPU进行神经网络运算时少了CPU和神经网络处理器之间的交互，即CPU不需要控制神经网络处理器开始或结束计算，因为它自己就是计算的主体；但是由于CPU和神经网络处理器在设计上的侧重不同，使得其对神经网络中必然包含的大规模并行运算的运算效率低于神经网络处理器，因此CPU加神经网络处理器的结合方式在神经网络计算中将更为常见，但是本申请实施例并不对此进行限定。
一个神经网络可以包括多种神经网络层，并且同一种神经网络层可以多次出现在一个神经网络中，例如，识别一个图像需要的神经网络层包括：卷积层、sigmoid层、池化层、sigmoid层、池化层和全连接层，上述的神经网络层包括了2层sigmoid层和2层池化层。当处理器102包括神经网络处理器时，在神经网络计算中，神经网络处理器依次对每一层神经网络层进行计算，其中，对当前神经网络层的计算结果可以作为下一层神经网络层计算的输入。在上面的例子中，卷积层的输入数据为输入参数和图像的输入数据，对卷积层进行计算后得到的计算结果被用于sigmoid层的输入数据。
接着上面的举例，当处理器102包括神经网络处理器时，在对一层神经网络进行一次完整的计算过程中，神经网络处理器先通过总线101从存储器103中读取需要处理的输入数据、输入参数和指令。由于计算不同的神经网络层需要对数据做不同模式的计算，因此读取的输入参数和指令与具体的神经网络层的种类有关。神经网络处理器根据对应的指令对输入数据和输入参数进行计算，得到的计算结果通过总线101写入存储器103，这些计算结果作为后续的神经网络层的计算所需要的数据。在下一层神经网络层的计算中，神经网络处理器再次通过总线101从存储器103中读取输入数据和输入参数，其中读取的输入数据可以包括上一层神经网络层的计算结果或者尚未经过计算的数据。
为了进一步了解神经网络处理器的运算能力，以区别于传统的处理器如CPU，下面对神经网络处理器的内部运算结构进行简单描述，常见的几种神经网络处理器中，目前应用最多的是GPU，因此以下主要以GPU举例，并将GPU与CPU进行对比以解释GPU的结构及运算特性。
GPU，又称显示核心，视觉处理器，显示芯片，使一种专门用于进行图像运算的微处理 器。与CPU类似，只不过GPU是专门执行复杂的数学和几何计算而设计的。如，GPU加速计算之所以可以提供非凡的应用程序性能，是因为其将应用程序密集部分的工作负载转移到GPU，同时仍由CPU运行其余程序代码。因为CPU是专门为顺序串行处理而优化的几个核心组成，而GPU则拥有一个由数以千计的更下更高效的核心(专为同时处理多重任务而设计)组成的大规模并行计算架构。GPU中单个核心可以处理的计算复杂度远低于CPU中单个核心可以处理的计算复杂度，随着技术的发展，GPU中单个核心可以处理的计算复杂度会越来越高，但是其计算工作仍需要由CPU进行控制触发。GPU和CPU之所以大不相同，是由于其设计目标的不同，它们分别针对了两种不同的应用场景。如，CPU需要很强的通用性来处理各种不同的数据类型，同时又要逻辑判断因此会引入大量的分支跳转和中断的处理。这些都使得CPU的内部结构异常复杂。而GPU面对的则是类型高度统一的、相互无依赖的大规模数据和不需要被打断的纯净的计算环境。GPU处理的首要目标是运算以及数据吞吐量，而CPU内部晶体管的首要目的是降低处理的延时以及保持管线繁忙，这也就决定了GPU在密集型计算方面比起CPU有更优越的表现。总而言之，CPU和GPU因为最初用来处理的任务就不同，所以设计上有不小的区别。而某些任务和GPU最初用来解决的问题比较相似，所以就可以用GPU来算。
接下来将结合本申请实施例的附图对本申请实施例提供的方法及装置展开介绍。
实施例一：
请参阅图2，图2是本申请实施例提供的一种图像超分方法的流程示意图。如图2所示，该图像超分方法包括但不限于如下步骤S201-S207。
S201、接收低分辨率图像。
需要说明的是，在步骤201中接收的低分辨率图像是对最初拿到的原始图像进行相关处理得到的，这个相关处理方式通常是双三次插值方式。具体的，双三次插值(Bicubic interpolation)又叫双立方插值，用于在图像中“插值”(Interpolating)或增加“像素”(Pixel)数量/密度的一种方法。通常利用插值技术增加图形数据，以便在它打印或其他形式输出的时候，能够增大打印面积以及(或者)分辨率。经过这个相关处理的目的是，使得该低分辨率图像相比原始图像更大，且其尺寸与最后得到的高分辨率图像的尺寸相同。
S202、对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图。
所述垂直梯度图(vertical gradient map)表示图像中相邻像素之间的纵向变换，水平梯度图(horizontal gradient map)表示图像中相邻像素之间的横向变换，亮度图(luminance map)表示图像的亮度信息。
步骤202的预处理方式，具体的，所述预处理即：从所述低分辨率图像中进行不同的特征提取以得到不同的特征，从而由不同的特征形成上述三个不同的特征图(垂直梯度图、水平梯度图以及亮度图)，当采用卷积操作的方式进行特征提取是，可以是利用三个不同的卷积核对所述低分辨率图像分别进行卷积操作，从而得到上述的垂直梯度图，水平梯度图以及亮度图。
S203、将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图。
具体的，将垂直梯度图，水平梯度图和亮度图分别作为三个不同维度的信息以构成待输入特征图可以是，将所述垂直梯度图，水平梯度图和亮度图进行拼接，举例来说，当垂直梯度图，水平梯度图和亮度图均为X*Y的二维图，则经过拼接得到的待输入特征图为3*X*Y， 即该待输入特征图的深度Z为3。
S204、对所述待输入特征图进行尺寸转换得到输入特征图。
值得说明的是，由于经过步骤202的预处理，图像的尺寸可能发生变化，为了使后续处理步骤顺利进行，需要将所述待输入特征图进行尺寸转换(也可称为维度转换，后面相同，不再赘述)得到所述输入特征图，尺寸转换的方式通常是卷积操作，当然也可以是别的方式，本申请实施例对此不作限定。
S205、对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图。
具体的，在本申请实施例的步骤205中，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，在本申请实施例中，可以是利用公路网络进行上述加权处理。公路网络在前文已经有相关介绍，其中的权重获取支路中引入传输门(transform gate)和携带门(carry gate)，后面将传输门简称为T，携带门简称为C，所述T本质上是传输门经过神经网络学习得到的权重值，因此后文也称权重值T，其中C＝1－T。如前文所述,在公路网络中，存在三条分支，一条权重获取分支用于根据输入特征图获取权重值并输出权重值T，一条分支用于根据至少一层隐含层(举例此处隐含层为n层)对输入信号进行非线性变换处理，另一条分支则是跳过所述n层隐含层直连到某一层。在本申请实施例中，用于进行加权处理的所述输入特征图即根据上述直连分支直接得到的，而非线性变换后的输入特征图则是经过n层隐含层进行非线性变换得到的。T和C两个概率分别被分配给所述非线性变换后的输入特征图和所述输入特征图，并进一步进行加权处理得到输出特征图,其中T被分配给非线性变换后的特征图,C被分配给输入特征图。
S206、对所述输出特征图进行尺寸转换得到残差图。
与步骤204的目的类似，由于经过中间几个步骤的处理，输出特征图的尺寸可能发生了变化，为了使步骤207得以实现，需要将输出特征图进行尺寸转换，使得转换后得到的所述残差图的尺寸与所述低分辨率图像的尺寸相同，从而为所述低分辨率图像和所述残差图结合提供实现的基础。尺寸转换的方式通常是卷积操作，当然也可以是别的方式，本申请实施例对此不作限定。
S207、将所述低分辨率图像和所述残差图结合得到高分辨率图像。
具体的，在步骤207中，利用残差网络，即除了经过上述一系列处理得到残差图的处理支路之外，还有一条直连分支，该直连分支即在卷积神经网络上直接跳过若干层连接到某一层的输出，本申请实施例中该直连分支的输出为步骤207中的低分辨率图像。步骤207中进行结合的两个特征图，一个是经过多层网络处理得到的残差图(此处的多层网络中包括但不限于公路网络，所包括的公路网络用于处理步骤205)，另一个则是由该直连分支直接得到的低分辨率图像，将该残差图和该低分辨率图像进行结合，本质上是将所述残差图中的像素与所述低分辨率图中的像素进行结合，从而得到高分辨率图像，完成图像超分的过程。
值得说明的是，实施例一提供的实施例可以是基于深度神经网络的图像超分方法，所述深度神经网络包括输入层，至少一层隐含层，输出层；所述输入层可以用于接收低分辨率图像；所述至少一层隐含层可以用于处理S202至S207，所述输出层可以用于输出所述高分辨率图像。上述公路网络和残差网络可以理解为是由所述深度神经网络中各层神经网络之间的某种连接关系形成的。后面的实施例及实现方式的相关部分与此处类似，不再赘述。
在本申请实施例中，由于步骤205中输出特征图是由输入特征图和非线性变换后的输入 特征图进行加权处理得到的，一方面，步骤205的处理过程中包括直连分支直接得到的输入特征图，该直连分支一定程度上克服了深度神经网络的梯度消失问题，并由此带来了更好的收敛效果。另一方面，由于步骤205中对所述输入特征图和所述非线性变换后的输入特征图进行加权处理，即为所述输入特征图和所述非线性变换后的输入特征图分别分配不同的权重值T和C，由此可以带来更好的超分效果。进一步的，由于步骤207中将所述低分辨率图像和所述残差图结合得到高分辨率图像，即其中的低分辨率图像是由直连分支直接得到的，使得本申请实施例进一步克服了深度神经网络的梯度消失问题，并进一步提高了深度神经网络的收敛性能，最终得到更好的图像超分效果。
实施例二：
实施例二包括实施例一的所有步骤，并在实施例一的基础上还包括S205a，所述S205a在所述步骤205之前，所述S205a包括：
根据所述输入特征图得到一组权重值；所述一组权重值用于对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理。
具体的，所述根据所述输入特征图得到所述一组权重值包括：对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T，所述权重值T用于对所述输入特征图以及非线性变换后的输入特征图进行加权处理。
具体的，如上述实施例一的步骤205中所描述的，即公路网络中包括一条用于获取权重值T的支路。该支路可以根据训练好的深度神经网络得到权重值T并输出权重值T，具体的权重获取方法可以是实施例二中S205a所述的方法，即对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T。这里的S型函数也称Sigmoid函数，由于其单增以及反函数单增等性质，Sigmoid函数常被用作神经网络的阈值函数，将变量映射到0,1之间。
可选的，在上述实施例一或实施例二的基础上，在一种可能的实施方式中，所述步骤205中的对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，包括：对所述输入特征图进行去拟合操作，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换后的输入特征图。
可选的，在上述实施例一或实施例二的基础上，可参考图2a，在一种可能的实施方式中，图2a可以用来表示遵循公路网络结构的单个公路单元，该公路单元中存在三条分支(请参阅前文对公路网络的介绍，此处不再赘述)，如图2a中所示的上通道分支，其用于获取权重值T，如图2a中的所示下通道分支，其用于根据至少一层隐含层(举例此处隐含层为n层)对输出信号进行非线性变换处理，如图2a中的所示直连分支，这条分支跳过了n隐含层的处理，将输入该公路网络单元的信号x直接传输给图2a中的聚合层，该聚合层用于根据上通道分支获取到的权重值T对所述输入特征图(图2a中以输入x表示)以及所述非线性变换后的输入特征图(图2a中以y表示)进行加权处理，这里所述的加权处理为加权求和。可选的，所述权重值T被分配给所述非线性变换后的输入特征图，(1-T)被分配给所述输入特征图。具体可以由如下公式表示：Output＝T*y+(1-T)*x；其中Output为图2a中聚合层的输出，即本申请实施例中所述的输出特征图，y为经过非线性变换后的输入特征图，x为输入特征图。
可选的，请参阅图2b，图2b是在图2a的基础上进一步细化，如图2b所示：所述上通道分支可以包括卷积层和Sigmiod层，具体的可以由卷积层1-1和Sigmiod层串联形成。所述下通道分支可以包括Dropout层，卷积层，ReLU层，具体的可以由Dropout层，卷积层2-1， ReLU1，卷积层2-2，ReLU2，卷积层2-3依次串联形成。其中所述ReLU即线性整流函数(Rectified Linear Unit)，ReLU函数常被作为神经元的激活函数，ReLU函数本质上为分段线性函数，可以把所有的负值都变成0而正值不变，这种操作被称为单侧抑制，正是因为有了这个单侧抑制，才可以使得神经网络中的神经元具有了稀疏激活性。通过ReLU实现稀疏后的神经网络能够更好的挖掘相关特征，拟合训练数据。
该处理过程可以对应前文中关于实施例一中步骤205的相应解释，即一条分支用于根据至少一层隐含层对输入信号进行非线性变换处理，在本申请实施例中，所述的非线性变换处理包括Dropout层的处理以及卷积层的处理，其中Dropout层用于进行去拟合操作。如上所述的，对输入特征图进行去拟合操作得到去拟合结果，并对所述去拟合结果进行卷积操作，本申请实施例通过对输入特征图先进行去拟合操作，从而在前端减少参数量，在进一步对去拟合结果进行卷积操作的过程中，由于卷积操作的输入参数减少，使得卷积操作的计算量降低，可以减少超分耗时，提高图像超分的速度。
在上述实施例一或实施例二，或基于实施例一或实施例二的基础上的一种实施方式中，进一步可选的，步骤207具体可以包括以下至少两种具体实现方式：
S2071：将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
或者；
S2072：将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
其中，步骤2071与步骤2072的区别在于，在步骤2071中，所述低分辨率图像和所述残差图的结合方式是将该两个图中对应位置的像素值进行相加，并将相加得到的结果作为所述高分辨率图像中与所述低分辨率图像和所述残差图的对应位置相应的位置的像素值。在步骤2072中，两个图像的结合不仅仅是相加，而是利用权重值进行加权求和。在步骤2072中，由于加权求和采用的系数是经过深度神经网络学习(也称神经网络训练)得到的，在深度神经网络学习该权重值的过程中，通过学习得到的权重系数是可以使得输出的高分辨率图像具有最佳的图像质量的，因此，通过步骤2072可以使图像超分输出的高分辨率图像具有更好的质量。
实施例三：
请参阅图3，图3是本申请实施例提供的一种图像超分方法的流程示意图。如图3所示，该图像超分方法包括但不限于如下步骤S301-S307。
S301、接收低分辨率图像；
S302、对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；
S303、将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；
S304、对所述待输入特征图进行尺寸转换得到输入特征图。
S305、根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个处理环节，所述n个处理环节以串联方式连接；所述串联的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个处理环节且与所述n个处理环节一一对应，其中n为大于或等于2的正整数。
S306、对所述输出特征图进行尺寸转换得到残差图。
S307、将所述低分辨率图像和所述残差图结合得到高分辨率图像。
上述S301至S304的步骤与实施例一的S201至S204相同，步骤306与步骤206处理方式相同，步骤307与步骤207处理方式相同，为了简洁起见，请参阅实施例一对相关步骤地展开，此处不再赘述。
在本申请实施例中，与实施例一不同的关键在于步骤305。
如实施例一中对步骤205的描述，步骤205采用了公路网络，引入了传输门T和携带门C，其本质上是利用公路网络的一条分支获取到权重值T，并根据该权重值T决定经过非线性变换处理得到的非线性变换后的输入特征图的加权比例。在这一点上，步骤305和步骤205是类似的，不同的是，步骤205经过一轮加权处理即结束了公路网络的处理过程，输出所述输出特征图进行步骤206的处理。而在实施例三中，步骤307进行了多轮加权处理，该多轮加权处理即S307中所述的串联的加权处理流程。具体的，该串联的加权处理流程中包含n个处理环节，每一个处理环节可以理解为一个公路网络处理过程，一个最小单元的完整的公路网络处理过程可以如步骤205所述，包括三条分支，此处不再赘述。不同的是，在实施例三中步骤307包含多个公路网络处理过程，且各个公路网络处理过程以串联方式相连，每一个公路网络处理过程中均有该公路网络的权重获取分支，并且通过该权重获取分支得到该公路网络的权重值，而本公路网络处理过程中获取到的权重值为该公路网络所用，即步骤305中描述的“所述n组权重值分别被用于所述n个处理环节且与所述n个处理环节一一对应”，在单个的公路网络处理过程中，即上述的n个处理环节中的单个的处理环节中，其处理流程与步骤205的过程相同，但是输入输出有差异，例如，串联流程中的第一个处理环节，它的输入是步骤304的输出，即输入特征图，但是其输出不会直接作为输出特征图进行输出，而是经过进一步的公路网络处理，即进入下一个处理环节，继续上面所述的公路网络的相关操作，得到的输出又将作为下一个公路网络的输入，进入再下一个处理环节，直到串联流程中的最后一个环节处理完，将它的输出作为步骤305的输出，即所述输出特征图，作为下一个步骤306的输入。
之所以将多个公路网络处理过程以串联的方式级联起来，是因为通过该串联的加权处理流程对所述输入特征图进行加权处理可以得到相较于单个的公路网络处理过程更优的结果，即最终得到的高分辨率图像的质量更佳，图像超分的效果更好。
值得说明的是，实施例三提供的实施例可以是基于深度神经网络的图像超分方法，所述深度神经网络包括输入层，至少一层隐含层，输出层；所述输入层可以用于接收低分辨率图像；所述至少一层隐含层可以用于处理S302至S307，所述输出层可以用于输出所述高分辨率图像。上述公路网络和残差网络可以理解为是由所述深度神经网络中各层神经网络之间的某种连接关系形成的。后面的实施例相关部分与此处类似，不再赘述。
需要说明的是，在实施例三的基础上，一种可能的实现方式中，所述n等于3。即步骤305中串联的加权处理流程中包含三个处理环节。
有实验显示，在三个公路网络处理过程串联的情况下，能得到相比较于其他取值更好的超分效果。
在实施例三的基础上，为了使n个处理环节中的每一个环节处理过程更清楚，一种可能的实现方式细化如下：
在所述n个处理环节中，其中当前处理环节的处理过程包括：将所述当前处理环节的输入特征图分别进行上通道处理和下通道处理，所述上通道处理包括：根据所述当前处理环节 的输入特征图获取所述当前处理环节的一组权重值；所述下通道处理包括非线性变换处理，经过所述下通道处理得到非线性变换结果；根据与所述当前处理环节一一对应的一组权重值对所述当前处理环节的输入特征图和所述非线性变换结果进行加权处理，得到所述当前处理环节的输出特征图；其中，所述当前处理环节为所述n个处理环节中的任意一个环节；其中，当所述当前处理环节为所述串联的加权处理流程中的第一个环节时，所述当前处理环节的输入特征图为所述输入特征图；当所述当前处理环节为所述串联的加权处理流程中的最后一个环节时，所述当前处理环节的输出特征图为所述输出特征图。
值得说明的是，在这种可能的实现方式中，所述的上通道获取权重值可以理解为步骤205中关于公路网络的三条分支中的用于获取权重值T的分支。该分支的具体操作可以包括：
对所述当前处理环节的输入特征图进行卷积操作并得到上通道卷积结果，并根据S型函数对所述上通道卷积结果进行处理，得到所述当前处理环节的一组权重值。
进一步可选的，所述的下通道处理可以理解为步骤205中关于公路网络的三条分支中用于根据至少一层隐含层(举例此处隐含层为n层)对输入信号进行非线性变换处理的分支。该分支的具体操作可以包括：
对所述当前处理环节的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换结果。
由于本申请实施例的这种实现方式通过对输入特征图先进行去拟合操作，从而在前端减少参数量，在进一步对去拟合结果进行卷积操作的过程中，由于卷积操作的输入参数减少，使得卷积操作的计算量降低，可以减少超分耗时，提高图像超分的速度。
在实施例三的基础上，为了使n个处理环节中的每一个环节处理过程更清楚，另一种可能的实现方式细化如下：
所述根据串联的加权处理流程对所述输入特征图进行加权处理，得到所述输出特征图，包括：
在所述n个处理环节中的一个处理环节i的处理过程包括：对所述处理环节i的输入特征图进行非线性变换处理，得到非线性变换后的处理环节i的输入特征图；根据与所述处理环节i一一对应的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理，得到所述处理环节i的输出特征图；其中，所述处理环节i为所述n个处理环节中的任意一个环节,1≤i≤n；其中，当所述处理环节i为所述串联的加权处理流程中的第一个环节时，所述处理环节i的输入特征图为所述输入特征图；当所述处理环节i为所述串联的加权处理流程中的最后一个环节时，所述处理环节i的输出特征图为所述输出特征图。
在这种可能的实现方式中，可选的，在所述根据与所述处理环节i一一对应的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理之前，所述方法还包括：
根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值；所述处理环节i的一组权重值用于对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理。
进一步可选的，所述根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值，包括：
对所述处理环节i的输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述 卷积结果进行处理，得到所述处理环节i的一组权重值。
在实施例三及基于实施例三的一些可能的实现方式中，进一步可选的，步骤307具体可以包括以下至少两种具体的实现方式：
S3071：将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
或者；
S3072：将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
由于步骤3071和步骤2071类似，步骤3072和步骤2072类似，且步骤3071和步骤3072的区别与步骤2071和步骤2072的区别相同，因此请参阅步骤2071与步骤2072处的相关描述，此处不再赘述。
上述详细阐述了本发明实施例的方法，下面提供了本发明实施例的装置。
实施例四：
请参考图4，图4是本申请实施例提供的一种图像超分装置400的结构示意图。如图4所示，该图像超分装置包括但不限于如下模块401-406。
接收模块401，用于接收低分辨率图像；
预处理模块402，用于对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；并将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成以待输入特征图；
第一尺寸转换模块403，用于对所述待输入特征图进行尺寸转换得到输入特征图；
公路模块404，用于对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，并对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图；
第二尺寸转换模块405，用于对所述输出特征图进行尺寸转换得到残差图；
超分模块406，用于将所述低分辨率图像和所述残差图结合得到高分辨率图像。
具体的，各个模块处理方式对应的解释可以参阅实施例一中对步骤201-步骤207的解释，因为本申请实施例四提供的装置400用于实现实施例一提供的方法，因此相应步骤地解释与实施例一中的解释一致，且对应的技术效果也应当相同，此处不再赘述。
在实施例四的基础上，在一种可选的实现方式中，所述装置400还可以包括：
权重获取模块，用于根据所述输入特征图得到一组权重值；所述一组权重值用于对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理。
可选的，所述权重获取模块具体可以用于对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T，所述权重值T用于所述公路模块对所述输入特征图以及非线性变换后的输入特征图进行加权处理。
结合上文可以看出，该权重获取模块可以被上述公路模块404包含，作为其中的权重获取分支的实现载体，即该权重获取模块可以是公路模块404的子模块，用于所述公路模块404获取并输出权重值T。相关处理细节与实施例一的相应步骤相同，此处不再赘述。
在实施例四或上述实现方式的技术上，进一步可选的，所述公路模块404具体用于：
对所述输入特征图进行去拟合操作，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换后的输入特征图。
结合上文的描述可以看出，该处理过程即上文关于公路网络的描述中另一分支(用于根据至少一层隐含层(举例此处隐含层为n层)对输入信号进行非线性变换处理)的处理过程，具体的处理方式及技术效果与实施例一或实施例二，或在其基础上可能的实施方式对应的处理方式及技术效果相同，此处不再赘述。
在实施例四及其在其基础上的可能的实现方式的基础上，进一步可选的，
所述超分模块406具体可以用于：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
或者；
所述超分模块406具体可以用于：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
所述超分模块406的上述两种处理方式的区别请参阅方法实施例中的描述，此处不再赘述。
实施例五：
请参考图5，图5是本申请实施例提供的一种图像超分装置500的结构示意图。如图5所示，该图像超分装置包括但不限于如下模块501-506：
接收模块501，用于接收低分辨率图像。
预处理模块502，用于对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；并将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图。
第一尺寸转换模块503，用于对所述待输入特征图进行尺寸转换得到输入特征图。
公路模块504，用于根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个公路单元，所述n个公路单元以串联方式连接；所述串联的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个公路单元且与所述n个公路单元一一对应，其中n为大于或等于2的正整数。其中图5公路模块204中，在公路单元2到公路单元n之间为了简洁起见，省略了更多可能的公路单元。
第二尺寸转换模块505，用于对所述输出特征图进行尺寸转换得到残差图。
超分模块506，用于将所述低分辨率图像和所述残差图结合得到高分辨率图像。
具体的，各个模块处理方式对应的解释可以参阅实施例三中对步骤301-步骤307的解释，因为本申请实施例五提供的装置500用于实现实施例三提供的方法，因此相应步骤地解释与实施例三中的解释一致，且对应的技术效果也应当相同，此处不再赘述。
需要说明的是，在实施例五的基础上，一种可能的实现方式中，所述n等于3。即所述公路模块504中包含三个串联的公路单元。
有实验显示，在三个公路网络处理过程串联的情况下，能得到相比较于其他取值更好的超分效果。
在实施例五的基础上，为了使n个公路单元中的每一个公路单元中的过程更清楚，一种可能的实现方式细化如下：
在所述n个公路单元中，其中第一公路单元具体用于：将所述第一公路单元的输入特征图分别进行上通道处理和下通道处理，所述上通道处理包括：根据所述第一公路单元的输入 特征图得到所述第一公路单元的一组权重值；所述下通道处理包括非线性变换处理，经过所述下通道处理得到非线性变换结果；根据与所述第一公路单元一一对应的一组权重值对所述第一公路单元的输入特征图和所述非线性变换结果进行加权处理，得到所述第一公路单元的输出特征图；其中，所述第一公路单元为所述n个公路单元中的任意一个公路单元；其中，当所述第一公路单元为所述串联的加权处理流程中的第一个公路单元时，所述第一公路单元的输入特征图为所述输入特征图；当所述第一公路单元为所述串联的加权处理流程中的最后一个环节时，所述第一公路单元的输出特征图为所述输出特征图。
值得说明的是，在这种可能的实现方式中，所述的上通道获取权重值可以理解为步骤205中关于公路网络的三条分支中的用于获取权重值T的分支。该分支的具体操作可以包括：
对所述第一公路单元的输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到所述第一公路单元的一组权重值。
进一步可选的，所述的下通道处理可以理解为步骤205中关于公路网络的三条分支中用于根据至少一层隐含层(举例此处隐含层为n层)对输入信号进行非线性变换处理的分支。该分支的具体操作可以包括：
对所述第一公路单元的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换结果。
由于本申请实施例的这种实现方式通过对输入特征图先进行去拟合操作，从而在前端减少参数量，在进一步对去拟合结果进行卷积操作的过程中，由于卷积操作的输入参数减少，使得卷积操作的计算量降低，可以减少超分耗时，提高图像超分的速度。
在实施例五的基础上，为了使n个公路单元中的每一个公路单元中的过程更清楚，另一种可能的实现方式细化如下：
在所述n个公路单元中，其中一个公路单元i具体用于：
对所述公路单元i的输入特征图进行非线性变换处理，得到非线性变换后的公路单元i的输入特征图；根据与所述公路单元i一一对应的一组权重值对所述公路单元i的输入特征图和所述非线性变换后的所述公路单元i的输入特征图进行加权处理，得到所述公路单元i的输出特征图；其中，所述公路单元i为所述n个公路单元中的任意一个公路单元,1≤i≤n；其中，当所述公路单元i为所述串联的加权处理流程中的第一个公路单元时，所述公路单元i的输入特征图为所述输入特征图；当所述公路单元i为所述串联的加权处理流程中的最后一个公路单元时，所述公路单元i的输出特征图为所述输出特征图。
在这种可能的实现方式中，可选的，所述装置还包括权重获取模块，用于根据所述公路单元i的输入特征图得到所述公路单元i的一组权重值；所述公路单元i的一组权重值用于对所述公路单元i的输入特征图和所述非线性变换后的所述公路单元i的输入特征图进行加权处理。
进一步可选的，所述权重获取模块具体用于：对所述公路单元i的输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到所述公路单元i的一组权重值。
在实施例五及基于实施例五的一些可能的实现方式中，进一步可选的，超分模块506具体可以用于：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值，将所述低分辨率图像的颜色作为所述高分辨 率图像的颜色。
或者；
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值，将所述低分辨率图像的颜色作为所述高分辨率图像的颜色。
所述超分模块506的上述两种处理方式的区别请参阅方法实施例中的描述，此处不再赘述。
请继续参阅图1，当图1用来执行本申请实施例提供的图像超分时，所述存储器103可以用于存储上文所述的超分模型。
通信模块107可以用于与其他设备进行通信，例如，可以用于接收需要重建的低分辨率图像。
处理器102可以是一个或多个中央处理器(central processing unit，CPU)，在处理器102是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU；处理器102还可以包含神经网络处理器，如前文所述的GPU、NPU或TPU，用于进行神经网络中的高密度可并行的海量运算。
处理器102用于读取存储器103中存储的程序代码，执行以下操作：
接收低分辨率图像；
对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；
将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成以待输入特征图；
对所述待输入特征图进行尺寸转换得到输入特征图；
对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图；
对所述输出特征图进行尺寸转换得到残差图；
将所述低分辨率图像和所述残差图结合得到高分辨率图像。
作为一种可能的实施方式，在所述对所述输入特征图以及非线性变换后的输入特征图进行加权处理之前，所述处理器102还执行：
对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T，所述权重值T用于对所述输入特征图以及非线性变换后的输入特征图进行加权处理。
作为一种可能的实施方式，所述处理器102对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，包括：
对所述输入特征图进行去拟合操作，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换后的输入特征图。
作为一种可能的实施方式，所述处理器102将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
或者；
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行 加权求和，得到所述高分辨率图像中的相应像素的像素值。
需要说明的是，上述各个操作的实现还可以对应参照图2所示的方法实施例一的相应描述，这里不再赘述。上述的图像超分设备可以是图像识别设备，例如人脸识别设备。上述的图像超分设备还可以是图像增强设备等。
或，图像超分设备中的处理器102用于读取存储器103中存储的程序代码，执行以下操作：
接收低分辨率图像；
对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；
将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；
对所述待输入特征图进行尺寸转换得到输入特征图；
根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个处理环节，所述n个处理环节以串联方式连接；所述串联的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个处理环节且与所述n个处理环节一一对应，其中n为大于或等于2的正整数；
对所述输出特征图进行尺寸转换得到残差图；
将所述低分辨率图像和所述残差图结合得到高分辨率图像。
作为一种可能的实施方式，所述处理器102在所述n个处理环节中的当前处理环节中，具体用于：将所述当前处理环节的输入特征图分别进行上通道处理和下通道处理，所述上通道处理包括：根据所述当前处理环节的输入特征图得到所述当前处理环节的一组权重值；所述下通道处理包括非线性变换处理，经过所述下通道处理得到非线性变换结果；根据与所述当前处理环节一一对应的一组权重值对所述当前处理环节的输入特征图和所述非线性变换结果进行加权处理，得到所述当前处理环节的输出特征图；其中，所述当前处理环节为所述n个处理环节中的任意一个环节；其中，当所述当前处理环节为所述串联的加权处理流程中的第一个环节时，所述当前处理环节的输入特征图为所述输入特征图；当所述当前处理环节为所述串联的加权处理流程中的最后一个环节时，所述当前处理环节的输出特征图为所述输出特征图。
可选的，所述n等于3。
作为一种可能的实施方式，所述处理器102具体用于：对所述当前处理环节的输入特征图进行卷积操作并得到上通道卷积结果，并根据S型函数对所述上通道卷积结果进行处理，得到所述当前处理环节的一组权重值。
作为一种可能的实施方式，所述处理器102具体用于：对所述当前处理环节的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换结果。
作为一种可能的实施方式，所述处理器102具体用于：
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
或；
将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
需要说明的是，上述各个操作的实现还可以对应参照图3所示的方法实施例三的相应描述，这里不再赘述。上述的图像超分设备可以是图像识别设备，例如人脸识别设备。上述的图像超分设备还可以是图像增强设备等。
本发明实施例还提供一种芯片系统，所述芯片系统包括至少一个处理模块，存储器和接口电路，所述存储器、所述接口电路和所述至少一个处理模块通过线路互联，所述至少一个存储器中存储有指令；所述指令被所述处理模块执行时，图2或图3所示的方法流程得以实现。其中，所述处理模块中至少包括中央处理器，还可以包括中央处理器和神经网络处理器，其中中央处理器可以是一个也可以是多个，神经网络处理器可以是一个也可以是多个，且神经网络处理器的类型不做限定。处理模块中包括中央处理器和神经网络处理器时，当所述指令被所述处理模块执行时，图2中的步骤205和步骤207将根据该处理模块中的神经网络处理器进行运算从而得以实现，图3中的步骤305和步骤307将根据该处理模块中的神经网络处理器进行运算得以实现，其他步骤可以由中央处理器实现，也可以由神经网络处理器实现，当然，上述的步骤205、207；305、307也可以由中央处理器实现，此处不做限定。
如图6所示的是另一种用于神经网络计算的电子设备600的结构示意图，包括IO(Input and Output，输入输出)口610、存储介质620和PLD(Programmable Logic Device，可编程逻辑器件)630。电子设备600通过IO口610读取其他设备输出的配置数据，例如，电子设备600通过JTAG(Joint Test Action Group，联合测试行为组织)接口从PC读取经过布局布线后的设计网表。存储介质620可以包括RAM或ROM，或者二者的组合，用于存储读取的配置数据；PLD 630可以为FPGA或其他可编程逻辑器件。存储介质620与PLD 630可以进行双向的数据传输。PLD 630从存储介质620中读取配置数据，并根据配置数据对其内部的多个逻辑块进行配置，例如对其中的多个查找表的连接方式进行配置，以形成处理器102中的神经网络处理器GPU、NPU或TPU。存储介质620还可以存储用于计算的数据，例如输入数据和输入参数，以及存储PLD 630的计算结果。
本发明实施例还提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在处理器上运行时，图2或图3所示的方法流程得以实现。
本发明实施例还提供一种计算机程序产品，当所述计算机程序产品在处理器上运行时，图2或图3所示的方法流程得以实现。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者通过所述计算机可读存储介质进行传输。所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如，固态硬盘(Solid State Disk，SSD))等。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，该流程可以由计算机程序来指令相关的硬件完成，该程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法实施例的流程。而前述的存储介质包括：ROM或随机存储记忆体RAM、磁碟或者光盘等各种可存储程序代码的介质。

Claims (30)

  1. 一种图像超分方法，其特征在于，包括：
    接收低分辨率图像；
    对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；
    将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；
    对所述待输入特征图进行尺寸转换得到输入特征图；
    对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图；
    对所述输出特征图进行尺寸转换得到残差图；
    将所述低分辨率图像和所述残差图结合得到高分辨率图像。
  2. 根据权利要求1所述的方法，其特征在于，在所述对所述输入特征图以及非线性变换后的输入特征图进行加权处理之前，所述方法还包括：
    根据所述输入特征图得到一组权重值；所述一组权重值用于对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理。
  3. 根据权利要求2所述的方法，其特征在于，根据所述输入特征图得到所述一组权重值，包括：
    对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T，所述权重值T用于对所述输入特征图以及非线性变换后的输入特征图进行加权处理。
  4. 根据权利要求1至3任一项所述的方法，其特征在于，所述对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，包括：
    对所述输入特征图进行去拟合操作，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换后的输入特征图。
  5. 根据权利要求1-4任一项所述的方法，其特征在于，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
  6. 根据权利要求1-4任一项所述的方法，其特征在于，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
  7. 一种图像超分方法，其特征在于，包括：
    接收低分辨率图像；
    对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；
    将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；
    对所述待输入特征图进行尺寸转换得到输入特征图；
    根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个处理环节，所述n个处理环节以串联方式连接；所述串联 的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个处理环节且与所述n个处理环节一一对应，其中n为大于或等于2的正整数；
    对所述输出特征图进行尺寸转换得到残差图；
    将所述低分辨率图像和所述残差图结合得到高分辨率图像。
  8. 根据权利要求7所述的方法，其特征在于，所述根据串联的加权处理流程对所述输入特征图进行加权处理，得到所述输出特征图，包括：
    在所述n个处理环节中的一个处理环节i的处理过程包括：对所述处理环节i的输入特征图进行非线性变换处理，得到非线性变换后的处理环节i的输入特征图；根据与所述处理环节i一一对应的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理，得到所述处理环节i的输出特征图；其中，所述处理环节i为所述n个处理环节中的任意一个环节,1≤i≤n；其中，当所述处理环节i为所述串联的加权处理流程中的第一个环节时，所述处理环节i的输入特征图为所述输入特征图；当所述处理环节i为所述串联的加权处理流程中的最后一个环节时，所述处理环节i的输出特征图为所述输出特征图。
  9. 根据权利要求7或8所述的方法，其特征在于，在所述根据与所述处理环节i一一对应的一组权重值对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理之前，所述方法还包括：
    根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值；所述处理环节i的一组权重值用于对所述处理环节i的输入特征图和所述非线性变换后的所述处理环节i的输入特征图进行加权处理。
  10. 根据权利要求9所述的方法，其特征在于，所述根据所述处理环节i的输入特征图得到所述处理环节i的一组权重值，包括：
    对所述处理环节i的输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到所述处理环节i的一组权重值。
  11. 根据权利要求7至10任一项所述的方法，其特征在于，所述n等于3。
  12. 根据权利要求8-11任意一项所述的方法，其特征在于，所述对所述处理环节i的输入特征图进行非线性变换处理，得到非线性变换后的所述处理环节i的输入特征图包括：
    对所述处理环节i的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到非线性变换后的所述处理环节i的输入特征图。
  13. 根据权利要求7-12任一项所述的方法，其特征在于，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
  14. 根据权利要求7-12任一项所述的方法，其特征在于，所述将所述低分辨率图像和所述残差图结合得到高分辨率图像，包括：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
  15. 一种图像超分装置，其特征在于，包括：
    接收模块，用于接收低分辨率图像；
    预处理模块，用于对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及 亮度图；并将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成以待输入特征图；
    第一尺寸转换模块，用于对所述待输入特征图进行尺寸转换得到输入特征图；
    公路模块，用于对所述输入特征图进行非线性变换处理得到非线性变换后的输入特征图，并对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理，得到输出特征图；
    第二尺寸转换模块，用于对所述输出特征图进行尺寸转换得到残差图；
    超分模块，用于将所述低分辨率图像和所述残差图结合得到高分辨率图像。
  16. 根据权利要求15所述的装置，其特征在于，所述装置还包括：权重获取模块，用于根据所述输入特征图得到一组权重值；所述一组权重值用于对所述输入特征图以及所述非线性变换后的输入特征图进行加权处理。
  17. 根据权利要求15或16所述的装置，其特征在于，所述权重获取模块，具体用于：对所述输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到权重值T，所述权重值T用于所述公路模块对所述输入特征图以及非线性变换后的输入特征图进行加权处理。
  18. 根据权利要求15至17任一项所述的装置，其特征在于，所述公路模块具体用于：
    对所述输入特征图进行去拟合操作，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到所述非线性变换后的输入特征图。
  19. 根据权利要求15-18任一项所述的装置，其特征在于，所述超分模块具体用于：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
  20. 根据权利要求15-18任一项所述的装置，其特征在于，所述超分模块具体用于：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
  21. 一种图像超分装置，其特征在于，包括：
    接收模块，用于接收低分辨率图像；
    预处理模块，用于对所述低分辨率图像进行预处理，得到垂直梯度图，水平梯度图以及亮度图；并将所述垂直梯度图，所述水平梯度图和所述亮度图分别作为三个不同维度的信息以构成待输入特征图；
    第一尺寸转换模块，用于对所述待输入特征图进行尺寸转换得到输入特征图；
    公路模块，用于根据串联的加权处理流程对所述输入特征图进行加权处理，得到输出特征图；其中，所述串联的加权处理流程中包含n个公路单元，所述n个公路单元以串联方式连接；所述串联的加权处理流程中包含n组权重值，所述n组权重值分别被用于所述n个公路单元且与所述n个公路单元一一对应，其中n为大于或等于2的正整数；
    第二尺寸转换模块，用于对所述输出特征图进行尺寸转换得到残差图；
    超分模块，用于将所述低分辨率图像和所述残差图结合得到高分辨率图像。
  22. 根据权利要求21所述的装置，其特征在于，
    在所述n个公路单元中，其中一个公路单元i具体用于：
    对所述公路单元i的输入特征图进行非线性变换处理，得到非线性变换后的公路单元i的输入特征图；根据与所述公路单元i一一对应的一组权重值对所述公路单元i的输入特征图和所述非线性变换后的所述公路单元i的输入特征图进行加权处理，得到所述公路单元i的输 出特征图；其中，所述公路单元i为所述n个公路单元中的任意一个公路单元,1≤i≤n；其中，当所述公路单元i为所述串联的加权处理流程中的第一个公路单元时，所述公路单元i的输入特征图为所述输入特征图；当所述公路单元i为所述串联的加权处理流程中的最后一个公路单元时，所述公路单元i的输出特征图为所述输出特征图。
  23. 根据权利要求22所述的装置，其特征在于，所述装置还包括权重获取模块，用于根据所述公路单元i的输入特征图得到所述公路单元i的一组权重值；所述公路单元i的一组权重值用于对所述公路单元i的输入特征图和所述非线性变换后的所述公路单元i的输入特征图进行加权处理。
  24. 根据权利要求23所述的装置，其特征在于，所述权重获取模块具体用于：对所述公路单元i的输入特征图进行卷积操作并得到卷积结果，并根据S型函数对所述卷积结果进行处理，得到所述公路单元i的一组权重值。
  25. 根据权利要求21至24任一项所述的装置，其特征在于，所述n等于3。
  26. 根据权利要求22-25任意一项所述的装置，其特征在于，
    所述公路单元i具体用于：对所述公路单元i的输入特征图进行去拟合，得到去拟合结果，并对所述去拟合结果进行卷积操作，得到非线性变换后的公路单元i的输入特征图。
  27. 根据权利要求21-26任一项所述的装置，其特征在于，所述超分模块具体用于：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值相加，得到所述高分辨率图像中的相应像素的像素值。
  28. 根据权利要求21-26任一项所述的装置，其特征在于，所述超分模块具体用于：
    将所述低分辨率图像中的每一个像素的像素值与所述残差图中的对应像素的像素值进行加权求和，得到所述高分辨率图像中的相应像素的像素值。
  29. 一种图像超分设备，其特征在于，包括处理器和存储器，所述存储器用于存储程序指令，所述处理器用于调用所述程序指令来执行权利要求1-14任一项所述的方法。
  30. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有程序指令，当所述程序指令由处理器运行时，实现权利要求1-14任一项所述的方法。
PCT/CN2019/081445 2018-04-04 2019-04-04 图像超分方法及装置 Ceased WO2019192588A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/029,554 US11593916B2 (en) 2018-04-04 2020-09-23 Image super-resolution method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810299530.9A CN109903221B (zh) 2018-04-04 2018-04-04 图像超分方法及装置
CN201810299530.9 2018-04-04

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/029,554 Continuation US11593916B2 (en) 2018-04-04 2020-09-23 Image super-resolution method and apparatus

Publications (1)

Publication Number Publication Date
WO2019192588A1 true WO2019192588A1 (zh) 2019-10-10

Family

ID=66943102

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/081445 Ceased WO2019192588A1 (zh) 2018-04-04 2019-04-04 图像超分方法及装置

Country Status (3)

Country Link
US (1) US11593916B2 (zh)
CN (1) CN109903221B (zh)
WO (1) WO2019192588A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111223046A (zh) * 2019-11-20 2020-06-02 中国科学院遥感与数字地球研究所 一种图像超分辨率重建方法及装置
CN111583107A (zh) * 2020-04-03 2020-08-25 长沙理工大学 一种基于注意力机制的图像超分辨率重建方法和系统
CN111754406A (zh) * 2020-06-22 2020-10-09 北京大学深圳研究生院 图像分辨率处理方法、装置、设备及可读存储介质
CN111754405A (zh) * 2020-06-22 2020-10-09 北京大学深圳研究生院 图像降分辨率及复原方法、设备及可读存储介质
CN112019861A (zh) * 2020-07-20 2020-12-01 清华大学 基于关键帧指导超分辨率的视频压缩方法及装置
CN115115912A (zh) * 2022-05-26 2022-09-27 清华大学 基于串并联查找表的轻量化图像超分辨率重建方法及装置
CN117036172A (zh) * 2023-08-15 2023-11-10 禹创半导体(深圳)有限公司 影像分辨率调整方法、装置、设备及介质
EP4207051A4 (en) * 2020-09-21 2024-03-27 Huawei Technologies Co., Ltd. HIGH-RESOLUTION IMAGE METHOD AND ELECTRONIC DEVICE

Families Citing this family (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11308598B2 (en) * 2019-02-14 2022-04-19 Sharif University Of Technology Quality assessment of an image
WO2020252764A1 (en) * 2019-06-21 2020-12-24 Intel Corporation Adaptive deep learning model for noisy image super-resolution
KR102865289B1 (ko) 2019-08-14 2025-09-29 삼성전자주식회사 전자 장치 및 그 제어 방법
CN112419146B (zh) * 2019-08-20 2023-12-29 武汉Tcl集团工业研究院有限公司 一种图像处理方法、装置及终端设备
CN112581362A (zh) * 2019-09-27 2021-03-30 美敦力公司 用于调整图像细节的图像处理方法和装置
CN110889809B9 (zh) * 2019-11-28 2023-06-23 RealMe重庆移动通信有限公司 图像处理方法及装置、电子设备、存储介质
US12367547B2 (en) * 2020-02-17 2025-07-22 Intel Corporation Super resolution using convolutional neural network
TWI733341B (zh) * 2020-02-20 2021-07-11 瑞昱半導體股份有限公司 用於圖像放大與增強的方法與裝置
CN111524123B (zh) 2020-04-23 2023-08-08 北京百度网讯科技有限公司 用于处理图像的方法和装置
US11288771B2 (en) * 2020-04-29 2022-03-29 Adobe Inc. Texture hallucination for large-scale image super-resolution
CN111368849B (zh) * 2020-05-28 2020-08-28 腾讯科技（深圳）有限公司 图像处理方法、装置、电子设备及存储介质
CN111652330B (zh) * 2020-08-05 2020-11-13 深圳市优必选科技股份有限公司 图像处理方法、装置、系统、电子设备及可读存储介质
CN111932463B (zh) * 2020-08-26 2023-05-30 腾讯科技（深圳）有限公司 图像处理方法、装置、设备及存储介质
KR20220071011A (ko) * 2020-11-23 2022-05-31 삼성전자주식회사 전자 장치 및 그 제어 방법
US12118692B2 (en) * 2020-12-17 2024-10-15 PicsArt, Inc. Image super-resolution
CN112862681B (zh) * 2021-01-29 2023-04-14 中国科学院深圳先进技术研究院 一种超分辨率方法、装置、终端设备及存储介质
CN112950470B (zh) * 2021-02-26 2022-07-15 南开大学 基于时域特征融合的视频超分辨率重建方法及系统
KR20220129995A (ko) 2021-03-17 2022-09-26 주식회사 에스아이에이 딥러닝 기반 초해상도 이미징 방법
CN115345777B (zh) * 2021-05-13 2026-03-20 南京大学 用于成像的方法、装置和计算机可读介质
CN113642496B (zh) * 2021-08-20 2025-01-10 京东鲲鹏(江苏)科技有限公司 交通要素的识别方法和装置
CN114283066A (zh) 2021-09-01 2022-04-05 鸿海精密工业股份有限公司 影像处理设备以及超解析处理方法
CN113763248B (zh) * 2021-09-08 2025-05-09 深圳前海微众银行股份有限公司 超分辨率图像重建方法、装置、设备及存储介质
CN116170548B (zh) * 2021-11-24 2026-01-23 瑞昱半导体股份有限公司 超分辨率图像产生装置
CN114782249A (zh) * 2022-04-14 2022-07-22 北京百度网讯科技有限公司 一种图像的超分辨率重建方法、装置、设备以及存储介质
WO2023229645A1 (en) * 2022-05-25 2023-11-30 Innopeak Technology, Inc. Frame-recurrent video super-resolution for raw images
US12608758B2 (en) * 2023-01-03 2026-04-21 Bae Systems Information And Electronic Systems Integration Inc. Maintaining original color when applying super resolution to individual bands
US20240303767A1 (en) * 2023-03-07 2024-09-12 Samsung Electronics Co., Ltd. Electronic device and controlling method to select scaler based on image characteristics
CN115982418B (zh) * 2023-03-17 2023-05-30 亿铸科技(杭州)有限责任公司 一种提升ai计算芯片超分运算性能的方法
CN116483587B (zh) * 2023-06-21 2023-09-08 湖南马栏山视频先进技术研究院有限公司 一种基于图像分割的视频超分并行方法、服务器及介质
US20250054099A1 (en) * 2023-08-07 2025-02-13 Saudi Arabian Oil Company Increasing Petrophysical Image Log Resolution using Deep Learning Techniques
CN117275006A (zh) * 2023-10-23 2023-12-22 联仁健康医疗大数据科技股份有限公司 一种图像处理方法、装置、电子设备及存储介质
KR20250061055A (ko) * 2023-10-26 2025-05-08 한국전자기술연구원 비디오 슈퍼 해상도를 이용한 분할 렌더링 컨텐츠 스트리밍 방법
CN118350997A (zh) * 2024-03-16 2024-07-16 东莞理工学院 一种车牌图像超分辨率重建模型构建方法及系统
CN121599837A (zh) * 2024-08-20 2026-03-03 华为技术有限公司 一种图像处理方法、装置和系统
CN119722456B (zh) * 2024-12-03 2026-04-03 天津市测绘院有限公司 基于卷积尺度不确定性的遥感图像超分重建方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070291170A1 (en) * 2006-06-16 2007-12-20 Samsung Electronics Co., Ltd. Image resolution conversion method and apparatus
CN106683067A (zh) * 2017-01-20 2017-05-17 福建帝视信息科技有限公司 一种基于残差子图像的深度学习超分辨率重建方法
CN106709875A (zh) * 2016-12-30 2017-05-24 北京工业大学 一种基于联合深度网络的压缩低分辨率图像复原方法
CN107358575A (zh) * 2017-06-08 2017-11-17 清华大学 一种基于深度残差网络的单幅图像超分辨率重建方法
CN108259994A (zh) * 2018-01-15 2018-07-06 复旦大学 一种提高视频空间分辨率的方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103854254B (zh) * 2012-11-28 2016-12-21 北京大学 一种基于自回归模型的任意比例图像插值方法
CN103366347B (zh) * 2013-07-16 2016-09-14 苏州新视线文化科技发展有限公司 基于稀疏表示的图像超分辨率重建方法
CN104376550A (zh) * 2014-12-01 2015-02-25 中南大学 基于含积分的平差模型的超分辨率图像重建方法
CN106600538A (zh) * 2016-12-15 2017-04-26 武汉工程大学 一种基于区域深度卷积神经网络的人脸超分辨率算法
CN106991646B (zh) 2017-03-28 2020-05-26 福建帝视信息科技有限公司 一种基于密集连接网络的图像超分辨率方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070291170A1 (en) * 2006-06-16 2007-12-20 Samsung Electronics Co., Ltd. Image resolution conversion method and apparatus
CN106709875A (zh) * 2016-12-30 2017-05-24 北京工业大学 一种基于联合深度网络的压缩低分辨率图像复原方法
CN106683067A (zh) * 2017-01-20 2017-05-17 福建帝视信息科技有限公司 一种基于残差子图像的深度学习超分辨率重建方法
CN107358575A (zh) * 2017-06-08 2017-11-17 清华大学 一种基于深度残差网络的单幅图像超分辨率重建方法
CN108259994A (zh) * 2018-01-15 2018-07-06 复旦大学 一种提高视频空间分辨率的方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KIM, JIWON: "Accurate Image Super-Resolution Using Very Deep Convolutio- nal Networks", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 30 June 2016 (2016-06-30), XP055570210 *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111223046A (zh) * 2019-11-20 2020-06-02 中国科学院遥感与数字地球研究所 一种图像超分辨率重建方法及装置
CN111223046B (zh) * 2019-11-20 2023-04-25 中国科学院遥感与数字地球研究所 一种图像超分辨率重建方法及装置
CN111583107A (zh) * 2020-04-03 2020-08-25 长沙理工大学 一种基于注意力机制的图像超分辨率重建方法和系统
CN111754406B (zh) * 2020-06-22 2024-02-23 北京大学深圳研究生院 图像分辨率处理方法、装置、设备及可读存储介质
CN111754406A (zh) * 2020-06-22 2020-10-09 北京大学深圳研究生院 图像分辨率处理方法、装置、设备及可读存储介质
CN111754405A (zh) * 2020-06-22 2020-10-09 北京大学深圳研究生院 图像降分辨率及复原方法、设备及可读存储介质
CN111754405B (zh) * 2020-06-22 2023-08-08 北京大学深圳研究生院 图像降分辨率及复原方法、设备及可读存储介质
CN112019861A (zh) * 2020-07-20 2020-12-01 清华大学 基于关键帧指导超分辨率的视频压缩方法及装置
CN112019861B (zh) * 2020-07-20 2021-09-14 清华大学 基于关键帧指导超分辨率的视频压缩方法及装置
EP4207051A4 (en) * 2020-09-21 2024-03-27 Huawei Technologies Co., Ltd. HIGH-RESOLUTION IMAGE METHOD AND ELECTRONIC DEVICE
CN115115912A (zh) * 2022-05-26 2022-09-27 清华大学 基于串并联查找表的轻量化图像超分辨率重建方法及装置
CN115115912B (zh) * 2022-05-26 2024-06-07 清华大学 基于串并联查找表的轻量化图像超分辨率重建方法及装置
CN117036172A (zh) * 2023-08-15 2023-11-10 禹创半导体(深圳)有限公司 影像分辨率调整方法、装置、设备及介质
CN117036172B (zh) * 2023-08-15 2024-09-17 禹创半导体(深圳)有限公司 影像分辨率调整方法、装置、设备及介质

Also Published As

Publication number Publication date
US11593916B2 (en) 2023-02-28
CN109903221A (zh) 2019-06-18
CN109903221B (zh) 2023-08-22
US20210004935A1 (en) 2021-01-07

Similar Documents

Publication Publication Date Title
WO2019192588A1 (zh) 图像超分方法及装置
CN109949255B (zh) 图像重建方法及设备
CN113011562B (zh) 一种模型训练方法及装置
CN112070664B (zh) 一种图像处理方法以及装置
US12530738B2 (en) Neural network training method, image processing method, and apparatus
WO2021018163A1 (zh) 神经网络的搜索方法及装置
CN112529150A (zh) 一种模型结构、模型训练方法、图像增强方法及设备
WO2021164731A1 (zh) 图像增强方法以及图像增强装置
WO2022134971A1 (zh) 一种降噪模型的训练方法及相关装置
WO2020177607A1 (zh) 图像去噪方法和装置
WO2020216227A1 (zh) 图像分类方法、数据处理方法和装置
CN109064405A (zh) 一种基于双路径网络的多尺度图像超分辨率方法
JP6684951B2 (ja) 人工知能推論演算装置
WO2022067508A1 (zh) 一种神经网络加速器、加速方法以及装置
WO2021042774A1 (zh) 图像恢复方法、图像恢复网络训练方法、装置和存储介质
WO2023231794A1 (zh) 一种神经网络参数量化方法和装置
WO2024002211A1 (zh) 一种图像处理方法及相关装置
CN120092246A (zh) 神经网络训练方法及装置、图像处理方法及装置
CN115409697A (zh) 一种图像处理方法及相关装置
CN113066018A (zh) 一种图像增强方法及相关装置
CN113674149B (zh) 一种基于卷积神经网络的新型超分辨率重建方法
CN116342389B (zh) 基于多尺度增强模块的遥感影像超分辨率重建方法及系统
CN114612296A (zh) 用于高分辨率图像重建的方法、设备和计算机可读存储介质
US20240232573A1 (en) Neural network architecture for a systolic processor array and method of processing data using a neural network
WO2024245216A1 (zh) 一种图像处理方法及其相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19781788

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19781788

Country of ref document: EP

Kind code of ref document: A1