WO2020177607A1 - 图像去噪方法和装置 - Google Patents

图像去噪方法和装置 Download PDF

Info

Publication number
WO2020177607A1
WO2020177607A1 PCT/CN2020/076928 CN2020076928W WO2020177607A1 WO 2020177607 A1 WO2020177607 A1 WO 2020177607A1 CN 2020076928 W CN2020076928 W CN 2020076928W WO 2020177607 A1 WO2020177607 A1 WO 2020177607A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processed
resolution
feature
denoising
Prior art date
Application number
PCT/CN2020/076928
Other languages
English (en)
French (fr)
Inventor
宋风龙
刘浏
汪涛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP20767267.6A priority Critical patent/EP3923233A4/en
Publication of WO2020177607A1 publication Critical patent/WO2020177607A1/zh
Priority to US17/462,176 priority patent/US20210398252A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • This application relates to the field of computer vision, and more specifically, to an image denoising method and device.
  • Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. Vividly speaking, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data.
  • computer vision uses various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information.
  • the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.
  • This application provides an image denoising method and device to improve the effect of image denoising.
  • an image denoising method which is characterized in that it comprises: acquiring K images according to the image to be processed; acquiring image characteristics of the image to be processed according to the K images; performing processing on the image to be processed according to the image characteristics of the image to be processed Denoising processing to get the denoising processed image.
  • the above K images are images obtained after the resolution of the image to be processed is reduced, and K is a positive integer.
  • the above K images include the first image to the Kth image, and the image to be processed is the K+1th image.
  • the first image to the K+1th image include the i-th image and the i+1th image.
  • the image features of the i+1th image are extracted from the image features of the i-th image.
  • the resolution of each image is greater than the resolution of the i-th image, and i is a positive integer less than or equal to K.
  • the resolution of the images increases in sequence, wherein the resolution of the first image is the lowest, and the resolution of the K+1th image is the highest.
  • the above-mentioned acquiring image features of the image to be processed based on K images may refer to extracting high-resolution image features from low-resolution image features, and finally obtain the image features of the image to be processed (the resolution of the image to be processed is The image with the highest resolution among the above K+1 images).
  • the above-mentioned acquiring image features of the image to be processed based on K images includes: Step 1: acquiring image features of the i-th image; Step 2: extracting the image features of the i+1-th image based on the image features of the i-th image Image feature: Repeat steps 1 and 2 above to get the image feature of the K+1 image.
  • the image feature of the first image can be obtained directly by convolution processing on the first image, and for the second image to the K+1
  • the image features of the image can be extracted based on the image features of the previous image.
  • the image feature extraction of the high-resolution image is guided by the image feature of the low-resolution image, and the global information of the image to be processed can be perceived as much as possible during the image feature extraction process of the high-resolution image, so that the extraction
  • the image features of the obtained high-resolution images are more accurate, and ultimately make the denoising effect of image denoising according to the image features of the image to be processed better.
  • the image feature of the i+1th image is extracted according to the image feature of the i-th image, including: using the first one of the N convolutional layers Convolutional layer to the nth convolutional layer to convolve the i+1th image to obtain the initial image feature of the i+1th image; compare the initial image feature of the i+1th image with the i-th image feature
  • the image features of the image are fused to obtain the fused image features; the n+1th convolutional layer to the Nth convolutional layer of the N convolutional layers are used to convolution process the fused image features to obtain the first Image characteristics of i+1 images.
  • n and N are both positive integers, n is less than or equal to N, and N is the total number of convolutional layers used when extracting the image feature of the i+1th image.
  • the initial image features of the i+1th image can be obtained as soon as possible, and the initial image features of the i+1th image and the image features of the i-th image can be merged as soon as possible, so that The image feature of the finally obtained i+1th image is more accurate.
  • n 1 above.
  • the obtained i+1th image can be The initial image feature of each image is fused with the image feature of the i-th image, so that the finally obtained image feature of the i+1-th image is more accurate.
  • acquiring K images according to the image to be processed includes: performing K downsampling operations on the image to be processed respectively to obtain the above-mentioned 1 image to the Kth image.
  • the resolution of the image to be processed can be reduced, and the image information contained in the first image to the K-th image can be reduced, thereby reducing the amount of calculation during feature extraction.
  • the image to be processed is reorganized K times to obtain the first image to the Kth image whose resolution and number of channels are different from the image to be processed. image.
  • the recombination operation here is equivalent to adjusting the resolution and the number of channels of the image to be processed to obtain an image whose resolution and number of channels are both different from the original image to be processed.
  • the resolution and the number of channels of any one image from the first image to the Kth image are different from the image to be processed.
  • the resolution of the i-th image among the K images obtained by the above recombination operation is lower than the resolution of the image to be processed, and the number of channels of the i-th image is based on the number of channels of the image to be processed and the resolution of the i-th image And the resolution of the image to be processed is determined.
  • the number of channels of the i-th image may be determined according to the number of channels of the image to be processed and the ratio of the resolution of the i-th image to the resolution of the image to be processed.
  • the number of channels of the i-th image may be determined according to the number of channels of the image to be processed, and the ratio of the resolution of the image to be processed to the resolution of the i-th image.
  • the ratio of A to B refers to the value of A/B. Therefore, the ratio of the resolution of the i-th image to the resolution of the image to be processed is a value obtained by dividing the value of the resolution of the i-th image by the value of the resolution of the image to be processed.
  • the first image to the Kth image are obtained, and the image information can be retained when obtaining low-resolution images from the image to be processed, so that more accurate image features can be extracted during feature extraction.
  • the resolution of the first image to the Kth image described above may be preset.
  • the resolution of the image to be processed is M ⁇ N, and two lower-resolution images need to be recombined. Then, the resolution of the two images can be M/2 ⁇ N/2 and M/4 respectively ⁇ N/4.
  • the ratio of the number of channels of the i-th image to the number of channels of the image to be processed is less than or equal to the ratio of the resolution of the i-th image to the resolution of the image to be processed ratio.
  • the number of channels of the i-th image is Ci
  • the number of channels of the image to be processed is C
  • the resolution of the i-th image is Mi ⁇ Ni
  • the resolution of the image to be processed is M ⁇ N
  • the The ratio of the number of channels to the number of channels of the image to be processed is Ci/C
  • the ratio of the resolution of the i-th image to the resolution of the image to be processed is (Mi ⁇ Ni)/(M ⁇ N).
  • the image can be maintained when the i-th image is obtained from the image to be processed
  • the content remains unchanged, so that the image feature of the i-th image obtained by extraction is more accurate (compared to the situation where the image content is lost, the image feature extracted in this way is more accurate).
  • acquiring the image features of the first image to the K+1 image includes: acquiring the image features of the K+1 image of the first image by using a neural network.
  • the aforementioned neural network may be a convolutional neural network, a deep convolutional neural network, or a recurrent neural network.
  • the aforementioned neural network includes a top-level sub-network, an intermediate sub-network, and a bottom-level sub-network.
  • using the neural network to obtain the image features of the first image to the K+1 image includes: using the top-level sub-network to obtain the image features of the first image; using the intermediate sub-network to obtain the second to the K Image features of each image; use the underlying sub-network to obtain the image features of the K+1th image.
  • the above-mentioned top-level sub-network is used to process images with the lowest resolution
  • the above-mentioned intermediate sub-network is used to process images with intermediate resolution
  • the above-mentioned bottom-level sub-network is used to process images with the highest resolution.
  • the number of the above-mentioned intermediate sub-networks is K-1
  • these K-1 intermediate sub-networks are used to process the second image to the K-th image
  • each intermediate sub-network is used to process the corresponding image to obtain the The image characteristics of the image.
  • performing denoising processing on the image to be processed according to the image characteristics of the image to be processed to obtain the denoising processed image includes: convolving the image characteristics of the image to be processed Processing to obtain the residual estimated value of the image to be processed; superimposing the residual estimated value of the image to be processed with the image to be processed to obtain a denoising processed image.
  • an image denoising device which includes a module for executing the method in the first aspect.
  • an image denoising device in a third aspect, includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the first Method in one aspect.
  • a computer-readable medium stores program code for device execution, and the program code includes the method for executing the method in the first aspect.
  • a computer program product containing instructions, which when the computer program product runs on a computer, causes the computer to execute the method in the first aspect.
  • a chip in a sixth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes the method in the first aspect.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute the instructions stored in the memory.
  • the processor is used to execute the method in the first aspect.
  • an electronic device which includes the action recognition device in any one of the second to fourth aspects.
  • FIG. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of image denoising according to a CNN model provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of an image denoising method according to an embodiment of the present application.
  • Figure 5 is a schematic diagram of each sub-network in the neural network extracting image features
  • Figure 6 is a schematic diagram of the structure of the residual network
  • FIG. 7 is a schematic block diagram of an image denoising device according to an embodiment of the present application.
  • FIG. 8 is a schematic block diagram of an image denoising device according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a process of image denoising performed by the image denoising device of an embodiment of the present application.
  • FIG. 10 is a schematic diagram of the hardware structure of a neural network training device according to an embodiment of the present application.
  • FIG. 11 is a schematic diagram of the hardware structure of an image denoising device according to an embodiment of the present application.
  • the image denoising method provided by the embodiments of this application can be applied to photographing, video recording, safe city, automatic driving, human-computer interaction, and scenes that need to process images, display images, and perform low-level or high-level visual processing on images, such as image recognition, images Classification, semantic segmentation, video semantic analysis, video behavior recognition, etc.
  • the image denoising method of the embodiment of the present application can be applied to a photographing scene and a scene based on image and video visual computing.
  • the image denoising method of the embodiments of the present application can be used to remove noise in the pictures obtained during or after taking pictures.
  • the image quality can be improved, the image display effect can be improved, and the accuracy of the image-based visual algorithm can be improved.
  • the noise of the image will affect the recognition effect of the image to a certain extent.
  • the denoising method of the embodiment of the present application to perform denoising processing on the image during the image recognition process or before the image recognition officially starts, the quality of the image can be improved, thereby improving the effect of subsequent image recognition.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer.
  • the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • DNN can be understood as a neural network with many hidden layers. There is no special metric for "many” here.
  • the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated in terms of the work of each layer.
  • it is the following linear relationship expression: among them, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also a lot.
  • DNN The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • the input layer has no W parameter.
  • more hidden layers make the network more capable of portraying complex situations in the real world. Theoretically speaking, a model with more parameters is more complex and has a greater "capacity", which means it can complete more complex learning tasks.
  • Training a deep neural network is also a process of learning a weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W of many layers).
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be seen as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or convolution feature map.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units.
  • Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Sharing weight can be understood as the way to extract image information has nothing to do with location. The underlying principle is that the statistical information of a certain part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. Therefore, the image information obtained by the same learning can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally, the more the number of convolution kernels, the richer the image information reflected by the convolution operation.
  • the convolution kernel can be initialized in the form of a matrix of random size. During the training of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.
  • RNN Recurrent Neural Networks
  • the specific form is that the network will memorize the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layer are no longer unconnected but connected, and the input of the hidden layer includes not only The output of the input layer also includes the output of the hidden layer at the previous moment.
  • RNN can process sequence data of any length.
  • the training of RNN is the same as the training of traditional CNN or DNN.
  • the error backpropagation algorithm is also used, but there is a difference: that is, if the RNN is network expanded, then the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above.
  • the output of each step depends not only on the current step of the network, but also on the state of the previous steps of the network. This learning algorithm is called backpropagation through time (BPTT).
  • BPTT backpropagation through time
  • Convolutional neural networks can use backpropagation (BP) algorithms to modify the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial super-resolution model are updated by backpropagating the error loss information, so that the error loss is converged.
  • the backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal super-resolution model parameters, such as a weight matrix.
  • the pixel value of the image can be a red-green-blue (RGB) color value, and the pixel value can be a long integer representing the color.
  • the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness.
  • the pixel values can be grayscale values.
  • an embodiment of the present application provides a system architecture 100.
  • the data collection device 160 is used to collect training data.
  • the training data includes the original image (here, the original image may be an image with little noise) and the noise after adding noise to the original image.
  • Image where the original image can be an image that contains little noise.
  • the data collection device 160 stores the training data in the database 130, and the training device 120 trains to obtain the target model/rule 101 based on the training data maintained in the database 130.
  • the training device 120 processes the input original image and compares the output image with the original image until the output image of the training device 120 differs from the original image. The difference is less than a certain threshold, thereby completing the training of the target model/rule 101.
  • Above object models / rules 101 can be used to implement the present application denoising method of the embodiment, i.e., the model of the target image to be processed / pre-related rules 101 by the input image can be obtained after denoising.
  • the target model/rule 101 in the embodiment of the present application may specifically be a neural network.
  • the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices.
  • the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of Examples.
  • the target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 1.
  • the execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, Notebook computers, augmented reality (AR) AR/virtual reality (VR), vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 112 through the client device 140.
  • the input data in this embodiment of the application may include: the image to be processed input by the client device.
  • the preprocessing module 113 and the preprocessing module 114 are used for preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112.
  • the preprocessing module 113 and the preprocessing module may not be provided.
  • 114 there may only be one preprocessing module, and the calculation module 111 is directly used to process the input data.
  • the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing .
  • the data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.
  • the I/O interface 112 returns the processing result, such as the denoising processed image obtained as described above, to the client device 140 to provide it to the user.
  • the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide the user with the desired result.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112.
  • the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140.
  • the user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure.
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure.
  • the data is stored in the database 130.
  • Fig. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
  • the target model/rule 101 is obtained by training according to the training device 120.
  • the target model/rule 101 may be the neural network in the present application in the embodiment of the application.
  • the neural network provided in the embodiment of the present application Can be CNN, deep convolutional neural networks (deep convolutional neural networks, DCNN), recurrent neural networks (recurrent neural network, RNNS) and so on.
  • CNN is a very common neural network
  • the structure of CNN will be introduced in detail below in conjunction with Figure 2.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • a deep learning architecture refers to a machine learning algorithm. Multi-level learning is carried out on the abstract level of
  • CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.
  • a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (the pooling layer is optional), and a neural network layer 230.
  • CNN convolutional neural network
  • the convolutional layer/pooling layer 220 shown in Figure 2 may include layers 221-226 as shown in the examples.
  • layer 221 is a convolutional layer
  • layer 222 is a pooling layer
  • layer 223 is a convolutional layer
  • Layers, 224 is the pooling layer
  • 225 is the convolutional layer
  • 226 is the pooling layer
  • 221 and 222 are the convolutional layers
  • 223 is the pooling layer
  • 224 and 225 are the convolutional layers.
  • Layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 221 can include many convolution operators.
  • the convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image.
  • the multiple weight matrices have the same size (row ⁇ column), and the feature maps extracted by the multiple weight matrices of the same size have the same size, and then the multiple extracted feature maps of the same size are combined to form a convolution operation. Output.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .
  • the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; with the convolutional neural network
  • the features extracted by the subsequent convolutional layers (for example, 226) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.
  • the pooling layer can be a convolutional layer followed by a layer
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 200 After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 200 needs to use the neural network layer 230 to generate one or a group of required classes of output. Therefore, the neural network layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. 2) and an output layer 240. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image recognition, image classification, image super-resolution reconstruction and so on.
  • the output layer 240 After the multiple hidden layers in the neural network layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240.
  • the output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.
  • FIG. 3 is a chip hardware structure provided by an embodiment of the application, and the chip includes a neural network processor 50.
  • the chip may be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101.
  • the algorithms of each layer in the convolutional neural network as shown in Figure 2 can be implemented in the chip as shown in Figure 3.
  • the neural network processor NPU 50 NPU is mounted on the host CPU as a coprocessor, and the host CPU distributes tasks.
  • the core part of the NPU is the arithmetic circuit 50.
  • the controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 503 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.
  • the arithmetic circuit fetches the corresponding data of matrix B from the weight memory 502 and buffers it on each PE in the arithmetic circuit.
  • the arithmetic circuit fetches matrix A data and matrix B from the input memory 501 to perform matrix operations, and the partial or final result of the obtained matrix is stored in an accumulator 508.
  • the vector calculation unit 507 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on.
  • the vector calculation unit 507 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 507 can store the processed output vector in the unified buffer 506.
  • the vector calculation unit 507 may apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 507 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 503, for example for use in subsequent layers in a neural network.
  • the unified memory 506 is used to store input data and output data.
  • the weight data directly transfers the input data in the external memory to the input memory 501 and/or the unified memory 506 through the storage unit access controller 505 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 502, And the data in the unified memory 506 is stored in the external memory.
  • DMAC direct memory access controller
  • the bus interface unit (BIU) 510 is used to implement interaction between the main CPU, the DMAC, and the fetch memory 509 through the bus.
  • An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504;
  • the controller 504 is configured to call the instructions cached in the memory 509 to control the working process of the computing accelerator.
  • the unified memory 506, the input memory 501, the weight memory 502, and the instruction fetch memory 509 are all on-chip (On-Chip) memories, and the external memory is a memory external to the NPU.
  • the external memory can be a double data rate synchronous dynamic random access Memory (double data rate synchronous dynamic random access memory, referred to as DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • DDR SDRAM double data rate synchronous dynamic random access Memory
  • HBM high bandwidth memory
  • each layer in the convolutional neural network shown in FIG. 2 can be executed by the arithmetic circuit 303 or the vector calculation unit 307.
  • the execution device 110 in FIG. 1 described above can execute each step of the image denoising method of the embodiment of the present application.
  • the CNN model shown in FIG. 2 and the chip shown in FIG. 3 can also be used to execute the implementation of the embodiment of the present application.
  • the steps of the image denoising method The image denoising method according to the embodiment of the present application will be described in detail below with reference to the accompanying drawings.
  • FIG. 4 is a schematic flowchart of an image denoising method according to an embodiment of the present application.
  • the method shown in FIG. 4 may be executed by an image denoising device, where the image denoising device may be an electronic device with image processing functions.
  • the electronic device may specifically be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of Things device or other devices capable of image processing.
  • the method shown in FIG. 4 includes steps 101 to 103, which will be described in detail below.
  • the above K images are images obtained by reducing the resolution of the image to be processed, or the K images are images obtained by reducing the resolution of the image to be processed, and K is a positive integer.
  • the aforementioned resolution reduction processing may specifically be a recombination operation or a down-sampling operation.
  • the above K images and images to be processed may be numbered.
  • the K images may be numbered from the first image to the Kth image, and the images to be processed may be numbered K+1 Images, the above K images and the image to be processed form K+1 images.
  • the resolution of the images increases in order (in the above K+1 images, the larger the image number, the The resolution is also greater), the first image is the image with the lowest resolution among the above K+1 images, and the K+1 image is the image with the highest resolution among the above K+1 images.
  • the resolution of the i-th image is less than the resolution of the i+1-th image, where i is less than or equal to Positive integer of K.
  • K images refer to images obtained by reducing the resolution of the image to be processed
  • K+1 images include K images and images to be processed.
  • the image feature of the i+1 image in the K+1 images is obtained by extracting the image feature of the i image in the K+1 images.
  • acquiring the image characteristics of the image to be processed according to the K images includes:
  • Step 1 Obtain the image feature of the i-th image
  • Step 2 Extract the image feature of the i+1th image according to the image feature of the i-th image.
  • the image features of the first image can be obtained by convolution processing on the first image, and then the second image can be extracted based on the image features of the first image.
  • the image features of each image are carried out in a similar process until the image features of the K+1 image are extracted from the image features of the Kth image.
  • the image feature of the lower resolution image is used to guide the extraction of the image feature of the higher resolution image, and the image feature extraction process of the higher resolution image can perceive as much as possible of the image to be processed.
  • Global information so as to extract more accurate image features for higher resolution images.
  • the image features of the low-resolution image may no longer be saved, thereby saving Certain storage overhead.
  • performing denoising processing on the image to be processed according to the image feature of the image to be processed to obtain the denoising processed image includes: performing convolution processing on the image feature of the image to be processed to obtain the residual estimation value of the image to be processed; The residual estimation value of the image to be processed is superimposed with the image to be processed to obtain a denoising processed image.
  • the image feature extraction of the high-resolution image is guided by the image feature of the low-resolution image, and the global information of the image to be processed can be perceived as much as possible during the image feature extraction process of the high-resolution image, so that the extraction
  • the image features of the obtained high-resolution images are more accurate, and ultimately make the denoising effect of image denoising according to the image features of the image to be processed better.
  • the following takes the i+1th image as an example to describe the image feature acquisition process of the i+1th image in detail.
  • acquiring the image feature of the (i+1)th image includes: extracting the image feature of the (i+1)th image according to the image feature of the i-th image.
  • extracting the image feature of the i+1th image according to the image feature of the i-th image includes: using the first convolutional layer to the nth convolutional layer pair i+1th Convolution processing on images to obtain the initial image features of the i+1th image; fuse the initial image features of the i+1th image with the image features of the i-th image to obtain the fused image features; use N
  • the n+1th convolutional layer to the Nth convolutional layer in the convolutional layer perform convolution processing on the fused image features to obtain the image feature of the i+1th image.
  • n and N are both positive integers, n is less than or equal to N, and N is the total number of convolutional layers used when extracting the image feature of the i+1th image.
  • the initial image features of the i+1th image can be obtained as soon as possible, and the initial image features of the i+1th image and the image features of the i-th image can be merged as soon as possible, so that The image feature of the finally obtained i+1th image is more accurate.
  • n 1 above.
  • the obtained i+1th image can be The initial image feature of each image is fused with the image feature of the i-th image, so that the finally obtained image feature of the i+1-th image is more accurate.
  • down-sampling When performing resolution reduction processing on the image to be processed, down-sampling, recombination operations, etc. may be used to obtain the K images.
  • acquiring K images according to the image to be processed includes: performing K downsampling operations on the image to be processed respectively to obtain the foregoing 1 image to the Kth image.
  • the resolution of the image to be processed can be reduced, and the image content of the first image to the Kth image can be reduced, thereby reducing the amount of calculation during feature extraction.
  • the image to be processed can be down-sampled once to obtain K images, and the K-th image is copied, and then the copied K-th image is down-sampled to obtain the K-1 images, follow a similar process until the first image is obtained.
  • acquiring K images according to the image to be processed includes: performing K recombination operations on the image to be processed respectively to obtain the first image to the Kth image.
  • the recombination operation here is equivalent to adjusting the resolution and the number of channels of the image to be processed to obtain an image whose resolution and number of channels are both different from the original image to be processed.
  • the resolution and the number of channels of any one image from the first image to the Kth image are different from the image to be processed.
  • the resolution of the i-th image among the K images obtained by the above recombination operation is lower than the resolution of the image to be processed, and the number of channels of the i-th image is based on the number of channels of the image to be processed and the resolution of the i-th image And the resolution of the image to be processed is determined.
  • the number of channels of the i-th image may be determined according to the number of channels of the image to be processed and the ratio of the resolution of the i-th image to the resolution of the image to be processed.
  • the first image to the Kth image are obtained, and the image information can be retained when obtaining low-resolution images from the image to be processed, so that more accurate image features can be extracted during feature extraction.
  • the number of K and the resolution of each image in the K images may be preset, so that the image to be processed can be reduced.
  • the resolution of the image to be processed is M ⁇ N
  • two images can be generated according to the image to be processed, and the resolution of the two images can be M/2 ⁇ N/2 and M/4 ⁇ N/ 4.
  • the ratio of the number of channels of the i-th image to the number of channels of the image to be processed is less than or equal to the ratio of the resolution of the i-th image to the resolution of the image to be processed.
  • the number of channels of the i-th image is Ci
  • the number of channels of the image to be processed is C
  • the resolution of the i-th image is Mi ⁇ Ni
  • the resolution of the image to be processed is M ⁇ N
  • the The ratio of the number of channels to the number of channels of the image to be processed is Ci/C
  • the ratio of the resolution of the i-th image to the resolution of the image to be processed is (Mi ⁇ Ni)/(M ⁇ N).
  • the image can be maintained when the i-th image is obtained from the image to be processed
  • the content remains unchanged, so that the extracted image feature of the i-th image is more accurate (compared to the situation where the image content is lost, the extracted image feature is more accurate).
  • the specification of the image to be processed is M ⁇ N ⁇ C (the resolution is M ⁇ N, the number of channels is C), and the specification of the i-th image can be M/2 i ⁇ N/2 i ⁇ 4 i C (resolution The rate is M/2 i ⁇ N/2 i and the number of channels is 4 i C), that is, the resolution of the image to be processed is 4 i times the resolution of the i-th image, and the channel of the i-th image
  • the number is 4 i times the number of channels of the image to be processed, which is exactly the same as the multiple of the resolution of the image to be processed and the resolution of the i-th image, so that the image content of the i-th image and the image to be processed are consistent, avoiding the image
  • the loss of information can extract more accurate image features.
  • the above example is based on the case where the resolution of the image to be processed is changed by a multiple of 2.
  • the resolution of the image can also be other multiples ( For example, 3, 4, etc.) make changes, which are not limited here.
  • acquiring the image features of the first image to the K+1 image includes: acquiring the image features of the K+1 image of the first image by using a neural network.
  • the aforementioned neural network can be CNN, DCNN, RNNS, and so on.
  • the aforementioned neural network may include top sub-networks, middle sub-networks, and bottom sub-networks.
  • the image features of the image to be processed can be obtained according to the neural network, but also the image to be processed can be denoised according to the image features of the image to be processed to obtain a denoising processed image.
  • the neural network here may be a neural network trained using training data, and the training data here may include the original image and the noise image obtained by adding noise to the original image.
  • the output image is obtained by inputting the noise image to the neural network and denoising the noise image.
  • the output image is compared with the original image, and the difference between the output image and the original image is less than the preset
  • the neural network parameter corresponding to the threshold is determined as the final parameter of the neural network, and then the neural network can be used to execute the image denoising method of the embodiment of the present application.
  • the foregoing using a neural network to obtain image features of the first image to K+1 images includes: using a top-level sub-network to obtain image features of the first image; using an intermediate sub-network to obtain the second to Kth images The image features of the image; the underlying sub-network is used to obtain the image features of the K+1th image.
  • the top-level sub-network can be denoted as f 1 ( ⁇ ), and the top-level sub-network is used to process the first image to obtain the image characteristics of the first image;
  • the bottom-level sub-network can be denoted as f k+1 ( ⁇ ), and the top-level sub-network is used to process the K+1th image to obtain the image features of the K+1th image.
  • the number of the above-mentioned top-level sub-networks and the number of the bottom-level sub-networks are both 1, and the number of intermediate sub-networks is K-1, and each intermediate sub-network is used to process the corresponding image to obtain the image characteristics of the image.
  • the top-level sub-network only the image features of the first image need to be extracted, while for the intermediate sub-network and the bottom-level sub-network, in addition to extracting the image features of the image, the lower resolution The image features are fused with the higher resolution image features to finally obtain the image features of the image.
  • the neural network in Figure 5 includes a top-level sub-network, a bottom-level sub-network, and two intermediate sub-networks.
  • the structure or composition of these sub-networks is as follows:
  • Top-level sub-network consists of 1 residual network (the residual network in this application can also be referred to as residual block) and 2 convolutional activation layers;
  • Intermediate sub-network consists of 1 residual network and 3 convolutional activation layers
  • the underlying sub-network consists of a convolutional activation layer and a convolutional layer with a channel number of 1.
  • the convolutional activation layer is composed of a convolutional layer with a channel number of C and an activation layer, the number of the intermediate sub-networks is 2, and the structures of the two intermediate sub-networks are the same.
  • top-level sub-network, intermediate sub-network, and bottom-level sub-network may all be relatively complete neural networks including input layers...output layers, etc.
  • the specific structure of each of the above-mentioned top-level sub-network, intermediate sub-network, and bottom-level sub-network may be as shown in the convolutional neural network (CNN) 200 in FIG. 2.
  • CNN convolutional neural network
  • the above residual network can be considered as a special deep neural network.
  • the residual network can be: in addition to layer-by-layer connection between multiple hidden layers in a deep neural network, for example, the first hidden layer is connected to the second hidden layer, and the second hidden layer is connected The third layer of hidden layer, the third layer of hidden layer is connected to the fourth layer of hidden layer (this is a neural network data operation path, can also be vividly called neural network transmission), the residual network also has a straight line Connected branch, this direct branch is directly connected from the hidden layer of the first layer to the hidden layer of the fourth layer, that is, the processing of the second and third hidden layers is skipped, and the first hidden layer The data is directly transmitted to the 4th hidden layer for calculation.
  • the bottom sub-network processes the image features from the intermediate sub-network, and the number of feature maps between each sub-network can satisfy a certain preset relationship .
  • the number of feature maps of the bottom-level sub-network is c 0
  • each sub-network in FIG. 5 is only an example, and this application does not limit the specific structure of each sub-network.
  • the second intermediate sub-network extracts image features:
  • the image to be processed can be denoised according to the image features of the fourth image to obtain a denoising processed image.
  • the image features of lower-resolution images can be considered at the initial stage of extracting image features of higher-resolution images. , So as to extract a high-resolution image to extract more accurate image features.
  • the number of feature maps corresponding to the initial image feature is 4, and the image features of the 3 images correspond to The number of feature maps is also 4. Then, after connecting the initial image features of the fourth image and the image features of the third image, the number of corresponding feature maps becomes 8. That's it. Next, when processing the connected image features to obtain the image features of the fourth image, the number of feature maps should be adjusted so that the number of feature maps corresponding to the image features of the fourth image is also 4. .
  • the image feature of the image to be processed may be convolved first to obtain the residual estimation value of the image to be processed, and then the image to be processed The residual estimation value of the image is superimposed with the image to be processed to obtain an image after denoising processing.
  • the following describes the process of obtaining the denoising processed image according to the residual estimation value of the image to be processed with reference to FIG. 5.
  • the underlying sub-network includes 4 convolutional activation layers and 1 convolutional layer (the convolutional layer with channel number 1 is shown in Figure 5), and the underlying sub-network is denoised
  • the specific process of the post image is as follows:
  • the first convolutional activation layer processes the fourth image to obtain the initial image characteristics of the fourth image
  • the second convolutional activation layer to the fourth convolutional activation layer process the fused image features to obtain the image features of the fourth image
  • the convolutional layer performs convolution processing on the image features of the fourth image to obtain the residual estimation value of the fourth image
  • the fourth image is equivalent to the image to be processed above.
  • the image characteristics of the image to be processed can be obtained, and the residual of the image to be processed can be obtained through the above process (4).
  • the difference estimated value can be obtained by the above process (5) after the denoising process.
  • the residual network in the top-level sub-network and the intermediate sub-network in FIG. 5 can be a residual network that skips g (g can be an integer greater than or equal to 1) convolutional layers, and the bottom sub-network Can not use residual network.
  • the structure of the residual network can be as shown in Figure 6.
  • the residual network includes 3 convolutional layers (CONV-C, where CONV-C means that the number of channels is C Convolutional layer) and 2 activation function layers (ReLU), that is, a total of 3 convolutional layers are skipped in the residual network.
  • CONV-C convolutional layers
  • ReLU activation function layers
  • the residual network has two processing procedures for the first intermediate image feature.
  • the first intermediate image feature directly skips the 3 convolutional layers and 2 activation function layers in Figure 6 (equivalent to not processing the first intermediate image feature), and the result is still the first intermediate image feature;
  • One is to input the features of the first intermediate image into the 3 convolutional layers and 2 activation function layers in FIG. 6 for processing to obtain the residual estimation value of the first intermediate image.
  • the first intermediate image feature obtained by the two processing procedures and the residual estimated value of the first intermediate image are superimposed to obtain the second intermediate image feature.
  • the second intermediate image feature may also continue to perform convolution processing in the top-level sub-network or the intermediate sub-network in FIG. 5 to obtain the image feature of the image corresponding to the corresponding sub-network.
  • the image denoising method of the embodiment of the present application is described in detail above with reference to FIGS. 4 to 6, and the image denoising device of the embodiment of the present application is described below in conjunction with FIG. 7. It should be understood that the various steps in the method shown in FIG. 4 can be executed by the image denoising device shown in FIG. 7.
  • the above description and definition of the image denoising method also apply to the image denoising method shown in FIG. Noise device, the following description of the image denoising device shown in FIG. 7 will appropriately omit the repeated description.
  • Fig. 7 is a schematic block diagram of an image denoising device according to an embodiment of the present application.
  • the image denoising device 600 shown in FIG. 7 includes:
  • the acquiring module 601 is configured to acquire K images according to the image to be processed, the K images are images obtained after the resolution of the image to be processed are reduced, and K is a positive integer, where the K images include the first Image to the Kth image, and the image to be processed is the K+1th image;
  • the acquisition module 601 is further configured to acquire the image features of the image to be processed according to the K images, the image feature of the i+1th image is extracted according to the image feature of the i-th image, and the i-th image
  • the resolution of the +1 image is greater than the resolution of the i-th image, where the first image to the K+1-th image include the i-th image and the i+1-th image Image, i is a positive integer less than or equal to K;
  • the denoising module 602 is configured to perform denoising processing on the to-be-processed image according to the image characteristics of the to-be-processed image to obtain a denoising-processed image.
  • the image feature extraction of the high-resolution image is guided by the image feature of the low-resolution image, and the global information of the image to be processed can be perceived as much as possible during the image feature extraction process of the high-resolution image, so that the extraction
  • the image features of the obtained high-resolution images are more accurate, and ultimately make the denoising effect of image denoising according to the image features of the image to be processed better.
  • the acquisition module 601 and the denoising module 602 in the image denoising device 600 described above are modules divided according to logical functions.
  • the image can also be denoised according to the specific process of image denoising performed by the image denoising device 600.
  • the device 600 is divided into other functional modules.
  • Fig. 8 is a schematic block diagram of an image denoising device according to an embodiment of the present application.
  • the image denoising device 700 includes a recombination module 701 (used to obtain images of different resolutions through a recombination operation), an image feature extraction module 702a, several image feature extraction and fusion modules 702b, and feature applications Module 703.
  • the recombination module 701, the image feature extraction module 702a, and several image feature extraction and fusion modules 702b in the image denoising device 700 are equivalent to the acquisition module 601 in the image denoising device 600, and the feature application module 703 is equivalent to image denoising The denoising module 602 in the device 600.
  • the image denoising device 700 since the image denoising device 700 obtains images with different resolutions through a recombination operation, the image denoising device 700 includes a recombination module 701. If the image denoising device 700 obtains images with different resolutions through down-sampling operations, then the image denoising device 700 may include a down-sampling module 701.
  • the image denoising device 700 obtains images with different resolutions by reorganizing the input image, then extracts the image features of the low-resolution image, and transfers the image features from the low-resolution image to the image feature level by level.
  • the input image self-guided method is used to make full use of a large amount of context information to achieve efficient multi-level image information merging and achieve better denoising effects.
  • the function of each module in the image denoising device in Figure 8 during the entire denoising process is described below:
  • Recombination module use shuffle operation to restructure the input image to obtain several tensor images with different resolutions and channels. For example, an input image of size M x N and number of channels C (dimensions denoted as M x N x C) can be transformed into an image of size M/2 x N/2 and the number of channels 4C (dimension denoted as M /2 x N/2 x 4C). It should be understood that when the recombination operation is performed on the input image, the inter-resolution of the image is reduced, but by increasing the number of channels of the image, all the information of the input image is still retained in the low-resolution image after the resolution is reduced.
  • Feature extraction module For the image with the lowest resolution, only the image features of the image need to be extracted without feature fusion. Therefore, the feature extraction module can be used to extract the image features of the image with the lowest resolution.
  • the feature extraction and fusion module is used to use concatenate operation (concatenate) to merge the image features extracted by the previous small resolution hierarchical network with the initial features extracted by the current hierarchical network for the larger resolution image, and finally Get the image characteristics of the image at this level.
  • concatenate concatenate
  • Feature application module Apply the final features to the input image to get the final output.
  • the final output image is the image obtained after the input image denoising processing.
  • FIG. 9 is a schematic diagram of a process of image denoising performed by the image denoising device according to an application embodiment.
  • the input image is recombined by the recombination module, and the first image to the K+1th image are obtained.
  • the resolution of the first image is the smallest
  • the resolution of the K+1 image is the largest
  • the K+1 image is the input image.
  • the module corresponding to each image can be used to process the image to obtain the image characteristics of the image.
  • the feature extraction module of the first image is used to extract the image features of the first image
  • the feature extraction and fusion module of the second image is used to extract the image features of the second image based on the image features of the first image
  • the feature extraction and fusion module of the Kth image is used to extract the image features of the Kth image based on the image features of the K-1th image
  • the feature extraction and fusion module of the K+1th image is used to The image feature of each image is extracted from the image feature of the K+1th image.
  • the residual error estimation value of the K+1 image can be obtained according to the feature application module, and then the residual error estimation value is superimposed with the input image to obtain the output image.
  • the output image is the image after denoising processing.
  • FIG. 10 is a schematic diagram of the hardware structure of a neural network training device provided by an embodiment of the present application.
  • the neural network training device 800 shown in FIG. 10 includes a memory 801, a processor 802, a communication interface 803, and a bus 804.
  • the memory 801, the processor 802, and the communication interface 803 realize the communication connection between each other through the bus 804.
  • the memory 801 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 801 may store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute each step of the neural network training method of the embodiment of the present application.
  • the processor 802 may adopt a general-purpose central processing unit (central processing unit, CPU), microprocessor, application specific integrated circuit (application specific integrated circuit, ASIC), graphics processing unit (graphics processing unit, GPU), or one or more
  • the integrated circuit is used to execute related programs to implement the functions required by the units in the neural network training device of the embodiment of the present application, or execute the neural network training method of the method embodiment of the present application.
  • the processor 802 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the neural network training method of the present application can be completed by hardware integrated logic circuits in the processor 802 or instructions in the form of software.
  • the aforementioned processor 802 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices , Discrete gates or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • Discrete gates or transistor logic devices discrete hardware components.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801, and combines its hardware to complete the functions required by the units included in the neural network training device of the embodiment of the present application, or perform the functions of the method embodiment of the present application. Training method of neural network.
  • the communication interface 803 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 800 and other devices or a communication network.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 800 and other devices or a communication network.
  • the training data (such as the original image in the embodiment of the present application and the noise image obtained by adding noise to the original image) can be obtained through the communication interface 803.
  • the bus 804 may include a path for transferring information between various components of the device 800 (for example, the memory 801, the processor 802, and the communication interface 803).
  • FIG. 11 is a schematic diagram of the hardware structure of an image denoising device according to an embodiment of the present application.
  • the image denoising apparatus 900 shown in FIG. 11 includes a memory 901, a processor 902, a communication interface 903, and a bus 904.
  • the memory 901, the processor 902, and the communication interface 903 implement communication connections between each other through the bus 904.
  • the memory 901 may be ROM, static storage device and RAM.
  • the memory 901 may store a program. When the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are used to execute each step of the image denoising method in the embodiment of the present application.
  • the processor 902 may adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits to execute related programs to realize the functions required by the units in the image denoising device of the embodiment of the present application. , Or execute the image denoising method in the method embodiment of this application.
  • the processor 902 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image denoising method in the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 902 or instructions in the form of software.
  • the aforementioned processor 902 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 901, and the processor 902 reads the information in the memory 901, and combines its hardware to complete the functions required by the units included in the image denoising device of the embodiment of the application, or execute the image of the method embodiment of the application. Denoising method.
  • the communication interface 903 uses a transceiving device such as but not limited to a transceiver to implement communication between the device 900 and other devices or a communication network.
  • a transceiving device such as but not limited to a transceiver to implement communication between the device 900 and other devices or a communication network.
  • the training data can be obtained through the communication interface 903.
  • the bus 904 may include a path for transferring information between various components of the device 900 (for example, the memory 901, the processor 902, and the communication interface 903).
  • the acquisition module 601 and the denoising module 602 in the image denoising device 600 are equivalent to the processor 902.
  • the devices 800 and 900 shown in FIG. 10 and FIG. 11 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the devices 800 and 900 also include implementations. Other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatuses 800 and 900 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the devices 800 and 900 may also only include the necessary components for implementing the embodiments of the present application, and not necessarily all the components shown in FIG. 10 or FIG. 11.
  • the device 800 is equivalent to the training device 120 in 1
  • the device 900 is equivalent to the execution device 110 in FIG. 1.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本申请提供了图像去噪方法和装置。涉及人工智能领域，具体涉及计算机视觉领域。该方法包括：对待处理图像进行降分辨率处理，得到多个分辨率低于待处理图像的图像，然后根据分辨率较低的图像的图像特征来提取分辨率较高的图像的图像特征，进而得到待处理图像的图像特征，接下来再根据待处理图像的图像特征对待处理图像进行去噪处理，从而得到去噪处理后的图像。本申请能够提高图像去噪的效果。

Description

图像去噪方法和装置
本申请要求于2019年03月01日提交中国专利局、申请号为201910156951.0、申请名称为“图像去噪方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机视觉领域，并且更具体地，涉及一种图像去噪方法和装置。
背景技术
计算机视觉是各个应用领域，如制造业、检验、文档分析、医疗诊断，和军事等领域中各种智能/自主系统中不可分割的一部分，它是一门关于如何运用照相机/摄像机和计算机来获取我们所需的，被拍摄对象的数据与信息的学问。形象地说，就是给计算机安装上眼睛(照相机/摄像机)和大脑(算法)用来代替人眼对目标进行识别、跟踪和测量等，从而使计算机能够感知环境。因为感知可以看作是从感官信号中提取信息，所以计算机视觉也可以看作是研究如何使人工系统从图像或多维数据中“感知”的科学。总的来说，计算机视觉就是用各种成像系统代替视觉器官获取输入信息，再由计算机来代替大脑对这些输入信息完成处理和解释。计算机视觉的最终研究目标就是使计算机能像人那样通过视觉观察和理解世界，具有自主适应环境的能力。
在计算机视觉领域内，常常需要利用成像设备获取数字图像，并对数字图像进行识别或分析。其中，数字图像在数字化过程和传输过程中常受到成像设备和外部环境噪声的干扰，形成含噪图像或者噪声图像。含有噪声的图像会影响图像的显示效果、以及图像的分析和识别，因此，如何较好地去除图像中的噪声是一个需要解决的问题。
发明内容
本申请提供一种图像去噪方法和装置，以提高图像去噪的效果。
第一方面，提供一种图像去噪方法，其特征在于，包括：根据待处理图像获取K个图像；根据K个图像获取待处理图像的图像特征；根据待处理图像的图像特征对待处理图像进行去噪处理，得到去噪处理后的图像。
其中，上述K个图像是待处理图像降低分辨率后得到的图像，K为正整数。
上述K个图像包括第1个图像至第K个图像，待处理图像为第K+1个图像。第1个图像至第K+1个图像包括第i个图像和第i+1个图像，第i+1个图像的图像特征是根据第i个图像的图像特征提取得到的，第i+1个图像的分辨率大于第i个图像的分辨率，i为小于或者等于K的正整数。
应理解，从上述第1个图像至第K+1个图像，图像的分辨率是依次升高的，其中，第1个图像的分辨率最低，第K+1个图像的分辨率最高。
另外，上述根据K个图像获取待处理图像的图像特征，可以是指根据低分辨率的图像特征来提取高分辨率的图像特征，最终得到待处理图像的图像特征(待处理图像的分辨率是上述K+1个图像中的分辨率最高的图像)。
可选地，上述根据K个图像获取待处理图像的图像特征，包括：步骤1：获取第i个图像的图像特征；步骤2：根据第i个图像的图像特征提取第i+1个图像的图像特征；重复上述步骤1和步骤2，以得到第K+1个图像的图像特征。
应理解，在上述过程中，对于第1个图像来说，可以直接通过对第1个图像进行卷积处理来得到第1个图像的图像特征，而对于第2个图像至第K+1个图像来说，都可以根据上一个图像的图像特征来提取该图像的图像特征。
本申请中，通过低分辨率图像的图像特征来指导高分辨率图像的图像特征提取，能够在高分辨率图像的图像特征提取过程中尽可能多的感知到待处理图像的全局信息，使得提取得到的高分辨率图像的图像特征更加准确，最终使得根据待处理图像的图像特征进行图像去噪时的去噪效果更好。
结合第一方面，在第一方面的某些实现方式中，第i+1个图像的图像特征是根据第i个图像的图像特征提取得到的，包括：采用N个卷积层中的第1个卷积层至第n个卷积层对第i+1个图像进行卷积处理，得到第i+1个图像的初始图像特征；将第i+1个图像的初始图像特征与第i个图像的图像特征融合，得到融合后的图像特征；采用N个卷积层中的第n+1个卷积层至第N个卷积层对融合后的图像特征进行卷积处理，以得到第i+1个图像的图像特征。
其中，n和N均为正整数，n小于或者等于N，N为提取第i+1个图像的图像特征时用到的卷积层的总数。
当上述n的数值越小时，越能尽早得到第i+1个图像的初始图像特征，并尽早的将第i+1个图像的初始图像特征与第i个图像的图像特征进行融合，从而使得最后得到的第i+1个图像的图像特征更加准确。
可选地，上述n＝1。
当上述n＝1时，通过对第i+1个图像进行一次卷积处理(进行一次卷积处理之后得到第i+1个图像的初始图像特征)之后，就可以将得到的第i+1个图像的初始图像特征与第i个图像的图像特征进行融合，使得最终得到的第i+1个图像的图像特征更加准确。
结合第一方面，在第一方面的某些实现方式中，根据待处理图像获取K个图像，包括：对待处理图像分别进行K次下采样操作，得到上述1个图像至第K个图像。
通过下采样操作，能够降低待处理图像的分辨率，并且能够减少第1个图像至第K个图像包含的图像信息，从而能够减少特征提取时的运算量。
结合第一方面，在第一方面的某些实现方式中，对待处理图像分别进行K次重组操作，以得到分辨率和通道数均与所述待处理图像不同的第1个图像至第K个图像。
这里的重组操作相当于对待处理图像的分辨率和通道数进行调整，以得到分辨率和通道数均与原来的待处理图像均不相同的图像。
其中，上述第1个图像至第K个图像中的任意1个图像的分辨率和通道数均与待处理图像不同。上述重组操作得到的K个图像中的第i个图像的分辨率低于待处理图像的分辨率，第i个图像的通道数是根据待处理图像的通道数，以及第i个图像的分辨率和待处理 图像的分辨率确定的。
具体地，上述第i个图像的通道数可以是根据待处理图像的通道数，以及第i个图像的分辨率与待处理图像的分辨率的比值确定的。或者，述第i个图像的通道数可以是根据待处理图像的通道数，以及待处理图像的分辨率与第i个图像的分辨率的比值确定的。
应理解，在本申请中，A与B的比值是指A/B的值。因此，上述第i个图像的分辨率与待处理图像的分辨率的比值为第i个图像的分辨率的数值除以待处理图像的分辨率的数值得到的值。
通过重组操作，得到第1个图像至第K个图像，能够在根据待处理图像得到低分辨率的图像时可以保留图像信息，从而在特征提取时能够提取得到较为准确的图像特征。
应理解，上述第1个图像至第K个图像的分辨率可以是预设的。
例如，待处理图像的分辨率为M×N，需要通过重组操作得到2个分辨率较低的图像，那么，这2个图像的分辨率可以分别为M/2×N/2和M/4×N/4。
结合第一方面，在第一方面的某些实现方式中，第i个图像的通道数与待处理图像的通道数的比值小于或者等于第i个图像的分辨率与待处理图像的分辨率的比值。
假设第i个图像的通道数为Ci，待处理图像的通道数为C，第i个图像的分辨率为Mi×Ni，待处理图像的分辨率为M×N，那么，第i个图像的通道数与待处理图像的通道数的比值为Ci/C，第i个图像的分辨率与待处理图像的分辨率的比值为(Mi×Ni)/(M×N)。
当第i个图像的通道数与待处理图像的通道数的比值等于第i个图像的分辨率与待处理图像的分辨率的比值时，能够使得根据待处理图像得到第i个图像时保持图像内容不变，使得提取得到的第i个图像的图像特征更加准确(相对于图像内容丢失的情况，这种方式提取到的图像特征更加准确)。
应理解，上述根据K个图像获取待处理图像的图像特征时，需要获取第1个图像第K+1个图像的图像特征。
可选地，上述获取第1个图像至K+1个图像的图像特征，包括：采用神经网络获取第1个图像第K+1个图像的图像特征。
其中，上述神经网络可以是卷积神经网络、深度卷积神经网络或循环神经网络。
可选地，上述神经网络包括顶层子网络、中间子网络和底层子网络。
可选地，上述采用神经网络获取第1个图像至第K+1个图像的图像特征，包括：采用顶层子网络获取第1个图像的图像特征；采用中间子网络获取第2个至第K个图像的图像特征；采用底层子网络获取第K+1个图像的图像特征。
上述顶层子网络用于处理分辨率最低的图像，上述中间子网络用于处理中间分辨率的图像，上述底层子网络用于处理分辨率最高的图像。上述中间子网络的个数为K-1个，这K-1个中间子网络用于处理第2个图像至第K个图像，其中，每个中间子网络用于处理对应的图像，得到该图像的图像特征。
结合第一方面，在第一方面的某些实现方式中，根据待处理图像的图像特征对待处理图像进行去噪处理，得到去噪处理后的图像，包括：对待处理图像的图像特征进行卷积处理，得到待处理图像的残差估计值；将待处理图像的残差估计值与待处理图像进行叠加，得到去噪处理后的图像。
第二方面，提供一种图像去噪装置，该装置包括用于执行第一方面中的方法的模块。
第三方面，提供一种图像去噪装置，该装置包括：存储器，用于存储程序；处理器，用于执行存储器存储的程序，当存储器存储的程序被执行时，所述处理器用于执行第一方面中的方法。
第四方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行第一方面中的方法。
第五方面，提供一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第一方面中的方法。
第六方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行第一方面中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面中的方法。
第七方面，提供一种电子设备，该电子设备包括上述第二方面至第四方面中的任意一个方面中的动作识别装置。
附图说明
图1是本申请实施例提供的系统架构的结构示意图；
图2是本申请实施例提供的根据CNN模型进行图像去噪的示意图；
图3是本申请实施例提供的一种芯片硬件结构示意图；
图4是本申请实施例的图像去噪方法的示意性流程图；
图5是神经网络中的各个子网络提取图像特征的示意图；
图6是残差网络的结构的示意图；
图7是本申请实施例的图像去噪装置的示意性框图；
图8是本申请实施例的图像去噪装置的示意性框图；
图9是本申请实施例的图像去噪装置进行图像去噪的过程的示意图；
图10是本申请实施例的神经网络训练装置的硬件结构示意图；
图11是本申请实施例的图像去噪装置的硬件结构示意图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
本申请实施例提供的图像去噪方法能够应用在拍照、录像、平安城市、自动驾驶、人机交互以及需要处理图像、显示图像、对图像进行低层或高层视觉处理的场景，如图像识别、图像分类、语义分割、视频语义分析、视频行为识别等。
具体而言，本申请实施例的图像去噪方法能够应用在拍照场景和基于图像和视频的视觉计算的场景中，下面分别对拍照场景和图像识别场景进行简单的介绍。
拍照场景：
在利用相机、终端设备或者其他智能电子设备进行拍照时，为了显示质量更好的图像，在拍照时或者拍照后可以采用本申请实施例的图像去噪方法来去除拍照得到的图像中的噪声。通过采用本申请实施例的图像去噪方法，能够提高图片质量，提升图片显示效果和 提高基于图像的视觉算法的准确度。
图像识别景：
随着人工智能技术的应用的范围越来越广，在很多情况下，需要对图像中的内容进行识别，在对图像进行识别时，图像的噪声会在一定程度上影响图像的识别效果。通过采用本申请实施例的去噪方法在图像识别过程中或者图像识别正式开始之前对图像进行去噪处理，能够提高图像的质量，从而提高后续的图像识别的效果。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例可能涉及的相关术语及神经网络等相关概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2020076928-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入。激活函数可以是sigmoid函数。神经网络是将许多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有很多层隐含层的神经网络，这里的“很多”并没有特别的度量标准。从DNN按不同层的位置划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2020076928-appb-000002
其中，
Figure PCTCN2020076928-appb-000003
是输入向量，
Figure PCTCN2020076928-appb-000004
是输出向量，
Figure PCTCN2020076928-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020076928-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2020076928-appb-000007
由于DNN层数多，则系数W和偏移向量
Figure PCTCN2020076928-appb-000008
的数量也就很多了。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020076928-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020076928-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器。该特征抽取器可以看作是滤波器，卷积过程可以看作是使用一个可训练的滤波器与一个输入的图像或者卷积特征平面(feature map)做卷积。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。这其中隐含的原理是：图像的某一部分的统计信息与其他部分是一样的。即意味着在某一部分学习的图像信息也能用在另一部分上。所以对于图像上的所有位置，都能使用同样的学习得到的图像信息。在同一卷积层中，可以使用多个卷积核来提取不同的图像信息，一般地，卷积核数量越多，卷积操作反映的图像信息越丰富。
卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)循环神经网络(recurrent neural networks，RNN)是用来处理序列数据的。在传统的神经网络模型中，是从输入层到隐含层再到输出层，层与层之间是全连接的，而对于每一层层内之间的各个节点是无连接的。这种普通的神经网络虽然解决了很多难题，但是却仍然对很多问题却无能无力。例如，你要预测句子的下一个单词是什么，一般需要用到前面的单词，因为一个句子中前后单词并不是独立的。RNN之所以称为循环神经网路，即一个序列当前的输出与前面的输出也有关。具体的表现形式为网络会对前面的信息进行记忆并应用于当前输出的计算中，即隐含层本层之间的节点不再无连接而是有连接的，并且隐含层的输入不仅包括输入层的输出还包括上一时刻隐含层的输出。理论上，RNN能够对任何长度的序列数据进行处理。对于RNN的训练和对传统的CNN或DNN的训练一样。同样使用误差反向传播算法，不过有一点区别：即，如果将RNN进行网络展开，那么其中的参数，如W，是共享的；而如上举例上述的传统神经网络却不是这样。并且在使用梯度下降算法中，每一步的输出不仅依赖当前步的网络，还依赖前面若干步网络的状态。该学习算法称为基于时间的反向传播算法(back propagation through time，BPTT)。
既然已经有了卷积神经网络，为什么还要循环神经网络？原因很简单，在卷积神经网络中，有一个前提假设是：元素之间是相互独立的，输入与输出也是独立的，比如猫和狗。但现实世界中，很多元素都是相互连接的，比如股票随时间的变化，再比如一个人说了：我喜欢旅游，其中最喜欢的地方是云南，以后有机会一定要去。这里填空，人类应该都知道是填“云南”。因为人类会根据上下文的内容进行推断，但如何让机器做到这一步？RNN就应运而生了。RNN旨在让机器像人一样拥有记忆的能力。因此，RNN的输出就需要依赖当前的输入信息和历史的记忆信息。
(5)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调 整权重向量让它预测低一些，不断的调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
(6)反向传播算法
卷积神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的超分辨率模型中参数的大小，使得超分辨率模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的超分辨率模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的超分辨率模型的参数，例如权重矩阵。
(7)像素值
图像的像素值可以是一个红绿蓝(RGB)颜色值，像素值可以是表示颜色的长整数。例如，像素值为256*Red+100*Green+76Blue，其中，Blue代表蓝色分量，Green代表绿色分量，Red代表红色分量。各个颜色分量中，数值越小，亮度越低，数值越大，亮度越高。对于灰度图像来说，像素值可以是灰度值。
如图1所示，本申请实施例提供了一种系统架构100。在图1中，数据采集设备160用于采集训练数据，本申请实施例中训练数据包括原始图像(这里的原始图像可以是包含噪声很少的图像)和在原始图像上加上噪声之后的噪声图像，其中，原始图像可以是包含噪声很少的图像。
在采集到训练数据之后，数据采集设备160将这些训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型/规则101。
下面对训练设备120基于训练数据得到目标模型/规则101进行描述，训练设备120对输入的原始图像进行处理，将输出的图像与原始图像进行对比，直到训练设备120输出的图像与原始图像的差值小于一定的阈值，从而完成目标模型/规则101的训练。
上述目标模型/规则101能够用于实现本申请实施例的 图像去噪方法，即，将待处理图像通过相关预处理后输入该目标模型/规则101，即可得到去噪处理后的图像。本申请实施例中的目标模型/规则101具体可以为神经网络。需要说明的是，在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型/规则101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备120训练得到的目标模型/规则101可以应用于不同的系统或设备中，如应用于图1所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)AR/虚拟现实(virtual reality，VR)，车载终端等，还可以是服务器或者云端等。在图1中，执行设备110配置输入/输出(input/output，I/O)接口112，用于与外部设备进行数据交互，用户可以通过客户设备140向I/O接口112输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待 处理图像。
预处理模块113和预处理模块114用于根据I/O接口112接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块113和预处理模块114(也可以只有其中的一个预处理模块)，而直接采用计算模块111对输入数据进行处理。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储系统150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统150中。
最后，I/O接口112将处理结果，如上述得到的去噪处理后的图像返回给客户设备140，从而提供给用户。
值得说明的是，训练设备120可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则101，该相应的目标模型/规则101即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在附图1中所示情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口112提供的界面进行操作。另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果，作为新的样本数据存入数据库130。
值得注意的是，附图1仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在附图1中，数据存储系统150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储系统150置于执行设备110中。
如图1所示，根据训练设备120训练得到目标模型/规则101，该目标模型/规则101在本申请实施例中可以是本申请中的神经网络，具体的，本申请实施例提供的神经网络可以CNN，深度卷积神经网络(deep convolutional neural networks,DCNN)，循环神经网络(recurrent neural network，RNNS)等等。
由于CNN是一种非常常见的神经网络，下面结合图2重点对CNN的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
如图2所示，卷积神经网络(CNN)200可以包括输入层210，卷积层/池化层220(其中池化层为可选的)，以及神经网络层230。下面对这些层的相关内容做详细介绍。
卷积层/池化层220：
卷积层：
如图2所示卷积层/池化层220可以包括如示例221-226层，举例来说：在一种实现中，221层为卷积层，222层为池化层，223层为卷积层，224层为池化层，225为卷积层，226为池化层；在另一种实现方式中，221、222为卷积层，223为池化层，224、225为卷积层，226为池化层。即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层221为例，介绍一层卷积层的内部工作原理。
卷积层221可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素……这取决于步长stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。不同的权重矩阵可以用来提取图像中不同的特征，例如一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的特征图的尺寸也相同，再将提取到的多个尺寸相同的特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络200进行正确的预测。
当卷积神经网络200有多个卷积层的时候，初始的卷积层(例如221)往往提取较多的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络200深度的加深，越往后的卷积层(例如226)提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，在如图2中220所示例的221-226各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的唯一目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
神经网络层230：
在经过卷积层/池化层220的处理后，卷积神经网络200还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层220只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络200需要利用神经网络层230来生成一个或者一组所需要的类的数量的输出。因此，在神经网络层230中可以包括多层隐含层(如图2所示的231、232至23n)以及输出层240，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像识别，图像分类，图像超分辨率重建等等。
在神经网络层230中的多层隐含层之后，也就是整个卷积神经网络200的最后层为输出层240，该输出层240具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络200的前向传播(如图2由210至240方向的传播为前向传播)完成，反向传播(如图2由240至210方向的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络200的损失，及卷积神经网络200通过输出层输出的结果和理想结果之间的误差。
需要说明的是，如图2所示的卷积神经网络200仅作为一种卷积神经网络的示例，在具体的应用中，卷积神经网络还可以以其他网络模型的形式存在。
图3为本申请实施例提供的一种芯片硬件结构，该芯片包括神经网络处理器50。该芯片可以被设置在如图1所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图1所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。如图2所示的卷积神经网络中各层的算法均可在如图3所示的芯片中得以实现。
神经网络处理器NPU 50NPU作为协处理器挂载到主CPU(host CPU)上，由主CPU分配任务。NPU的核心部分为运算电路50，控制器504控制运算电路503提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路503内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路503是二维脉动阵列。运算电路503还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路503是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器502中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器501中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)508中。
向量计算单元507可以对运算电路的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元507可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现种，向量计算单元能507将经处理的输出的向量存储到统一缓存器506。例如，向量计算单元507可以将非线性函数应用到运算电路503的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元507生成归一化的值、合并值，或二 者均有。在一些实现中，处理过的输出的向量能够用作到运算电路503的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器506用于存放输入数据以及输出数据。
权重数据直接通过存储单元访问控制器505(direct memory access controller，DMAC)将外部存储器中的输入数据搬运到输入存储器501和/或统一存储器506、将外部存储器中的权重数据存入权重存储器502，以及将统一存储器506中的数据存入外部存储器。
总线接口单元(bus interface unit，BIU)510，用于通过总线实现主CPU、DMAC和取指存储器509之间进行交互。
与控制器504连接的取指存储器(instruction fetch buffer)509，用于存储控制器504使用的指令；
控制器504，用于调用指存储器509中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器506，输入存储器501，权重存储器502以及取指存储器509均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，简称DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
其中，图2所示的卷积神经网络中各层的运算可以由运算电路303或向量计算单元307执行。
上文中介绍的图1中的执行设备110能够执行本申请实施例的图像去噪方法的各个步骤，图2所示的CNN模型和图3所示的芯片也可以用于执行本申请实施例的图像去噪方法的各个步骤。下面结合附图对本申请实施例的图像去噪方法进行详细的介绍。
图4是本申请实施例的图像去噪方法的示意性流程图。图4所示的方法可以由图像去噪装置来执行，这里的图像去噪装置可以是具有图像处理功能的电子设备。该电子设备具体可以是移动终端(例如，智能手机)，电脑，个人数字助理，可穿戴设备，车载设备，物联网设备或者其他能够进行图像处理的设备。
图4所示的方法包括步骤101至103，下面分别对这些步骤进行详细的描述。
101、根据待处理图像获取K个图像。
其中，上述K个图像是待处理图像降低分辨率后得到的图像，或者，上述K个图像是对待处理图像进行降低分辨率处理后得到的图像，K为正整数。
上述降低分辨率处理具体可以是重组操作或者下采样操作。
为了便于描述方案，可以对上述K个图像和待处理图像进行编号，具体地，可以将上述K个图像编号为第1个图像至第K个图像，将上述待处理图像编号为第K+1个图像，上述K个图像和待处理图像组成了K+1个图像。
其中，在上述K+1个图像中，按照从第1个图像到第K+1个图像的顺序，图像的分辨率依次升高(在上述K+1个图像中，图像的编号越大，分辨率也越大)，第1个图像是上述K+1个图像中分辨率最低的图像，第K+1个图像是上述K+1个图像中分辨率最高的图像。
也就是说，当上述K+1个图像包括第i个图像和第i+1个图像时，第i个图像的分辨 率小于第i+1个图像的分辨率，其中，i为小于或者等于K的正整数。
102、根据上述K个图像获取待处理图像的图像特征。
应理解，在本申请中，K个图像是指待处理图像降低分辨率后得到的图像，K+1个图像包括K个图像和待处理图像。
另外，上述K+1个图像中的第i+1个图像的图像特征是根据上述K+1个图像中的第i个图像的图像特征提取得到的。
可选地，上述根据K个图像获取待处理图像的图像特征，包括：
步骤1：获取第i个图像的图像特征；
步骤2：根据第i个图像的图像特征提取第i+1个图像的图像特征。
重复上述步骤1和步骤2，以得到第K+1个图像的图像特征。
在得到第K+1个图像的图像特征的过程中，可以先通过对第1个图像进行卷积处理来得到第1个图像的图像特征，然后根据第1个图像的图像特征来提取第2个图像的图像特征，按照类似的过程一直进行下去，直到根据第K个图像的图像特征提取到第K+1个图像的图像特征。
在上述过程中，通过较低分辨率图像的图像特征来指导较高分辨率图像的图像特征的提取，能够在较高分辨率图像的图像特征提取过程中尽可能多的感知到待处理图像的全局信息，从而为较高分辨率的图像提取出更加准确的图像特征。
应理解，在获取待处理图像的图像特征的过程中，在根据低分辨率图像的图像特征提取到高分辨率图像的图像特征后，可以不再保存低分辨率图像的图像特征，从而能够节省一定的存储开销。
例如，在根据第1个图像的图像特征提取到第2个图像的图像特征之后，可以不再保存第1个图像的图像特征，而只需要保存第2个图像的图像特征，在根据第2个图像的图像特征提取到第3个图像的图像特征之后，可以不再保护第2个图像的图像特征而只需要保存第3个图像的图像特征。按照类似的过程进行下去直到得到待处理图像的图像特征。也就是说，在根据K个图像获取待处理图像的图像特征的过程中的任一时刻，可以只保护当前分辨率较高的图像的图像特征，能够减少一定的存储开销。
103、根据待处理图像的图像特征对待处理图像进行去噪处理，得到去噪处理后的图像。
可选地，根据待处理图像的图像特征对待处理图像进行去噪处理，得到去噪处理后的图像，包括：对待处理图像的图像特征进行卷积处理，得到待处理图像的残差估计值；将待处理图像的残差估计值与待处理图像进行叠加，得到去噪处理后的图像。
本申请中，通过低分辨率图像的图像特征来指导高分辨率图像的图像特征提取，能够在高分辨率图像的图像特征提取过程中尽可能多的感知到待处理图像的全局信息，使得提取得到的高分辨率图像的图像特征更加准确，最终使得根据待处理图像的图像特征进行图像去噪时的去噪效果更好。
应理解，在根据K个图像获取待处理图像的图像特征的过程中，是需要先获取第1个图像的图像特征，然后根据第1个图像的图像特征获取第2个图像的图像特征，再根据第2个图像的图像特征获取第3个图像的图像特征，等等。在这个过程中，相当于需要获取第1个图像至第K+1个图像(待处理图像)的图像特征。
为了详细的了解图像特征的获取过程，下面以第i+1个图像为例来详细描述第i+1个图像的图像特征的获取过程。
可选地，获取第i+1个图像的图像特征，包括：根据第i个图像的图像特征提取第i+1个图像的图像特征。
具体地，根据第i个图像的图像特征提取第i+1个图像的图像特征，包括：采用N个卷积层中的第1个卷积层至第n个卷积层对第i+1个图像进行卷积处理，得到第i+1个图像的初始图像特征；将第i+1个图像的初始图像特征与第i个图像的图像特征融合，得到融合后的图像特征；采用N个卷积层中的第n+1个卷积层至第N个卷积层对融合后的图像特征进行卷积处理，以得到第i+1个图像的图像特征。
其中，n和N均为正整数，n小于或者等于N，N为提取第i+1个图像的图像特征时用到的卷积层的总数。
当上述n的数值越小时，越能尽早得到第i+1个图像的初始图像特征，并尽早的将第i+1个图像的初始图像特征与第i个图像的图像特征进行融合，从而使得最后得到的第i+1个图像的图像特征更加准确。
可选地，上述n＝1。
当上述n＝1时，通过对第i+1个图像进行一次卷积处理(进行一次卷积处理之后得到第i+1个图像的初始图像特征)之后，就可以将得到的第i+1个图像的初始图像特征与第i个图像的图像特征进行融合，使得最终得到的第i+1个图像的图像特征更加准确。
在对待处理图像进行降低分辨率处理时，可以采用下采样、重组操作等方式来得到上述K个图像。
可选地，根据待处理图像获取K个图像，包括：对待处理图像分别进行K次下采样操作，得到上述1个图像至第K个图像。
通过下采样操作，能够降低待处理图像的分辨率，并且能够减少第1个图像至第K个图像的图像内容，从而能够减少特征提取时的运算量。
另外，在通过下采样操作时，还可以先对待处理图像进行1次下采样操作得到K个图像，并复制第K个图像，然后再对复制后的第K个图像进行下采样操作，得到第K-1个图像，按照类似的过程，直到得到第1个图像。
但是在通过下采样操作得到低分辨率的图像时会导致图像信息损失，因此，为了减少图像内容的损失，还可以通过重组操作的方式来得到较高分辨率的图像。
可选地，根据待处理图像获取K个图像，包括：对待处理图像分别进行K次重组操作，得到第1个图像至第K个图像。
这里的重组操作相当于对待处理图像的分辨率和通道数进行调整，以得到分辨率和通道数均与原来的待处理图像均不相同的图像。
其中，上述第1个图像至第K个图像中的任意1个图像的分辨率和通道数均与待处理图像不同。上述重组操作得到的K个图像中的第i个图像的分辨率低于待处理图像的分辨率，第i个图像的通道数是根据待处理图像的通道数，以及第i个图像的分辨率和待处理图像的分辨率确定的。
具体地，上述第i个图像的通道数可以是根据待处理图像的通道数，以及第i个图像的分辨率与待处理图像的分辨率的比值确定的。
通过重组操作，得到第1个图像至第K个图像，能够在根据待处理图像得到低分辨率的图像时可以保留图像信息，从而在特征提取时能够提取得到较为准确的图像特征。
应理解，在本申请中，在根据待处理图像获取K个图像之前，K的数目以及K个图像中的每个图像的分辨率可以是预设好的，这样就可以通过对待处理图像进行降分辨率处理，以得到数目为K，且分辨率为预设分辨率的K个图像。
例如，待处理图像的分辨率为M×N，那么，可以根据待处理图像生成2个图像，并且这2个图像的分辨率可以分别为M/2×N/2和M/4×N/4。
可选地，第i个图像的通道数与待处理图像的通道数的比值小于或者等于第i个图像的分辨率与待处理图像的分辨率的比值。
假设第i个图像的通道数为Ci，待处理图像的通道数为C，第i个图像的分辨率为Mi×Ni，待处理图像的分辨率为M×N，那么，第i个图像的通道数与待处理图像的通道数的比值为Ci/C，第i个图像的分辨率与待处理图像的分辨率的比值为(Mi×Ni)/(M×N)。
当第i个图像的通道数与待处理图像的通道数的比值等于第i个图像的分辨率与待处理图像的分辨率的比值时，能够使得根据待处理图像得到第i个图像时保持图像内容不变，使得提取得到的第i个图像的图像特征更加准确(相对于图像内容丢失的情况提取到的图像特征更加准确)。
为了更好地理解待处理图像与K个图像的通道数和分辨率的对应关系，下面结合具体实例进行说明。
例如，待处理图像的规格为M×N×C(分辨率为M×N，通道数为C)，第i个图像的规格可以为M/2 i×N/2 i×4 iC(分辨率为M/2 i×N/2 i，通道数为4 iC)，也就是说，待处理图像的分辨率为第i个图像的分辨率的4 i倍，而第i个图像的通道数为待处理图像的通道数的4 i倍，刚好与待处理图像的分辨率与第i个图像的分辨率的倍数相等，使得第i个图像与待处理图像的图像内容保持一致，避免图像信息的损失，从而能够提取得到比较准确的图像特征。
应理解，上述例子是以待处理图像的分辨率以2的倍数进行变化的情况为例进行的说明，实际上在根据待处理图像得到第i个图像时，图像的分辨率还可以其它倍数(例如，3、4等等)进行变化，这里不做限定。
可选地，上述获取第1个图像至K+1个图像的图像特征，包括：采用神经网络获取第1个图像第K+1个图像的图像特征。
其中，上述神经网络可以是CNN，DCNN以及RNNS等等。
上述神经网络可以包括顶层子网络(top sub-network)、中间子网络(middle sub-networks)和底层子网络(bottom sub-networks)。
应理解，在本申请中，不仅可以根据神经网络来获取待处理图像的图像特征，还可以根据待处理图像的图像特征对所述待处理图像进行去噪处理，得到去噪处理后的图像。
这里的神经网络可以是采用训练数据训练得到的神经网络，这里的训练数据可以包括原始图像，以及在原始图像上加上噪声得到的噪声图像。在训练过程中，通过向神经网络输入噪声图像，并对噪声图像进行去噪处理，得到输出图像，将得到的输出图像与原始图像进行对比，并将输出图像与原始图像的差值小于预设阈值时对应的神经网络参数确定为该神经网络的最终参数，接下来，就可以采用该神经网络来执行本申请实施例的图像去噪 方法了。
这里的神经网络的具体结构，形态等信息可以参见上文中图1至图3中描述的相关内容。
可选地，上述采用神经网络获取第1个图像至K+1个图像的图像特征，包括：采用顶层子网络获取第1个图像的图像特征；采用中间子网络获取第2个至第K个图像的图像特征；采用底层子网络获取第K+1个图像的图像特征。
其中，顶层子网络可以记为f 1(·)，该顶层子网络用于处理第1个图像，以获取第1个图像的图像特征；
中间子网络可以记为{f k(·)} k＝2,...,K，该中间子网络用于处理第2个图像至第K个图像，以获取第2个图像至第K个图像的图像特征；
底层子网络可以记为f k+1(·)，该顶层子网络用于处理第K+1个图像，以获取第K+1个图像的图像特征。
应理解，上述顶层子网络和底层子网络的个数均为1，而中间子网络的个数为K-1个，每一个中间子网络用于处理对应的图像，得到该图像的图像特征。另外，对于顶层子网络来说只需要提取第1个图像的图像特征即可，而对于中间子网络和底层子网络来说，除了提取图像的图像特征之外，还需将较低分辨率的图像特征与较高分辨率的图像特征进行融合，以最终得到图像的图像特征。
下面结合图5对顶层子网络、中间子网络和底层子提取图像特征的过程进行简单的介绍。
图5中的神经网络包括一个顶层子网络、一个底层子网络以及两个中间子网络，这些子网络的结构或者构成如下：
顶层子网络：由1个残差网络(本申请中的残差网络还可以称为残差块)和2个卷积激活层构成；
中间子网络：由1个残差网络和3个卷积激活层构成；
底层子网络：由1个卷积激活层和1个通道数为1的卷积层构成。
其中，上述卷积激活层由通道数为C的卷积层以及激活层构成，上述中间子网络的数量为2，且这两个中间子网络的结构相同。
应理解，上述顶层子网络、中间子网络和底层子网络均可以是包含输入层…输出层等的相对完整的神经网络。上述顶层子网络、中间子网络和底层子网络中的每个子网络的具体结构均可以如图2中的卷积神经网络(CNN)200所示。
上述残差网络可以认为是一种特殊的深度神经网络。简单来说，残差网络可以是：深度神经网络中多个隐含层之间除了逐层相连之外，例如第1层隐含层连接第2层隐含层，第2层隐含层连接第3层隐含层，第3层隐含层连接第4层隐含层(这是一条神经网络的数据运算通路，也可以形象的称为神经网络传输)，残差网络还多了一条直连支路，这条直连支路从第1层隐含层直接连到第4层隐含层，即跳过第2层和第3层隐含层的处理，将第1层隐含层的数据直接传输给第4层隐含层进行运算。
另外，在本申请中，为了方便中间子网络处理来自顶层子网络的图像特征，底层子网络处理来自中间子网络的图像特征，各个子网络之间的特征图的数目可以满足一定的预设关系。
例如，假设底层子网络的特征图数目为c 0，则顶层子网络的特征图数目可以设置为C K＝2Kc 0。
应理解，图5中各个子网络的结构只是一种示例，本申请对各个子网络的具体结构并不限定。
图5所示的神经网络需要提取第1个图像至第4个图像(第4个图像为待处理图像)的图像特征(相当于K＝3的情况)，其中，按照第1个图像到第4个图像的顺序，图像的分辨率依次变大。
图5中的各个子网络提取图像特征的过程如下：
顶层子网络提取图像特征的过程：
通过顶层子网络对第1个图像进行卷积处理，得到第1个图像的图像特征；
第1个中间子网络提取图像特征的过程：
通过第1个中间子网络中的一个卷积激活层对第2个图像进行一次卷积处理，得到第2个图像的初始图像特征，接下来再将第2个图像的初始图像特征与第1个图像的图像特征连接(也可以称为融合)，最后再对连接后的图像特征再进行卷积处理，以得到第2个图像的图像特征；
第2个中间子网络提取图像特征的过程：
通过中间子网络中的一个卷积激活层对第3个图像进行一次卷积处理，得到第3个图像的初始图像特征，接下来再将第3个图像的初始图像特征与第2个图像的图像特征连接，最后再对连接后的图像特征再进行卷积处理，以得到第3个图像的图像特征；
底层子网络提取图像特征的过程：
通过底层子网络中的一个卷积激活层对第4个图像进行一次卷积处理，得到第4个图像的初始图像特征，接下来再将第4个图像的初始图像特征与第3个图像的图像特征连接，最后再对连接后的图像特征再进行卷积处理，以得到第4个图像的图像特征。
在图5中，当得到第4个图像的图像特征之后，就可以根据第4个图像的图像特征对待处理图像进行去噪处理，得到去噪处理后的图像了。
在本申请中，在根据低分辨率图像的图像特征提取高分辨率图像的图像特征时，可以在提取较高分辨率图像的图像特征的初始阶段就考虑到较低分辨率的图像的图像特征，从而提取为高分辨率的图像提取出更准确的图像特征。
应理解，在上述中间子网络和底层子网络进行特征提取时，当对初始图像特征和上一层的图像特征进行连接或者融合时，会导致特征图的数目增加，那么，接下来，需要将特征图的数目调整为原来的数目。
例如，在上述图5所示过程中，当底层子网络在得到第4个图像的初始图像特征时，如果该初始图像特征对应的特征图的数目为4，并且3个图像的图像特征对应的特征图的数目也为4，那么，在对第4个图像的初始图像特征和第3个图像的图像特征进行连接后得到的连接后的图像特征之后，对应的特征图的数目就变成8个了。接下来，在对连接后的图像特征进行处理以得到第4个图像的图像特征时，要对特征图的数目进行调整，使得第4个图像的图像特征对应的特征图的数目也为4个。
本申请中，在根据待处理图像的图像特征对待处理图像进行去噪处理时，具体可以先对待处理图像的图像特征进行卷积处理，得到待处理图像的残差估计值，然后再将待处理 图像的残差估计值与待处理图像进行叠加，得到去噪处理后的图像。
下面结合图5，对根据待处理图像的残差估计值得到去噪处理后的图像的过程进行描述。
例如，如图5所示，底层子网络包括4个卷积激活层和1个卷积层(图5中所示的为通道数为1的卷积层)，该底层子网络得到去噪处理后的图像的具体过程如下：
(1)第1个卷积激活层对第4个图像进行处理，得到第4个图像的初始图像特征；
(2)将第4个图像的初始图像特征与第3个图像的图像特征进行融合，得到融合后的图像特征；
(3)第2个卷积激活层至第4个卷积激活层对融合后的图像特征进行处理，得到第4个图像的图像特征；
(4)卷积层对第4个图像的图像特征进行卷积处理，得到第4个图像的残差估计值；
(5)将第4个图像的残差估计值与第4个图像进行叠加，得到去噪处理后的图像。
在上述过程中，第4个图像相当于上文中的待处理图像，通过上述过程(1)至(3)能够得到待处理图像的图像特征，通过上述过程(4)能够得到待处理图像的残差估计值，通过上述过程(5)能够得到去噪处理后的图像。
可选地，图5中的顶层子网络和中间子网络中的残差网络可以是跳过g(g可以为大于或者等于1的整数)个卷积层的残差网络，而在底层子网络中可以不使用残差网络。
通过在顶层子网络和中间子网络中采用跳过连接的残差网络，能够在卷积处理过程中更好地去噪(在卷积过程中去除噪声)。
例如，当g＝3时，残差网络的结构可以如图6所示，在图6中，残差网络包括3个卷积层(CONV-C，其中，CONV-C表示通道数为C的卷积层)和2个激活函数层(ReLU)，即残差网络中总共跳过3个卷积层。
下面对图6中的残差网络的主要工作流程进行描述。
如图6所示，假设输入的图像特征是第一中间图像特征(第一中间图像特征可以是图5中的顶层子网络或者中间子网络对融合后的图像特征进行卷积处理后得到的图像特征)，那么，残差网络对第一中间图像特征有两种处理流程。一种是第一中间图像特征直接跳过图6中的3个卷积层和2个激活函数层(相当于不对第一中间图像特征进行处理)，得到的仍然是第一中间图像特征；另一种是将第一中间图像特征输入到图6中的3个卷积层和2个激活函数层中进行处理，得到第一中间图像的残差估计值。
最后，将这两种处理流程得到的第一中间图像特征和第一中间图像的残差估计值进行叠加，得到第二中间图像特征。其中，该第二中间图像特征还可以在图5中的顶层子网络或者中间子网络中继续进行卷积处理，以得到相应子网络对应的图像的图像特征。
上文结合图4至图6对本申请实施例的图像去噪方法进行了详细的描述，下面结合图7对本申请实施例的图像去噪装置进行描述。应理解，上述图4所示的方法中的各个步骤可以由图7所示的图像去噪装置来执行，上文中对图像去噪方法的相关描述和限定也适用于图7所示的图像去噪装置，下面在描述图7所示的图像去噪装置时适当省略重复的描述。
图7是本申请实施例的图像去噪装置的示意性框图。图7所示的图像去噪装置600包括：
获取模块601，用于根据待处理图像获取K个图像，所述K个图像是所述待处理图像 降低分辨率后得到的图像，K为正整数，其中，所述K个图像包括第1个图像至第K个图像，所述待处理图像为第K+1个图像；
所述获取模块601还用于根据所述K个图像获取所述待处理图像的图像特征，第i+1个图像的图像特征是根据第i个图像的图像特征提取得到的，所述第i+1个图像的分辨率大于所述第i个图像的分辨率，其中，所述第1个图像至所述第K+1个图像包括所述第i个图像和所述第i+1个图像，i为小于或者等于K的正整数；
去噪模块602，用于根据所述待处理图像的图像特征对所述待处理图像进行去噪处理，得到去噪处理后的图像。
本申请中，通过低分辨率图像的图像特征来指导高分辨率图像的图像特征提取，能够在高分辨率图像的图像特征提取过程中尽可能多的感知到待处理图像的全局信息，使得提取得到的高分辨率图像的图像特征更加准确，最终使得根据待处理图像的图像特征进行图像去噪时的去噪效果更好。
应理解，上述图像去噪装置600中的获取模块601和去噪模块602是按照逻辑功能划分的模块，事实上，还可以根据图像去噪装置600进行图像去噪的具体处理过程将图像去噪装置600划分成其他的功能模块。
图8是本申请实施例的图像去噪装置的示意性框图。如图8所示，图像去噪装置700包括一个重组模块701(用于通过重组操作来得到不同分辨率的图像)和一个图像特征提取模块702a、若干个图像特征提取和融合模块702b以及特征应用模块703。
其中，图像去噪装置700中的重组模块701、图像特征提取模块702a以及若干个图像特征提取和融合模块702b相当于图像去噪装置600中的获取模块601，特征应用模块703相当于图像去噪装置600中的去噪模块602。
应理解，由于图像去噪装置700是通过重组操作得到不同分辨率的图像，因此，图像去噪装置700包含有重组模块701。而如果图像去噪装置700是通过下采样操作得到不同分辨率的图像，那么，图像去噪装置700可以包含下采样模块701。
在图8中，图像去噪装置700通过对输入图像进行重组，得到分辨率不同的图像，然后提取低分辨率图像的图像特征，并将从低分辨率图像提取到图像特征逐层级传递，以指导高分辨率图像的特征提取，这里通过输入图像自引导的方法，充分利用大量的上下文信息，实现高效的多级图像信息合并，能够取得更好的去噪效果。下面对图8中的图像去噪装置中的各个模块在整个去噪过程中的功能进行描述：
重组模块：利用重组(shuffle)操作，对输入图像进行结构重组，得到若干个不同分辨率和通道的张量(tensor)的图像。例如，可以将大小为M x N、通道数为C的输入图像(维度记为M x N x C)变换为大小为M/2 x N/2、通道数为4C的图像(维度记为M/2 x N/2 x 4C)。应理解，这里在针对输入图像进行重组操作时，降低了图像的间分辨率，但是通过增加图像的通道数，使得输入图像的所有信息仍然保留在降低分辨率后的小分辨率图像中。
特征提取模块：对于分辨率最低的图像来说，只需要提取该图像的图像特征，而不需要进行特征融合。因此，该特征提取模块可以用于提取分辨率最低的图像的图像特征。
若干个特征提取和融合模块：除了分辨率最低的图像之外，在获取其分辨率的图像特征时，既需要进行特征提取，也需要进行特征融合。
具体地，特征提取和融合模块用于利用串接操作(concatenate)，将上一个小分辨率层级网络提取的图像特征，与本层级网络针对较大分辨率图像所提取的初始特征进行合并，最终得到该层级的图像的图像特征。
特征应用模块：将最终得到的特征应用到输入图像，得到最终的输出，最终得到的输出图像就是输入图像去噪处理后得到的图像。
为了更好地理解图8中的图像去噪装置700进行图像去噪的过程，下面结合图9进行说明。
图9是申请实施例的图像去噪装置进行图像去噪的过程的示意图。
在图9中，通过重组模块对输入图像进行重组操作，得到了第1个图像至第K+1的图像。其中，第1个图像的分辨率最小，第K+1个图像的分辨率最大，第K+1个图像为输入图像。
在得到分辨率不同的图像之后，可以采用每个图像对应的模块对该图像进行处理，得到该图像的图像特征。
具体地，第1个图像的特征提取模块用于提取第1个图像的图像特征，第2个图像的特征提取和融合模块用于根据第1个图像的图像特征提取第2个图像的图像特征……第K个图像的特征提取和融合模块用于根据第K-1个图像的图像特征提取第K个图像的图像特征，第K+1个图像的特征提取和融合模块用于根据第K个图像的图像特征提取第K+1个图像的图像特征。
在得到第K+1个图像的图像特征之后，可以根据特征应用模块获取第K+1个图像的残差估计值，然后根据该残差估计值与输入图像进行叠加，从而得到输出图像，该输出图像就是去噪处理后的图像。
图10是本申请实施例提供的神经网络训练装置的硬件结构示意图。图10所示的神经网络训练装置800(该装置800具体可以是一种计算机设备)包括存储器801、处理器802、通信接口803以及总线804。其中，存储器801、处理器802、通信接口803通过总线804实现彼此之间的通信连接。
存储器801可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器801可以存储程序，当存储器801中存储的程序被处理器802执行时，处理器802和通信接口803用于执行本申请实施例的神经网络的训练方法的各个步骤。
处理器802可以采用通用的中央处理器(central processing unit，CPU)，微处理器，应用专用集成电路(application specific integrated circuit，ASIC)，图形处理器(graphics processing unit，GPU)或者一个或多个集成电路，用于执行相关程序，以实现本申请实施例的神经网络的训练装置中的单元所需执行的功能，或者执行本申请方法实施例的神经网络的训练方法。
处理器802还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请的神经网络的训练方法的各个步骤可以通过处理器802中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器802还可以是通用处理器、数字信号处理器(digital signal processing，DSP)、专用集成电路(ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组 件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器801，处理器802读取存储器801中的信息，结合其硬件完成本申请实施例的神经网络的训练装置中包括的单元所需执行的功能，或者执行本申请方法实施例的神经网络的训练方法。
通信接口803使用例如但不限于收发器一类的收发装置，来实现装置800与其他设备或通信网络之间的通信。例如，可以通过通信接口803获取训练数据(如本申请实施例中的原始图像和在原始图像上加上噪声后得到的噪声图像)。
总线804可包括在装置800各个部件(例如，存储器801、处理器802、通信接口803)之间传送信息的通路。
图11是本申请实施例的图像去噪装置的硬件结构示意图。图11所示的图像去噪装置900(该装置900具体可以是一种计算机设备)包括存储器901、处理器902、通信接口903以及总线904。其中，存储器901、处理器902、通信接口903通过总线904实现彼此之间的通信连接。
存储器901可以是ROM，静态存储设备和RAM。存储器901可以存储程序，当存储器901中存储的程序被处理器902执行时，处理器902和通信接口903用于执行本申请实施例的图像去噪方法的各个步骤。
处理器902可以采用通用的，CPU，微处理器，ASIC，GPU或者一个或多个集成电路，用于执行相关程序，以实现本申请实施例的图像去噪装置中的单元所需执行的功能，或者执行本申请方法实施例的图像去噪方法。
处理器902还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请实施例的图像去噪方法的各个步骤可以通过处理器902中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器902还可以是通用处理器、DSP、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器901，处理器902读取存储器901中的信息，结合其硬件完成本申请实施例的图像去噪装置中包括的单元所需执行的功能，或者执行本申请方法实施例的图像去噪方法。
通信接口903使用例如但不限于收发器一类的收发装置，来实现装置900与其他设备或通信网络之间的通信。例如，可以通过通信接口903获取训练数据。
总线904可包括在装置900各个部件(例如，存储器901、处理器902、通信接口903)之间传送信息的通路。
应理解，图像去噪装置600中的获取模块601和去噪模块602相当于处理器902。
应注意，尽管图10和图11所示的装置800和900仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，装置800和900还包括实现正常运行所必须的其他器件。同时，根据具体需要，本领域的技术人员应当理解，装置800和900还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，装置800和900也可仅仅包括实现本申请实施例所必须的器件，而不必包括图10或图11中所示的全部器件。
可以理解，所述装置800相当于1中的训练设备120，所述装置900相当于图1中的执行设备110。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (11)

  1. 一种图像去噪方法，其特征在于，包括：
    根据待处理图像获取K个图像，所述K个图像是所述待处理图像降低分辨率后得到的图像，K为正整数，其中，所述K个图像包括第1个图像至第K个图像，所述待处理图像为第K+1个图像；
    根据所述K个图像获取所述待处理图像的图像特征，其中，第i+1个图像的图像特征是根据第i个图像的图像特征提取得到的，所述第i+1个图像的分辨率大于所述第i个图像的分辨率，其中，所述第1个图像至所述第K+1个图像包括所述第i个图像和所述第i+1个图像，i为小于或者等于K的正整数；
    根据所述待处理图像的图像特征对所述待处理图像进行去噪处理，得到去噪处理后的图像。
  2. 如权利要求1所述的方法，其特征在于，所述第i+1个图像的图像特征是根据第i个图像的图像特征提取得到的，包括：
    采用N个卷积层中的第1个卷积层至第n个卷积层对所述第i+1个图像进行卷积处理，得到所述第i+1个图像的初始图像特征，其中，n和N均为正整数，n小于或者等于N，N为提取所述第i+1个图像的图像特征时采用的卷积层的总数；
    将所述第i+1个图像的初始图像特征与所述第i个图像的图像特征融合，得到融合后的图像特征；
    采用所述N个卷积层中的第n+1个卷积层至第N个卷积层对所述融合后的图像特征进行卷积处理，得到所述第i+1个图像的图像特征。
  3. 如权利要求1或2所述的方法，其特征在于，所述根据待处理图像获取K个图像，包括：
    对所述待处理图像分别进行K次重组操作，以得到分辨率和通道数均与所述待处理图像不同的所述第1个图像至第K个图像，其中，所述第i个图像的分辨率低于待处理图像的分辨率，所述第i个图像的通道数是根据所述待处理图像的通道数，以及所述第i个图像的分辨率与所述待处理图像的分辨率的比值确定的。
  4. 如权利要求3所述的方法，其特征在于，所述第i个图像的通道数与所述待处理图像的通道数的比值小于或者等于所述第i个图像的分辨率与所述待处理图像的分辨率的比值。
  5. 如权利要求1-4中任一项所述的方法，其特征在于，所述根据所述待处理图像的图像特征对所述待处理图像进行去噪处理，得到去噪处理后的图像，包括：
    对所述待处理图像的图像特征进行卷积处理，得到所述待处理图像的残差估计值；
    将所述待处理图像的残差估计值与所述待处理图像进行叠加，得到所述去噪处理后的图像。
  6. 一种图像去噪装置，其特征在于，包括：
    获取模块，用于根据待处理图像获取K个图像，所述K个图像是所述待处理图像降低分辨率后得到的图像，K为正整数，其中，所述K个图像包括第1个图像至第K个图 像，所述待处理图像为第K+1个图像；
    所述获取模块还用于根据所述K个图像获取所述待处理图像的图像特征，其中，第i+1个图像的图像特征是根据第i个图像的图像特征提取得到的，所述第i+1个图像的分辨率大于所述第i个图像的分辨率，其中，所述第1个图像至所述第K+1个图像包括所述第i个图像和所述第i+1个图像，i为小于或者等于K的正整数；
    去噪模块，用于根据所述待处理图像的图像特征对所述待处理图像进行去噪处理，得到去噪处理后的图像。
  7. 如权利要求6所述的装置，其特征在于，所述第i+1个图像的图像特征是根据所述第i个图像的图像特征提取得到的，包括：
    采用N个卷积层中的第1个卷积层至第n个卷积层对所述第i+1个图像进行卷积处理，得到所述第i+1个图像的初始图像特征，其中，n和N均为正整数，n小于或者等于N，N为提取所述第i+1个图像的图像特征时采用的卷积层的总数；
    将所述第i+1个图像的初始图像特征与所述第i个图像的图像特征融合，得到融合后的图像特征；
    采用所述N个卷积层中的第n+1个卷积层至第N个卷积层对所述融合后的图像特征进行卷积处理，以得到所述第i+1个图像的图像特征。
  8. 如权利要求6或7所述的装置，其特征在于，所述获取模块用于：
    对所述待处理图像分别进行K次重组操作，以得到分辨率和通道数均与所述待处理图像不同的所述第1个图像至第K个图像，其中，所述第i个图像的分辨率低于待处理图像的分辨率，所述第i个图像的通道数是根据所述待处理图像的通道数，以及所述第i个图像的分辨率与所述待处理图像的分辨率的比值确定的。
  9. 如权利要求8所述的装置，其特征在于，所述第i个图像的通道数与所述待处理图像的通道数的比值小于或者等于所述第i个图像的分辨率与所述待处理图像的分辨率的比值。
  10. 如权利要求6-9中任一项所述的装置，其特征在于，所述去噪模块用于：
    对所述待处理图像的图像特征进行卷积处理，得到所述待处理的残差估计值；
    将所述待处理图像的残差估计值与所述待处理图像进行叠加，得到所述去噪处理后的图像。
  11. 一种图像去噪装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行权利要求1-5中任一项所述的方法。
PCT/CN2020/076928 2019-03-01 2020-02-27 图像去噪方法和装置 WO2020177607A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20767267.6A EP3923233A4 (en) 2019-03-01 2020-02-27 METHOD AND APPARATUS FOR DENOOING PICTURES
US17/462,176 US20210398252A1 (en) 2019-03-01 2021-08-31 Image denoising method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910156951.0 2019-03-01
CN201910156951.0A CN109993707B (zh) 2019-03-01 2019-03-01 图像去噪方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/462,176 Continuation US20210398252A1 (en) 2019-03-01 2021-08-31 Image denoising method and apparatus

Publications (1)

Publication Number Publication Date
WO2020177607A1 true WO2020177607A1 (zh) 2020-09-10

Family

ID=67129944

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/076928 WO2020177607A1 (zh) 2019-03-01 2020-02-27 图像去噪方法和装置

Country Status (4)

Country Link
US (1) US20210398252A1 (zh)
EP (1) EP3923233A4 (zh)
CN (1) CN109993707B (zh)
WO (1) WO2020177607A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112990215A (zh) * 2021-03-04 2021-06-18 腾讯科技（深圳）有限公司 图像去噪方法、装置、设备及存储介质
US11540798B2 (en) 2019-08-30 2023-01-03 The Research Foundation For The State University Of New York Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109993707B (zh) * 2019-03-01 2023-05-12 华为技术有限公司 图像去噪方法和装置
CN112668366B (zh) * 2019-10-15 2024-04-26 华为云计算技术有限公司 图像识别方法、装置、计算机可读存储介质及芯片
CN111091503B (zh) * 2019-11-09 2023-05-02 复旦大学 基于深度学习的图像去失焦模糊方法
CN111402146B (zh) * 2020-02-21 2022-05-10 华为技术有限公司 图像处理方法以及图像处理装置
CN111402166A (zh) * 2020-03-18 2020-07-10 上海嘉沃光电科技有限公司 图像去噪方法及装置、服务终端及计算机可读存储介质
CN111582457A (zh) * 2020-05-11 2020-08-25 陈永聪 一种模仿人类记忆来实现通用机器智能的方法
US11887279B2 (en) * 2020-08-25 2024-01-30 Sharif University Of Technology Machine learning-based denoising of an image
CN111967538B (zh) * 2020-09-25 2024-03-15 北京康夫子健康技术有限公司 应用于小目标检测的特征融合方法、装置、设备以及存储介质
CN113591843B (zh) * 2021-07-12 2024-04-09 中国兵器工业计算机应用技术研究所 仿初级视觉皮层的目标检测方法、装置及设备

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108416748A (zh) * 2018-02-26 2018-08-17 阿博茨德（北京）科技有限公司 Jpeg压缩文档的图像预处理方法及装置
CN108492258A (zh) * 2018-01-17 2018-09-04 天津大学 一种基于生成对抗网络的雷达图像去噪方法
CN109993707A (zh) * 2019-03-01 2019-07-09 华为技术有限公司 图像去噪方法和装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7856150B2 (en) * 2007-04-10 2010-12-21 Arcsoft, Inc. Denoise method on image pyramid
JP5753791B2 (ja) * 2008-12-25 2015-07-22 メディック ビジョン イメージング ソルーション リミティッド ノイズ除去された所定の解像度の医療画像を提供する方法、所定の解像度のノイズ除去された所定の解像度の医療画像を提供するシステム
JP2011180798A (ja) * 2010-03-01 2011-09-15 Sony Corp 画像処理装置、および画像処理方法、並びにプログラム
CN105335947B (zh) * 2014-05-26 2019-03-01 富士通株式会社 图像去噪方法和图像去噪装置
CN106447616B (zh) * 2015-08-12 2021-10-08 中兴通讯股份有限公司 一种实现小波去噪的方法和装置
CN108604369B (zh) * 2016-07-27 2020-10-27 华为技术有限公司 一种去除图像噪声的方法、装置、设备及卷积神经网络
US10753997B2 (en) * 2017-08-10 2020-08-25 Siemens Healthcare Gmbh Image standardization using generative adversarial networks
CN107578377A (zh) * 2017-08-31 2018-01-12 北京飞搜科技有限公司 一种基于深度学习的超分辨率图像重建方法及系统
CN108805840B (zh) * 2018-06-11 2021-03-26 Oppo（重庆）智能科技有限公司 图像去噪的方法、装置、终端及计算机可读存储介质
US10929708B2 (en) * 2018-12-10 2021-02-23 International Business Machines Corporation Deep learning network for salient region identification in images

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108492258A (zh) * 2018-01-17 2018-09-04 天津大学 一种基于生成对抗网络的雷达图像去噪方法
CN108416748A (zh) * 2018-02-26 2018-08-17 阿博茨德（北京）科技有限公司 Jpeg压缩文档的图像预处理方法及装置
CN109993707A (zh) * 2019-03-01 2019-07-09 华为技术有限公司 图像去噪方法和装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11540798B2 (en) 2019-08-30 2023-01-03 The Research Foundation For The State University Of New York Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising
CN112990215A (zh) * 2021-03-04 2021-06-18 腾讯科技（深圳）有限公司 图像去噪方法、装置、设备及存储介质
CN112990215B (zh) * 2021-03-04 2023-12-12 腾讯科技（深圳）有限公司 图像去噪方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN109993707A (zh) 2019-07-09
EP3923233A4 (en) 2022-04-13
US20210398252A1 (en) 2021-12-23
CN109993707B (zh) 2023-05-12
EP3923233A1 (en) 2021-12-15

Similar Documents

Publication Publication Date Title
WO2020177607A1 (zh) 图像去噪方法和装置
WO2020177651A1 (zh) 图像分割方法和图像处理装置
CN110188795B (zh) 图像分类方法、数据处理方法和装置
WO2021043273A1 (zh) 图像增强方法和装置
WO2021043168A1 (zh) 行人再识别网络的训练方法、行人再识别方法和装置
WO2021164731A1 (zh) 图像增强方法以及图像增强装置
WO2021018163A1 (zh) 神经网络的搜索方法及装置
WO2021164234A1 (zh) 图像处理方法以及图像处理装置
WO2022001372A1 (zh) 训练神经网络的方法、图像处理方法及装置
WO2021018245A1 (zh) 图像分类方法及装置
CN112446380A (zh) 图像处理方法和装置
WO2022001805A1 (zh) 一种神经网络蒸馏方法及装置
WO2021063341A1 (zh) 图像增强方法以及装置
WO2021018251A1 (zh) 图像分类方法及装置
CN113065645B (zh) 孪生注意力网络、图像处理方法和装置
CN113011562A (zh) 一种模型训练方法及装置
WO2024002211A1 (zh) 一种图像处理方法及相关装置
CN113807183A (zh) 模型训练方法及相关设备
WO2022179606A1 (zh) 一种图像处理方法及相关装置
CN115131256A (zh) 图像处理模型、图像处理模型的训练方法及装置
CN110705564B (zh) 图像识别的方法和装置
WO2021042774A1 (zh) 图像恢复方法、图像恢复网络训练方法、装置和存储介质
WO2021057091A1 (zh) 视点图像处理方法及相关设备
US12008797B2 (en) Image segmentation method and image processing apparatus
WO2023028866A1 (zh) 图像处理方法、装置和车辆

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20767267

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020767267

Country of ref document: EP

Effective date: 20210909