WO2022116856A1 - 一种模型结构、模型训练方法、图像增强方法及设备 - Google Patents

一种模型结构、模型训练方法、图像增强方法及设备 Download PDF

Info

Publication number
WO2022116856A1
WO2022116856A1 PCT/CN2021/131704 CN2021131704W WO2022116856A1 WO 2022116856 A1 WO2022116856 A1 WO 2022116856A1 CN 2021131704 W CN2021131704 W CN 2021131704W WO 2022116856 A1 WO2022116856 A1 WO 2022116856A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature
blocks
neural network
module
feature blocks
Prior art date
Application number
PCT/CN2021/131704
Other languages
English (en)
French (fr)
Inventor
郭天宇
陈汉亭
王云鹤
许春景
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21899890.4A priority Critical patent/EP4242917A4/en
Publication of WO2022116856A1 publication Critical patent/WO2022116856A1/zh
Priority to US18/203,337 priority patent/US20230306719A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/771—Feature selection, e.g. selecting representative features from a multi-dimensional feature space
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the field of computer vision, and in particular, to a model structure, a model training method, an image enhancement method and device.
  • Computer vision is an integral part of various intelligent/autonomous systems in various application fields (such as manufacturing, inspection, document analysis, medical diagnosis, and military) The knowledge of acquiring the data and information of the subjects that people need. According to whether the semantic information of the image is needed, computer vision tasks can be divided into two categories: low-level visual tasks and high-level visual tasks.
  • Low-level visual tasks generally refer to pixel-level image processing tasks that do not need to use the semantic information of images, or Most of the underlying features (eg, image edges, textures, etc.) are used, and these tasks include image enhancement (eg, denoising, deblurring, deraining, super-resolution reconstruction, etc.), image encryption, etc.
  • High-level vision tasks need to use the semantic information of the image, and the extracted features are high-level features, such as target localization, recognition, detection, classification, segmentation, and image generation using semantic features.
  • CNN convolutional neural networks
  • CNN can flex its muscles in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks, and each image enhancement task pair needs to train the corresponding CNN, as shown in (b) in Figure 1.
  • image enhancement tasks denoising, dehazing, and deraining
  • 3 different CNNs need to be trained correspondingly, which is not universal.
  • the embodiments of the present application provide a model structure, a model training method, an image enhancement method and equipment, and a new model structure is obtained by combining a transformer module for processing natural language tasks with different neural network structures, which breaks through the limitation of the transformer module only.
  • the model structure can be applied to the underlying vision tasks
  • the model structure has multiple first neural network layers and multiple second neural network layers, different first/second neural network layers Corresponding to different image enhancement tasks, so that the model can be used to process different image enhancement tasks after training, and compared with the existing models for processing low-level visual tasks, most of them are based on CNN (CNN is an excellent feature extractor in high-level It can show its strengths in visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks). With the help of the transformer module, this model can pay attention to global information, which can improve the effect of image enhancement.
  • CNN is an excellent feature extractor in high-level It can show its strengths in visual tasks, but it is difficult to pay attention to global information when dealing with low-
  • the embodiments of the present application first provide a model structure, which can be used in the field of computer vision in the field of artificial intelligence.
  • the structure of the model includes: a selection module, m first neural network layers, and m second neural network layers , segmentation module, reorganization module and transformer module, each first neural network layer uniquely corresponds to a second neural network layer, each first neural network layer can also be called a head module or head structure, each second neural network layer A layer can also be called a tail module or tail structure, where m ⁇ 2.
  • the selection module is used for acquiring an input image and determining a first target neural network layer corresponding to the input image, where the first target neural network layer is one of the m first neural network layers.
  • the selection module of the model determines the first target neural network layer corresponding to the input image according to the input image, it will input the input image to the first target neural network layer, and the first target neural network layer is used for Feature extraction is performed on the input image to obtain a feature map (which may be referred to as a first feature map).
  • the obtained first feature map will be further input to the segmentation module, and the segmentation module is used to segment the first feature map to obtain n feature blocks (which can be called first feature blocks), n ⁇ 2 .
  • the segmentation module obtains n first feature blocks
  • the n first feature blocks are further input into the transformer module for processing, and the transformer module is used to generate a one-to-one correspondence with the n first feature blocks according to the relevant information n second feature blocks
  • the correlation information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, that is, each first feature block has its own
  • the feature information of other first feature blocks is also fused according to the correlation between itself and other first feature blocks.
  • the transformer module After the transformer module obtains n second feature blocks from n first feature blocks based on relevant information, it will be sent to the recombination module, which is used to splicing and recombining the n second feature blocks according to their relative spatial positions, so A second feature map with the same dimension as the input first feature map is obtained, and the operation of the recombination module is the inverse operation of the segmentation module.
  • the reorganization module splices and reorganizes the n second feature blocks to obtain a second feature map, and the second feature map is input into the second target neural network layer uniquely corresponding to the first target neural network layer.
  • the second target neural network The layer belongs to one of m second neural network layers.
  • the second target neural network layer is used to decode the second feature map to obtain an output image.
  • a new model structure is obtained by combining the transformer module for processing natural language tasks with different neural network structures, which breaks through the limitation that the transformer module can only be used for processing natural language tasks. It can be applied to the underlying vision tasks.
  • the model structure has multiple first neural network layers and multiple second neural network layers. Different first/second neural network layers correspond to different image enhancement tasks. It can be used to deal with different image enhancement tasks, and compared with the existing models for dealing with low-level visual tasks, most of them are based on CNN (CNN, as an excellent feature extractor, can show its strength in high-level visual tasks, but it is not suitable for processing low-level visual tasks. It is difficult to pay attention to the global information during the task), the model can pay attention to the global information with the help of the transformer module, which can improve the image enhancement effect.
  • CNN as an excellent feature extractor
  • the selection module after receiving the input image, the selection module will determine which first neural network layer should perform the feature extraction operation for the input image. Specifically, the selection module is used to first determine the input image Which type of image enhancement task the image belongs to, and then input the input image to the first neural network layer corresponding to the task.
  • the image enhancement task to which the input image belongs may be called the first image enhancement task.
  • the selection module is further configured to input the received input image into the first image enhancement task. a target neural network layer.
  • model selection module determines that the first target neural network layer corresponding to the input image is identified through the first image enhancement task, which is achievable.
  • the input image is the training sample in the training set, and each training sample will have a corresponding label indicating which training sample belongs to.
  • this label is used to indicate which first neural network layer the training sample should use to extract features.
  • the selection module of the model can determine that the training sample belongs to the first image enhancement task according to the label of the training sample.
  • the input image is the real target image to be processed.
  • the selection module will not only receive the input image, but also It will also receive an instruction from the device that deploys the model, which is used to indicate which type of image enhancement task the target image belongs to. That is, in the inference phase, the model's selection module is based on the received instruction. to determine whether the target image belongs to the first image enhancement task.
  • the transformer module includes an encoder and a decoder.
  • the transformer module generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which may be: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one, where the first related information is used to indicate the nth
  • the first correlation between any two first feature blocks in a feature block, and the dimensions of the n first feature blocks input by the encoder are consistent with the dimensions of the n third feature blocks; second related information, and according to the second related information, generate n second feature blocks corresponding to the n third feature blocks one-to-one, the second related information is used to indicate any one of the n third feature blocks
  • the second correlation degree between the two third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the first task code is integrated into the second related information, the first task code acts as an input to the decoder, and the first task code is the corresponding identifier of the first image enhancement task, which can also be considered as the first task code.
  • Each image enhancement task corresponds to a task code. Since the input image corresponding to each image enhancement task will be input to the corresponding first neural network layer, through the task code, not only can Knowing which input image of the image enhancement task the n first feature blocks received by the transformer module are from, and also knowing which first neural network layer performs the feature extraction operation for the n first feature blocks.
  • the transformer module specifically generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which is achievable.
  • the process of segmenting the first feature map by the segmenting module may specifically be: first segment the first feature map to obtain n segments, and then segment the n segments.
  • Each of the divided blocks is extended into a feature block represented by a one-dimensional vector (ie, the first feature block), so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map
  • the obtained n segmentation blocks may have the same size or different sizes, which are not specifically limited here.
  • the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments
  • the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the sizes of the n segmentation blocks obtained by segmentation by the segmentation module may be the same or different, and may be preset according to requirements and have selectivity.
  • a second aspect of the embodiments of the present application further provides a model structure, and the model may specifically include: a first neural network layer 1, a segmentation module, a transformer module, a reorganization module, and a second neural network layer, wherein the first neural network layer It can also be called head module or head structure, and the second neural network layer can also be called tail module or tail structure.
  • the first neural network layer is used to perform feature extraction on the input image to obtain a feature map (which can be called a first feature map), and then the first feature map is input to the segmentation module, and the segmentation module is used for the segmentation module.
  • the first feature map is segmented to obtain n feature blocks (which may be referred to as first feature blocks), where n ⁇ 2.
  • the n first feature blocks are further input into the transformer module for processing.
  • the transformer module generates relevant information based on the n first feature blocks, and the relevant information is used to indicate the degree of correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates the relevant information according to the relevant information.
  • each first feature block also fuses feature information of other first feature blocks according to the degree of correlation between itself and other first feature blocks.
  • the recombination module is used to splicing and recombining the n second feature blocks according to their relative spatial positions, so as to obtain the first feature with the input.
  • the second feature map with consistent graph dimensions.
  • the recombination module splices and recombines the n second feature blocks to obtain a second feature map, and the second feature map is input into the second neural network layer, and the second neural network layer decodes the received second feature map , to obtain an output image, which is an enhanced image of the input image after model processing.
  • a new model structure is obtained by combining the transformer module for processing natural language tasks with different neural network structures, which breaks through the limitation that the transformer module can only be used for processing natural language tasks. It can be applied to low-level vision tasks.
  • the model structure has a first neural network layer and a second neural network layer for processing a specific image enhancement task.
  • CNN as an excellent feature extractor, can perform well in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks), the model can pay attention to global information with the help of the transformer module, which can improve the image quality. Enhancement.
  • the transformer module includes an encoder and a decoder.
  • the transformer module generates n second feature blocks one-to-one corresponding to the n first feature blocks based on the relevant information, which may be: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one, where the first related information is used to indicate the nth
  • the first correlation between any two first feature blocks in a feature block, and the dimensions of the n first feature blocks input by the encoder are consistent with the dimensions of the n third feature blocks; second related information, and according to the second related information, generate n second feature blocks corresponding to the n third feature blocks one-to-one, the second related information is used to indicate any one of the n third feature blocks
  • the second correlation degree between the two third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the second related information incorporates the first task code
  • the first task code acts as an input to the decoder
  • the first task code is the corresponding identifier of the image enhancement task to which the input image belongs. Encoding, it can be known that the n first feature blocks received by the transformer module are from the input image of the image enhancement task.
  • the transformer module specifically generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which is achievable.
  • the process of segmenting the first feature map by the segmenting module may specifically be: first segment the first feature map to obtain n segments, and then segment the n segments.
  • Each of the divided blocks is extended into a feature block represented by a one-dimensional vector (ie, the first feature block), so that n first feature blocks can be obtained.
  • the segmentation module segments the first feature map, and the obtained n segmentation blocks may all have the same size or may have different sizes, which are not specifically limited here.
  • the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments
  • the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the sizes of the n segmentation blocks obtained by segmentation by the segmentation module may be the same or different, and may be preset according to requirements and have selectivity.
  • a third aspect of the embodiments of the present application provides a method for training a model.
  • the method includes: a training device first obtains a training sample from a constructed training set, where the training sample may be any degraded image in the constructed training set, and each The degraded image is obtained through image degradation processing through a clear image.
  • the training device After the training device obtains the training sample, it will input the training sample into the model, and then the selection module in the model will determine the first target neural network layer corresponding to the training sample.
  • the first target neural network layer will perform feature extraction on the training sample to obtain a feature map (which may be referred to as a first feature map).
  • the obtained first feature map will be further input to the segmentation module of the model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), where n ⁇ 2.
  • the segmentation module in the model obtains n first feature blocks
  • the n first feature blocks are further input into the transformer module in the model for processing.
  • the transformer module generates relevant information based on the n first feature blocks.
  • the correlation information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates n second feature blocks corresponding to the n first feature blocks one-to-one according to the related information .
  • each first feature block in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks.
  • the transformer module in the model After the transformer module in the model obtains n second feature blocks from n first feature blocks based on the relevant information, it will splicing and reorganizing the n second feature blocks according to the relative spatial positions through the reorganization module in the model, and obtain the same as the input.
  • the first feature map has the same dimension as the second feature map.
  • the reorganization module in the model splices and reorganizes the n second feature blocks to obtain a second feature map, and the second feature map is input into the second target neural network layer uniquely corresponding to the first target neural network layer.
  • the target neural network layer belongs to one of m second neural network layers in the model.
  • the second target neural network layer then decodes the received second feature map, thereby obtaining an enhanced image of the training sample (which may be referred to as a first enhanced image).
  • the training device obtains the first enhanced image output via the model, it will train the model according to the first enhanced image, the clear image and the loss function to obtain a trained model.
  • the training sample is obtained from the clear image through image degradation processing, so it can be said that the clear image corresponds to the training sample.
  • the model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying visual tasks.
  • the model structure has Multiple first neural network layers and multiple second neural network layers, different first/second neural network layers correspond to different image enhancement tasks, so the model can be used to process different image enhancement tasks after training, and compared to Since most of the existing models dealing with low-level visual tasks are based on CNN (CNN, as an excellent feature extractor, can flex its muscles in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks).
  • CNN as an excellent feature extractor, can flex its muscles in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks).
  • the transformer module can pay attention to global information, which can improve the image enhancement effect.
  • the labels are used to indicate which first neural network layer the training samples should be processed by. Extract features. Then, the selection module of the model can determine that the training sample belongs to the first image enhancement task according to the label of the training sample, and further determine the first target neural network layer corresponding to the first image enhancement task.
  • the transformer module includes an encoder and a decoder.
  • the transformer module generates n second feature blocks one-to-one corresponding to the n first feature blocks based on the relevant information, which may be: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one, where the first related information is used to indicate the nth
  • the first correlation between any two first feature blocks in a feature block, and the dimensions of the n first feature blocks input by the encoder are consistent with the dimensions of the n third feature blocks; second related information, and according to the second related information, generate n second feature blocks corresponding to the n third feature blocks one-to-one, the second related information is used to indicate any one of the n third feature blocks
  • the second correlation degree between the two third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the first task code is integrated into the second related information, the first task code acts as an input to the decoder, and the first task code is the corresponding identifier of the first image enhancement task, which can also be considered as the first task code.
  • Each image enhancement task corresponds to a task code. Since the input image corresponding to each image enhancement task will be input to the corresponding first neural network layer, through the task code, not only can Knowing which input image of the image enhancement task the n first feature blocks received by the transformer module are from, and also knowing which first neural network layer performs the feature extraction operation for the n first feature blocks.
  • the transformer module specifically generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which is achievable.
  • the process of segmenting the first feature map by the segmenting module may specifically be: first segment the first feature map to obtain n segments, and then segment the n segments.
  • Each of the divided blocks is extended into a feature block represented by a one-dimensional vector (ie, the first feature block), so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may all have the same size or may have different sizes, which are not specifically limited here.
  • the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments
  • the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the sizes of the n segmentation blocks obtained by segmentation by the segmentation module may be the same or different, and may be preset according to requirements and have selectivity.
  • the trained model can be deployed on target devices, such as edge devices or end-side devices, such as mobile phones, tablets, laptops, supervision systems (such as cameras, etc.) )and many more.
  • target devices such as edge devices or end-side devices, such as mobile phones, tablets, laptops, supervision systems (such as cameras, etc.) )and many more.
  • a fourth aspect of the embodiments of the present application further provides a method for training a model.
  • the method may include: the training device obtains a training sample, where the training sample is any degraded image in the constructed training set, wherein each degraded image in the training set is composed of A clear image is obtained after image degradation processing.
  • the training device After the training device obtains the training sample, it will input the training sample into the model, and the first neural network layer in the model will perform feature extraction on the training sample to obtain a first feature map.
  • the obtained first feature map will be further input to the segmentation module of the model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), where n ⁇ 2.
  • the n first feature blocks are further input into the transformer module in the model for processing.
  • the transformer module generates relevant information based on the n first feature blocks.
  • the correlation information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates n second feature blocks corresponding to the n first feature blocks one-to-one according to the related information . That is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks.
  • the transformer module in the model After the transformer module in the model obtains n second feature blocks from n first feature blocks based on the relevant information, it will splicing and reorganizing the n second feature blocks according to the relative spatial positions through the reorganization module in the model, and obtain the same as the input.
  • the first feature map has the same dimension as the second feature map.
  • the reorganization module in the model splices and reorganizes the n second feature blocks to obtain a second feature map, which will be input into the second neural network layer, and then the second neural network layer will receive the second feature.
  • the image is decoded to obtain an enhanced image of the training sample (which may be referred to as a first enhanced image).
  • the trained model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying vision tasks.
  • the model structure has a first neural network layer and a second neural network layer, which are used to deal with a specific image enhancement task.
  • CNN is an excellent feature.
  • the extractor can flex its muscles in high-level vision tasks, but it is difficult to pay attention to global information when dealing with low-level vision tasks).
  • the transformer module includes an encoder and a decoder.
  • the transformer module generates n second feature blocks one-to-one corresponding to the n first feature blocks based on the relevant information, which may be: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one, where the first related information is used to indicate the nth
  • the first correlation between any two first feature blocks in a feature block, and the dimensions of the n first feature blocks input by the encoder are consistent with the dimensions of the n third feature blocks; second related information, and according to the second related information, generate n second feature blocks corresponding to the n third feature blocks one-to-one, the second related information is used to indicate any one of the n third feature blocks
  • the second correlation degree between the two third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the second related information incorporates the first task code
  • the first task code acts as an input to the decoder
  • the first task code is the corresponding identifier of the image enhancement task to which the input image belongs. Encoding, it can be known that the n first feature blocks received by the transformer module are from the input image of the image enhancement task.
  • the transformer module specifically generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which is achievable.
  • the process of segmenting the first feature map by the segmenting module may specifically be: first segment the first feature map to obtain n segments, and then segment the n segments.
  • Each of the divided blocks is extended into a feature block represented by a one-dimensional vector (ie, the first feature block), so that n first feature blocks can be obtained.
  • the segmenting module segments the first feature map, and the obtained n segmented blocks may all have the same size or may have different sizes, which are not specifically limited here.
  • the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments
  • the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the sizes of the n segmentation blocks obtained by segmentation by the segmentation module may be the same or different, and may be preset according to requirements and have selectivity.
  • the trained model can be deployed on a target device, such as an edge device or an end-side device, such as a mobile phone, tablet, laptop, supervision system (such as a camera, etc.) )and many more.
  • a target device such as an edge device or an end-side device, such as a mobile phone, tablet, laptop, supervision system (such as a camera, etc.) )and many more.
  • a fifth aspect of the embodiments of the present application provides an image enhancement method.
  • the method includes: an execution device (that is, the above-mentioned target device) acquires a target image to be processed, for example, an image captured by a mobile phone through a camera, and a monitoring Images captured by the device through the camera, etc.
  • a trained model is deployed on the execution device. After the execution device obtains the target image, it will input the target image into the trained model, and the selection module in the trained model will determine the first target corresponding to the target image.
  • a neural network layer where the first target neural network layer is one of m first neural network layers in the trained model. The first target neural network layer will perform feature extraction on the target image to obtain a feature map (which may be referred to as a first feature map).
  • the obtained first feature map will be further input into the segmentation module of the trained model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), n ⁇ 2.
  • the segmentation module in the trained model obtains n first feature blocks
  • the n first feature blocks are further input into the transformer module in the trained model for processing, and the transformer module is based on the n first features block, and generate related information, the related information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates a one-to-one correspondence with the n first feature blocks according to the related information
  • the n second feature blocks of are examples of the transformer module in the trained model for processing, and the transformer module is based on the n first features block, and generate related information, the related information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates a one-to-one correspondence with the n first feature blocks according to
  • each first feature block in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks.
  • the transformer module in the trained model obtains n second feature blocks from the n first feature blocks based on the relevant information, the reorganization module in the trained model will perform the processing on the n second feature blocks according to their relative spatial positions. After splicing and recombination, a second feature map with the same dimension as the input first feature map is obtained.
  • the reorganization module in the trained model splices and reorganizes the n second feature blocks to obtain a second feature map, and the second feature map is input into the second target neural network layer uniquely corresponding to the first target neural network layer,
  • the second target neural network layer belongs to one of m second neural network layers in the trained model. Then the second target neural network layer decodes the received second feature map, thereby obtaining an enhanced image of the training target image (which may be referred to as a second enhanced image).
  • the trained model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying vision tasks.
  • the model structure has multiple first neural network layers and multiple second neural network layers, and different first/second neural network layers correspond to different image enhancement tasks, so the model can be used to process different image enhancement tasks after training, And compared with the existing models that deal with low-level visual tasks, most of them are based on CNN (CNN, as an excellent feature extractor, can flex its muscles in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks). With the help of the transformer module, the model can pay attention to the global information, which can improve the image enhancement effect.
  • CNN CNN
  • the transformer module the model can pay attention to the global information, which can improve the image enhancement effect.
  • the execution device will additionally Send an instruction to the trained model, the instruction is used to indicate which type of image enhancement task the target image belongs to, that is, in the inference phase, the selection module of the trained model is based on the received instruction. It is determined that the target image belongs to the first image enhancement task, and the first target neural network layer corresponding to the first image enhancement task is further determined.
  • the transformer module includes an encoder and a decoder.
  • the transformer module generates n second feature blocks one-to-one corresponding to the n first feature blocks based on the relevant information, which may be: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one, where the first related information is used to indicate the nth
  • the first correlation between any two first feature blocks in a feature block, and the dimensions of the n first feature blocks input by the encoder are consistent with the dimensions of the n third feature blocks; second related information, and according to the second related information, generate n second feature blocks corresponding to the n third feature blocks one-to-one, the second related information is used to indicate any one of the n third feature blocks
  • the second correlation degree between the two third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the first task code is integrated into the second related information, the first task code acts as an input to the decoder, and the first task code is the corresponding identifier of the first image enhancement task, which can also be considered as the first task code.
  • Each image enhancement task corresponds to a task code. Since the input image corresponding to each image enhancement task will be input to the corresponding first neural network layer, through the task code, not only can Knowing which input image of the image enhancement task the n first feature blocks received by the transformer module are from, and also knowing which first neural network layer performs the feature extraction operation for the n first feature blocks.
  • the transformer module specifically generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which is achievable.
  • the process of segmenting the first feature map by the segmenting module may specifically be: first segment the first feature map to obtain n segments, and then segment the n segments.
  • Each of the divided blocks is extended into a feature block represented by a one-dimensional vector (ie, the first feature block), so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may all have the same size or may have different sizes, which are not specifically limited here.
  • the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments
  • the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the sizes of the n segmentation blocks obtained by segmentation by the segmentation module may be the same or different, and may be preset according to requirements and have selectivity.
  • a sixth aspect of the embodiments of the present application provides an image enhancement method.
  • the method includes: an execution device (that is, the above-mentioned target device) acquires a target image to be processed, for example, an image captured by a mobile phone through a camera, and the image is captured by a monitoring device. Images captured by the device through the camera, etc.
  • a trained model is deployed on the execution device. After the execution device obtains the target image, it will input the target image into the trained model, and the first neural network layer in the trained model will perform feature extraction on the target image. Get the first feature map.
  • the obtained first feature map will be further input into the segmentation module of the trained model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), n ⁇ 2.
  • the segmentation module in the trained model obtains n first feature blocks
  • the n first feature blocks are further input into the transformer module in the trained model for processing, and the transformer module is based on the n first features block, and generate related information, the related information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates a one-to-one correspondence with the n first feature blocks according to the related information
  • the n second feature blocks of are examples of the transformer module in the trained model for processing, and the transformer module is based on the n first features block, and generate related information, the related information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates a one-to-one correspondence with the n first feature blocks according to
  • each first feature block in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks.
  • the transformer module in the trained model obtains n second feature blocks from n first feature blocks based on relevant information, the n second feature blocks are spliced according to their relative spatial positions through the reorganization module in the trained model. Recombination to obtain a second feature map with the same dimension as the input first feature map.
  • the reorganization module in the trained model splices and reorganizes n second feature blocks to obtain a second feature map, which will be input into the second neural network layer, and then the second neural network layer will receive the received
  • the second feature map is decoded to obtain an enhanced image of the training sample (which may be referred to as a second enhanced image).
  • the trained model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying vision tasks.
  • the model structure has a first neural network layer and a second neural network layer, which are used to deal with a specific image enhancement task.
  • CNN is an excellent feature.
  • the extractor can flex its muscles in high-level vision tasks, but it is difficult to pay attention to global information when dealing with low-level vision tasks).
  • the transformer module includes an encoder and a decoder.
  • the transformer module generates n second feature blocks one-to-one corresponding to the n first feature blocks based on the relevant information, which may be: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one, where the first related information is used to indicate the nth
  • the first correlation between any two first feature blocks in a feature block, and the dimensions of the n first feature blocks input by the encoder are consistent with the dimensions of the n third feature blocks; second related information, and according to the second related information, generate n second feature blocks corresponding to the n third feature blocks one-to-one, the second related information is used to indicate any one of the n third feature blocks
  • the second correlation degree between the two third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the second related information incorporates the first task code
  • the first task code acts as an input to the decoder
  • the first task code is the corresponding identifier of the image enhancement task to which the input image belongs. Encoding, it can be known that the n first feature blocks received by the transformer module are from the input image of the image enhancement task.
  • the transformer module specifically generates n second feature blocks corresponding to the n first feature blocks one-to-one based on the relevant information, which is achievable.
  • the process of segmenting the first feature map by the segmenting module may specifically be: firstly segment the first feature map to obtain n segments, and then segment the n segments.
  • Each of the divided blocks is extended into a feature block represented by a one-dimensional vector (ie, the first feature block), so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may all have the same size or may have different sizes, which are not specifically limited here.
  • the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments
  • the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the sizes of the n segmentation blocks obtained by segmentation by the segmentation module may be the same or different, and may be preset according to requirements and have selectivity.
  • a seventh aspect of the embodiments of the present application provides a training device, where the training device has the function of implementing the method of the third/fourth aspect or any one of the possible implementation manners of the third/fourth aspect.
  • This function can be implemented by hardware or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • An eighth aspect of the embodiments of the present application provides an execution device, and the training device has a function of implementing the method of the fifth/sixth aspect or any one of the possible implementation manners of the fifth/sixth aspect.
  • This function can be implemented by hardware or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • a ninth aspect of an embodiment of the present application provides a training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to call a program stored in the memory to execute the third/3rd embodiment of the present application.
  • a training device which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to call a program stored in the memory to execute the third/3rd embodiment of the present application.
  • a tenth aspect of an embodiment of the present application provides an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to call the program stored in the memory to execute the fifth/ A method for any possible implementation of the sixth aspect or the fifth/sixth aspect.
  • An eleventh aspect of the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer can execute the third/fourth aspect or the third/fourth aspect Any one of the possible implementations of the method, or enabling a computer to execute the fifth/sixth aspect or the method of any one of the fifth/sixth possible implementations.
  • a twelfth aspect of the embodiments of the present application provides a computer program, which, when run on a computer, enables the computer to execute the method of the third/fourth aspect or any one of the possible implementation manners of the third/fourth aspect, or, The computer can execute the method of the fifth/sixth aspect or any one possible implementation manner of the fifth/sixth aspect.
  • a thirteenth aspect of an embodiment of the present application provides a chip, the chip includes at least one processor and at least one interface circuit, the interface circuit is coupled to the processor, and the at least one interface circuit is configured to perform a transceiving function and send an instruction
  • At least one processor is given to at least one processor for running a computer program or instruction, which has the function of implementing the method as described in the third/fourth aspect or any one of the possible implementations of the third/fourth aspect, or, it has the function of implementing the method as The function of the method of the fifth/sixth aspect or any one of the possible implementations of the fifth/sixth aspect, the function can be realized by hardware, can also be realized by software, and can also be realized by a combination of hardware and software, and the hardware or software includes One or more modules corresponding to the above functions.
  • Fig. 1 is a schematic diagram of processing the underlying visual task based on CNN
  • Fig. 2 is a schematic diagram of the standard structure of the transformer module
  • FIG. 3 is a schematic structural diagram of an artificial intelligence main body framework provided by an embodiment of the present application.
  • Fig. 4 is a schematic diagram of the structure of the model provided by the embodiment of the present application.
  • FIG. 5 is a schematic diagram of performing image enhancement processing on an input image by a model provided by an embodiment of the present application
  • FIG. 6 is a schematic diagram of a transformer encoder provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a transformer decoder provided by an embodiment of the present application.
  • FIG. 8 is another schematic diagram of the structure of a model provided by an embodiment of the present application.
  • FIG. 9 is a system architecture diagram of an image enhancement system provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of a training method for a model provided by an embodiment of the present application.
  • FIG. 11 is another schematic flowchart of a training method for a model provided by an embodiment of the present application.
  • FIG. 12 is a schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • FIG. 13 is another schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • FIG. 14 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 15 is a schematic diagram of a training device provided by an embodiment of the application.
  • 16 is a schematic diagram of an execution device provided by an embodiment of the present application.
  • FIG. 17 is another schematic diagram of the training device provided by the embodiment of the application.
  • FIG. 18 is another schematic diagram of an execution device provided by an embodiment of the present application.
  • FIG. 19 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • the embodiments of the present application provide a model structure, a model training method, an image enhancement method and device, and a new model structure is obtained by combining a transformer module for processing natural language tasks with different neural network structures, which breaks through the limitation of the transformer module only.
  • the model structure can be applied to the underlying vision tasks, the model structure has multiple first neural network layers and multiple second neural network layers, different first/second neural network layers Corresponding to different image enhancement tasks, so the model can be used to deal with different image enhancement tasks after training, and compared with the existing models dealing with low-level visual tasks, most of them are based on CNN (CNN is an excellent feature extractor in high-level It can show its strengths in visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks). With the help of the transformer module, this model can pay attention to global information, which can improve the effect of image enhancement.
  • CNN is an excellent feature extractor in high-level It can show its strengths in visual tasks, but it is difficult to pay attention to global information when dealing with low-level
  • the embodiments of the present application involve a lot of related knowledge about neural networks, models, etc.
  • related terms and concepts that may be involved in the embodiments of the present application are first introduced below. It should be understood that the related concept interpretation may be limited due to the specific circumstances of the embodiments of the present application, but it does not mean that the present application can only be limited to the specific circumstances, and there may be differences in the specific circumstances of different embodiments. There is no specific limitation here.
  • a neural network is a model.
  • a neural network can be composed of neural units. Specifically, it can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, and the last layer is the output layer. The layers in the middle are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN).
  • DNN deep neural network
  • the work of each layer in a neural network can be expressed mathematically To describe, from the physical level, the work of each layer in the neural network can be understood as completing the transformation from the input space to the output space (that is, the row space of the matrix to the column through five operations on the input space (set of input vectors) Space), these five operations include: 1. Dimension raising/lowering; 2.
  • the models used to process the image enhancement task are essentially all neural networks or a part of the model structure is a neural network.
  • the application of the model generally includes two stages of training and inference.
  • the training stage is used to train the model according to the training set to obtain the trained model;
  • the quality of the enhanced image obtained after image enhancement processing is one of the important indicators to measure the quality of a model training.
  • CNN is a neural network with a convolutional structure.
  • CNN contains a feature extractor consisting of convolutional and subsampling layers.
  • the feature extractor can be viewed as a filter, and the convolution process can be viewed as convolution with an input image or a convolutional feature map using a trainable filter.
  • the convolutional layer refers to the neuron layer in the CNN that convolves the input signal.
  • a neuron can only be connected to some of its neighbors.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangle. Neural units in the same feature plane share weights, and the shared weights here are convolution kernels.
  • Shared weights can be understood as the way to extract image information is independent of location.
  • the underlying principle is that the statistics of one part of the image are the same as the other parts. This means that image information learned in one part can also be used in another part. So for all positions on the image, the same learned image information can be used.
  • multiple convolution kernels can be used to extract different image information. Generally, the more convolution kernels, the richer the image information reflected by the convolution operation.
  • the convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights by learning during the training process of CNN.
  • the immediate benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.
  • the error back propagation (BP) algorithm can be used to correct the size of the parameters in the initial neural network model, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal until the output will generate an error loss, and updating the parameters in the initial neural network model by back-propagating the error loss information, so that the error loss converges.
  • the back-propagation algorithm is a back-propagation movement dominated by error loss, aiming to obtain the parameters of the optimal neural network model, such as the weight matrix.
  • the self-attention module is a structure of a neural network characterized by calculating each unit in the input module (the self-attention module was originally used in natural language processing, where each unit refers to each word) The degree of correlation between them, and capture information between input units according to the degree of correlation.
  • the self-attention module first converts it into 3 vectors Then multiply these three vectors by three weight matrices to obtain three new vectors q, k, and v. These three different weight matrices can be recorded as Q, K, and V.
  • the multi-head self-attention module is generally used, that is, for the input unit, it is firstly divided into h blocks, which are respectively input into the h above-mentioned self-attention modules to obtain h outputs z, and then z is divided according to the segmentation method. Put it back together, and after a layer of fully connected network, the final output is obtained.
  • MSA MSA
  • Transformer module can also be called transformer model, transformer structure, etc. It is a multi-layer neural network based on self-attention module. At present, it is mainly used to process natural language tasks.
  • the transformer module is mainly composed of a stacked multi-head self-attention module (also known as MSA module) and feed forward neural networks (FFN).
  • the transformer module can be further divided into an encoder and a decoder (also referred to as an encoding module and a decoding module), which are roughly similar in composition but also different.
  • each encoder can include any number of encoding sub-modules, and each encoding sub-module includes a multi-head self-attention module and a feed-forward neural network; similarly, each decoder may include any number of decoding sub-modules, each decoding sub-module including two multi-head self-attention modules and a feed-forward neural network.
  • the number of encoding sub-modules and the number of decoding sub-modules may be different.
  • the transformer module is used to process natural language tasks and cannot be directly applied to computer vision tasks. That is, the input to both the encoder and decoder of the transformer module is the encoding of the word.
  • Figure 3 shows a schematic structural diagram of the main frame of artificial intelligence.
  • the "intelligent information chain” reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensed process of "data-information-knowledge-wisdom".
  • the "IT value chain” from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system reflects the value brought by artificial intelligence to the information technology industry.
  • the infrastructure provides computing power support for artificial intelligence systems, realizes communication with the outside world, and supports through the basic platform. Communication with the outside world through sensors; computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA); the basic platform includes distributed computing framework and network-related platform guarantee and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.
  • smart chips hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA
  • the basic platform includes distributed computing framework and network-related platform guarantee and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.
  • the data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence.
  • the data involves graphics, images, voice, and text, as well as IoT data from traditional devices, including business data from existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
  • machine learning and deep learning can perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on data.
  • Reasoning refers to the process of simulating human's intelligent reasoning method in a computer or intelligent system, using formalized information to carry out machine thinking and solving problems according to the reasoning control strategy, and the typical function is search and matching.
  • Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
  • some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image identification, etc.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall solution of artificial intelligence, the productization of intelligent information decision-making, and the realization of landing applications. Its application areas mainly include: intelligent terminals, intelligent manufacturing, Smart transportation, smart home, smart medical, smart camera, autonomous driving, smart city, etc.
  • the embodiments of the present application can be applied to the optimization design of the network structure of the model, and the model whose structure has been optimized by the present application can be specifically applied to various sub-fields in the field of artificial intelligence, for example, the field of image processing in the field of computer vision, the field of semantics areas of analysis, etc.
  • the first neural network layer and the second neural network layer included in the structure of the model are both m, and m ⁇ 2.
  • the model 400 may specifically include: a selection module 401, m first neural network layers 402, a segmentation module 403, a transformer module 404, The reorganization module 405 and m second neural network layers 406, each first neural network layer uniquely corresponds to a second neural network layer, and each first neural network layer may also be called a head module or a head structure, each second neural network layer. Neural network layers can also be called tail modules or tail structures.
  • first neural network layers correspond to different image enhancement tasks, that is, each image enhancement task has a corresponding first neural network layer, a first neural network layer
  • the network layer processes the corresponding input image for a specific type of image enhancement task, for example, the 2x super-resolution reconstruction task, the 3x super-resolution reconstruction task, the denoising task, etc., each of which has a corresponding first neural network.
  • Network layer processes the corresponding input image for a specific type of image enhancement task, for example, the 2x super-resolution reconstruction task, the 3x super-resolution reconstruction task, the denoising task, etc.
  • each first neural network layer can be set by itself, as long as it can be run.
  • the number m of the first neural network layer can also be set according to user requirements, which depends on what types of image enhancement tasks the model 400 is used to process.
  • the selection module 401 in the model 400 is used to obtain an input image and determine a first target neural network layer 4021 corresponding to the input image, where the first target neural network layer 4021 is the m first neural network layers 402 in the model 400 one of the. That is to say, after the selection module 401 receives the input image, it will determine which first neural network layer should perform the feature extraction operation for the input image. Specifically, the selection module 401 will first determine which type of the input image belongs to. Image enhancement task, and then input the input image to the first neural network layer corresponding to the task. The image enhancement task to which the input image belongs can be called the first image enhancement task. Assuming that the first image enhancement task corresponds to the first target neural network layer 4021, the selection module 401 can determine that the received input image is input to the into the first target neural network layer 4021.
  • the selection module 401 determines which type of image enhancement task the input image belongs to for the input images in different phases.
  • the methods are also slightly different, which are explained below:
  • the input image is the training sample in the training set.
  • the input image is the training sample in the training set, and each training sample will have a corresponding label indicating which category the training sample belongs to.
  • this label is used to indicate which first neural network layer the training sample should use to extract features.
  • the selection module 401 of the model 400 can determine that the training sample belongs to the first image enhancement task according to the label of the training sample.
  • the input image is the target image to be processed.
  • the input image is the real target image to be processed.
  • the selection module 401 not only receives the input image, but also The instruction issued by the device deploying the model 400 will be received, and the instruction is used to indicate which type of image enhancement task the target image belongs to. That is, in the inference stage, the selection module 401 of the model 400 is based on the received of instructions to determine that the target image belongs to the first image enhancement task.
  • the selection module 401 of the model 400 determines the first target neural network layer 4021 corresponding to the input image according to the input image
  • the input image is input to the first target neural network layer 4021, and the first target neural network layer 4021 will perform feature extraction on the input image to obtain a feature map (which may be referred to as a first feature map).
  • the obtained first feature map will be further input to the segmentation module 403, and the segmentation module 403 will segment the first feature map to obtain n feature blocks (may be referred to as first feature blocks), where n ⁇ 2.
  • the process of segmenting the first feature map by the segmenting module 403 may specifically be as follows: firstly segment the first feature map to obtain n segmented blocks, and then Each of the n segmentation blocks is extended into a feature block (ie, the first feature block) represented by a one-dimensional vector, so that n first feature blocks can be obtained.
  • the segmentation module 403 performs segmentation on the first feature map, and the obtained n segmentation blocks may have the same size or different sizes. Specifically, here Not limited. In the case where the sizes of the obtained n segments are all the same, the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments When the size of the block is different, the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the segmentation module 403 After the segmentation module 403 obtains the n first feature blocks, the n first feature blocks are further input into the transformer module 404 for processing.
  • the transformer module 404 generates correlation information based on the n first feature blocks, where the correlation information is used to indicate the degree of correlation between any two first feature blocks in the n first feature blocks, and then the transformer module 404 determines the correlation according to the correlation
  • the information generates n second feature blocks corresponding to the n first feature blocks one-to-one. That is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks. It should be noted here that the dimensions of the n first feature blocks input by the transformer module 404 are consistent with the dimensions of the n second feature blocks output.
  • the transformer module 404 including at least one encoder and at least one decoder as an example, how the transformer module 404 generates the n first feature blocks one by one based on the relevant information
  • the corresponding n second feature blocks are described: first, the first correlation information is generated by the encoder, and according to the first correlation information, n third feature blocks corresponding to the n first feature blocks one-to-one are generated, The first correlation information is used to indicate the first correlation between any two of the n first feature blocks, and the dimensions of the n first feature blocks input by the encoder are related to the n third feature blocks The dimensions of the n third feature blocks are kept consistent; after that, the second related information is generated by the decoder, and according to the second related information, n second feature blocks corresponding to the n third feature blocks are generated. is used to indicate the second correlation between any two third feature blocks among the n third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the
  • the first task code is integrated into the second related information, the first task code acts as an input to the decoder, and the first task code is the corresponding identifier of the first image enhancement task, which can also be considered as the first task code.
  • Each image enhancement task corresponds to a task code. Since the input image corresponding to each image enhancement task will be input to the corresponding first neural network layer, through the task code, not only can Knowing which input images of the image enhancement task the n first feature blocks received by the transformer module 404 are from, and also knowing which first neural network layer performs the feature extraction operation for the n first feature blocks.
  • the first task code may be sent by the encoder to the decoder, and then the first task code acts on the decoder as an input, and the first task code It can also be that when the first target neural network layer is triggered to receive the input image, the first task code is received through an instruction sent by the device deploying the model 400, and then the first task code acts as an input to the decoder. , specifically, the present application does not limit the acquisition method of the first task code.
  • each task code may be self-labeled according to the image enhancement task, or may be learned by the model itself, which is not specifically limited here.
  • the recombination module 405 After the transformer module 404 obtains n second feature blocks from the n first feature blocks based on the relevant information, the recombination module 405 will splicing and reorganize the n second feature blocks according to the relative spatial positions, so as to obtain the input first feature block.
  • the operation of the recombination module 405 is the inverse operation of the segmentation module 403, which will not be repeated here. It should be noted here that the size of the second feature map should be consistent with the size of the first feature map.
  • the recombination module 405 splices and recombines the n second feature blocks to obtain a second feature map, and inputs the second feature map into the second target neural network layer 4061 uniquely corresponding to the first target neural network layer 4021 .
  • the target neural network layer 4061 belongs to one of the m second neural network layers 406 . Then the second target neural network layer 4061 decodes the received second feature map to obtain an output image, which is an enhanced image of the input image after being processed by the model 400 .
  • each second neural network layer can also be set by themselves, as long as they can run.
  • the number m of the second neural network layer needs to be consistent with the number of the first neural network layer.
  • the structure of the transformer module 404 can be not only a standard structure including an encoder and a decoder as shown in FIG. 2 , but also its structure can be fine-tuned to The structure of the adjusted transformer module 404 is obtained.
  • the structure of the adjusted transformer module 404 may only include an encoder, or may only include a decoder.
  • the transformer module 404 should include at least two encoders, and at least one encoder is used to undertake the operation originally undertaken by the decoder; if the structure of the transformer module 404 only includes the decoder, then The transformer module 404 should include at least two decoders, at least one of which is used to undertake operations originally undertaken by the encoder.
  • m first neural network layers 402 and m second neural network layers 406 are located at the head and tail of the model, respectively, in some implementations of this application.
  • the m first neural network layers 402 may also be referred to as a multi-head structure, and the m second neural network layers 402 may also be referred to as a multi-tailed structure.
  • Each first neural network layer may be referred to as "XX" according to its corresponding image enhancement task.
  • the model includes 4 first neural network layers, and the image enhancement tasks corresponding to these 4 first neural network layers are: denoising, deraining, and 2x super-resolution reconstruction.
  • the 4 first neural network layers can be referred to as "denoising head”, “removing rain head”, “2 times super division head” and “4 times super division head” respectively.
  • the four first neural network layers also uniquely correspond to one second neural network layer, and there are four second neural network layers in total. ", "2 times super split tail” and “4 times super split tail”, similarly, if there are other image enhancement tasks, the abbreviation of the corresponding first neural network layer can be obtained according to the above method, which will not be repeated here. .
  • FIG. 5 is an embodiment of the present application.
  • this application uses a multi-head structure to process each task separately, and each task has a corresponding head module.
  • each feature block can be regarded as the encoding of a "word”.
  • the feature Divide and reshape into a series of feature blocks in N represents the number of blocks (that is, the length of the input sequence).
  • the maximum value of N is determined by the specific structure of the transformer model, and the number of feature blocks f pi cut by the segmentation module cannot exceed
  • the maximum value of N in addition, the size of the feature block f pi can also be determined by the preset size of P. In the embodiments of the present application, the size of each feature block f pi is consistent.
  • the size of the feature block f pi may also be inconsistent, which is not specifically limited here.
  • this application adds a learnable position code to each feature block f pi (In some embodiments, the position code can also be set by yourself), add each feature block f pi and the position code of the corresponding position to obtain Epi +f pi , and then input each Epi +f pi into the transformer encoder.
  • the structure of the transformer encoder in the transformer module may be as shown in FIG. 6 .
  • the sub-schematic diagram (a) in FIG. 6 illustrates an encoding sub-module in the transformer encoder, and the encoding sub-module has a Multi-head self-attention module (can be recorded as MSA module) and a feedforward neural network (can be recorded as FFN), and the transformer encoder can have multiple such encoding sub-modules (the number can be set according to your needs), as shown in the figure
  • Sub-schematic diagram (b) in 6 shows that a transformer encoder includes multiple encoding sub-modules.
  • the following describes the processing flow of the transformer encoder based on each encoding sub-module in the transformer encoder shown in FIG. 6 .
  • Equation (1) The input to the first encoding sub-module of the transformer encoder can be expressed in the form described in Equation (1):
  • y 0 represents the input of the first encoding submodule
  • y 0 is the feature block obtained after segmentation by the segmentation module
  • y 0 is the feature block obtained after segmentation by the segmentation module
  • y 0 is the feature block obtained after segmentation by the segmentation module
  • the encoding submodule processes the output obtained from each feature block f pi The same size as the input feature block fpi .
  • the calculation formula (2) of a coding submodule is as follows:
  • LN represents layer normalization (a kind of normalization operation)
  • y i-1 is the input of the current coding sub-module, for the first coding sub-module, its input is the above y 0
  • the input of the i -th encoding sub-module is the output y i-1 of the i - 1th encoding sub-module.
  • the input of the MSA module that is, the multi-head self-attention module in the encoding sub-module, and the output of the MSA module of the current encoding sub-module is shown in formula (3):
  • y' i is the output of the MSA module in the current encoding sub-module
  • y' i is used as the input part of the FFN (that is, the feedforward neural network) of the current encoding sub-module, as shown in the following formula (4):
  • yi is the output of the ith encoding sub-module
  • m in the above formula represents the number of layers in the transformer encoder (that is, there are m encoding sub-modules in total).
  • the output of the last encoding sub-module of the transformer encoder is y m (denoted as z 0 in the decoder), as shown in the following formula (5):
  • the structure of the transformer decoder in the transformer module may be as shown in FIG. 7 , the transformer decoder and the transformer encoder have a similar system, and the sub-schematic diagram (a) in FIG. 7 shows the transformer A decoding sub-module in the decoder, the decoding sub-module has 2 multi-head self-attention modules (respectively denoted as MSA1 module and MSA2 module) and a feedforward neural network (denoted as FFN), and the transformer decoder in There may be a plurality of such decoding sub-modules (the number can be set according to needs), as shown in the sub-schematic diagram (b) in FIG. 7 , a transformer decoder includes a plurality of decoding sub-modules.
  • the following describes the processing flow of the transformer decoder based on each decoding sub-module in the transformer decoder shown in FIG. 7 .
  • the difference from the transformer module for processing natural language tasks is that the present application uses the task encoding of a specific image enhancement task as one of the inputs of the transformer decoder.
  • These task codes The features of different image enhancement tasks can be encoded. It should be noted that the task encoding can be preset or learned, which is not limited here.
  • the input of the first decoding sub-module of the transformer decoder is the output y m of the last encoding sub-module of the transformer encoder, which can be expressed in the form described in formula (6):
  • E t is the task code, which is used to calculate the qi and ki vectors, while v i has nothing to do with this, and zi -1 is the input of the current decoding sub-module.
  • the input of the i-th decoding sub-module is the output zi -1 of the i-1-th decoding sub-module, and these three vectors q i , k i , and vi are then converted into It is sent to the MSA1 module of the decoding sub-module, and the output z′ i of the MSA1 module is obtained according to the following formula (8):
  • the calculation of the q' i vector is calculated according to the output z' i of the MSA1 module, and the calculation of the k' i and v' i vectors is based on the output z 0 of the transformer encoder, thereby obtaining the input q' i of the MSA2 module ,k′ i ,v′ i , so the output z′′ i of the MSA2 module can be calculated by the following formula (10):
  • the decoded N feature blocks of size P 2 ⁇ C are reshaped into features f D of size C ⁇ H ⁇ W (ie, the second feature map) through the recombination module.
  • the recombination module will reshape the feature f D is input to the tail structure corresponding to the head structure of the processing input image. For example, assuming that the feature extraction of the input image is a "denoising head”, then the feature f D will be input to the "denoising tail", and the The tail structure decodes the feature f D to obtain an output image, which is an enhanced image of the input image after model processing.
  • the calculation formula (13) of the tail structure is as follows:
  • N t represents the number of types of image enhancement tasks.
  • the output f T is the resulting image of size 3 ⁇ H′ ⁇ W′.
  • a new model structure is obtained by combining the transformer module for processing natural language tasks with different neural network structures, which breaks through the limitation that the transformer module can only be used for processing natural language tasks. It can be applied to the underlying vision tasks.
  • the model structure has multiple first neural network layers and multiple second neural network layers. Different first/second neural network layers correspond to different image enhancement tasks. It can be used to deal with different image enhancement tasks, and compared with the existing models for dealing with low-level visual tasks, most of them are based on CNN (CNN, as an excellent feature extractor, can show its strength in high-level visual tasks, but it is not suitable for processing low-level visual tasks. It is difficult to pay attention to the global information during the task), the model can pay attention to the global information with the help of the transformer module, which can improve the image enhancement effect.
  • CNN as an excellent feature extractor
  • Both the first neural network layer and the second neural network layer included in the structure of the model are one.
  • the model 800 may specifically include: a first neural network layer 801 , a segmentation module 802 , a transformer module 803 , and a reorganization module 804 and the second neural network layer 805, wherein the first neural network layer 801 may also be called a head module or a head structure, and the second neural network layer 805 may also be called a tail module or a tail structure.
  • the first neural network layer 801 and the second neural network layer 805 since each of the first neural network layer 801 and the second neural network layer 805 has only one, there is no selection module in the model 800 .
  • the first neural network layer 801 corresponds to only one type of image enhancement task, and the first neural network layer 801 processes a corresponding input image for a certain type of image enhancement task .
  • the size, depth, parameter quantity, etc. of the first neural network layer 801 and the second neural network layer 805 can be set by themselves, as long as they can be run.
  • the first neural network layer 801 is used to perform feature extraction on the input image to obtain a feature map (which may be referred to as a first feature map), and then the first feature map is input to the segmentation module 802, where the The segmentation module 802 segments the first feature map to obtain n feature blocks (which may be referred to as first feature blocks), where n ⁇ 2.
  • the process of segmenting the first feature map by the segmenting module 802 may specifically be: first segment the first feature map to obtain n segments, and then segment the n segments Each segmented block in the block is extended into a feature block (ie, the first feature block) represented by a one-dimensional vector, so that n first feature blocks can be obtained.
  • the segmentation module 802 performs segmentation on the first feature map, and the obtained n segmentation blocks may have the same size or different sizes. Specifically, here Not limited. In the case where the sizes of the obtained n segments are all the same, the subsequent transformer module can process the n segments through a self-attention module, reducing the amount of computation; in the obtained n segments When the size of the block is different, the subsequent transformer module needs to process the n segmentation blocks through multiple self-attention modules. There are several different sizes (eg, x different sizes).
  • At least the corresponding x self-attention modules need to be configured, but the advantage of this different segmentation size is that for areas that require more detailed features (such as birds flying in the sky), the segmentation module can be divided into a larger number However, for areas that do not require too many detailed features (such as the sky), the segmentation module can be divided into a few large-sized slices, thus providing flexibility.
  • the segmentation module 802 After the segmentation module 802 obtains the n first feature blocks, the n first feature blocks are further input into the transformer module 803 for processing.
  • the transformer module 803 generates related information based on the n first feature blocks, where the related information is used to indicate the degree of correlation between any two first feature blocks in the n first feature blocks, and then the transformer module 803 calculates the correlation according to the correlation
  • the information generates n second feature blocks corresponding to the n first feature blocks one-to-one. That is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks. It should be noted here that the dimensions of the n first feature blocks input by the transformer module 803 are consistent with the dimensions of the n second feature blocks output.
  • the transformer module 803 including at least one encoder and at least one decoder as an example, how the transformer module 803 generates n first feature blocks one by one based on the relevant information
  • the corresponding n second feature blocks are described: first, the first correlation information is generated by the encoder, and according to the first correlation information, n third feature blocks corresponding to the n first feature blocks one-to-one are generated, The first correlation information is used to indicate the first correlation between any two of the n first feature blocks, and the dimensions of the n first feature blocks input by the encoder are related to the n third feature blocks The dimensions of the n third feature blocks are kept consistent; after that, the second related information is generated by the decoder, and according to the second related information, n second feature blocks corresponding to the n third feature blocks are generated. is used to indicate the second correlation between any two third feature blocks among the n third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions
  • the second related information incorporates the first task code
  • the first task code acts as an input to the decoder
  • the first task code is the corresponding identifier of the image enhancement task to which the input image belongs. Encoding, it can be known from what image enhancement task the n first feature blocks received by the transformer module 803 are input images.
  • the first task code may be sent by the encoder to the decoder, and then the first task code acts on the decoder as an input, and the first task code It can also be that when the first target neural network layer is triggered to receive the input image, the first task code is received through an instruction sent by the device deploying the model 800, and then the first task code acts as an input to the decoder. , specifically, the present application does not limit the acquisition method of the first task code.
  • each task code may be self-labeled according to the image enhancement task, or may be learned by the model itself, which is not specifically limited here.
  • the recombination module 804 After the transformer module 803 obtains n second feature blocks from the n first feature blocks based on the relevant information, the recombination module 804 will firstly splicing and recombining the positions of the n second feature blocks according to the space, so as to obtain the inputted first feature block. For a second feature map with the same dimension of the feature map, the operation of the recombination module 804 is the inverse operation of the segmentation module 802, which is not repeated here. It should be noted here that the size of the second feature map should be consistent with the size of the first feature map.
  • the reorganization module 804 splices and reorganizes the n second feature blocks to obtain a second feature map, and the second feature map is input into the second neural network layer 805, and the second neural network layer 805 receives the second feature map.
  • the image is decoded to obtain an output image, which is an enhanced image of the input image after being processed by the model 800 .
  • the input image in the training phase of the model 800, refers to the training samples in the training set; in the inference phase of the model 800, the input image refers to the real target image to be processed .
  • the model 800 does not have the selection module 401 of the model 400, the difference is that the model 800 has only one first neural network layer and one second neural network layer.
  • the processing process of each module and the type of the above-mentioned model 400 can be specifically referred to the corresponding implementation manner in which the model 400 performs image enhancement processing on the input image in FIG. 4 , and details are not repeated here.
  • a new model structure is obtained by combining the transformer module for processing natural language tasks with different neural network structures, which breaks through the limitation that the transformer module can only be used for processing natural language tasks. It can be applied to low-level vision tasks.
  • the model structure has a first neural network layer and a second neural network layer for processing a specific image enhancement task.
  • CNN as an excellent feature extractor, can perform well in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks), the model can pay attention to global information with the help of the transformer module, which can improve the image quality. Enhancement.
  • FIG. 9 is a system architecture diagram of the image enhancement system provided by the embodiment of the application.
  • the image enhancement system 200 includes an execution device 210, a training device 220, a database 230, a client The device 240 , the data storage system 250 and the data acquisition device 260 , and the execution device 210 includes a computing module 211 .
  • the data collection device 260 is used to obtain the open-source large-scale data set (that is, the training set) required by the user, and store the training set in the database 230 .
  • the model 201 is trained, and the trained model 201 obtained by training is then used on the execution device 210 (the execution device may also be referred to as a target device).
  • the execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .
  • the data storage system 250 may be placed in the execution device 210 , or the data storage system 250 may be an external memory relative to the execution device 210 .
  • the trained model 201 obtained through the training of the training device 220 can be applied to different systems or devices (that is, the execution device 210 ), and can specifically be an edge device or an end-side device, such as a mobile phone, a tablet, a laptop, a supervision system (such as cameras) and so on.
  • the execution device 210 is configured with an I/O interface 212 for data interaction with external devices, and a “user” can input data to the I/O interface 212 through the client device 240 .
  • the client device 240 may be a camera device of a monitoring system, and an image captured by the camera device is input to the computing module 211 of the execution device 210 as input data, and the computing module 211 performs image enhancement processing on the input image to obtain an enhanced image.
  • the obtained enhanced image can be output to the camera device for display or storage, or the obtained enhanced image can be directly displayed or stored on the display interface (if any) of the execution device 210; in addition, in some embodiments of the present application,
  • the client device 240 can also be integrated in the execution device 210.
  • the target image to be processed can be directly obtained through the mobile phone (for example, the image can be captured by the camera of the mobile phone) or Receive the target image sent by other devices (such as another mobile phone), and then perform image enhancement on the target image by the computing module 211 in the mobile phone to obtain an enhanced image, and directly present the enhanced image on the display interface or display interface of the mobile phone. stored in the phone.
  • the product form of the execution device 210 and the client device 240 is not limited here.
  • FIG. 9 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage The system 250 is an external memory relative to the execution device 210.
  • the data storage system 250 can also be placed in the execution device 210;
  • the client device 240 is an external device relative to the execution device 210.
  • Client device 240 may also be integrated in execution device 210 .
  • the training of the model 201 described in this embodiment of the present application may be implemented on the cloud side, for example, by a training device 220 on the cloud side (the training device 220 may be set on one or more servers or virtual machines) ) to obtain a training set, and train the model 201 according to multiple groups of training samples in the training set to obtain a trained model 201, after which the trained model 201 is sent to the execution device 210 for application, for example, sent to the execution
  • the device 210 performs image enhancement tasks such as image super-resolution reconstruction, denoising, and rain removal.
  • the training device 220 trains the model 201, and the trained model 201 is sent again.
  • the training of the model 201 described in the above-mentioned embodiment can also be implemented on the terminal side, that is, the training device 220 can be located on the terminal side, for example, it can be performed by a terminal device (such as a mobile phone, a smart watch, etc.) , wheeled mobile devices (such as self-driving vehicles, assisted driving vehicles, etc.), etc., to obtain a training set, and train the model 201 according to multiple sets of training samples in the training set to obtain a trained model 201.
  • the trained model 201 can be used directly in the terminal device, or can be sent by the terminal device to other devices for use.
  • the embodiment of the present application does not limit the device (cloud side or terminal side) on which the model 201 is trained or applied.
  • the model structure of the model 201 may be the structure of the model 400 corresponding to the above-mentioned FIG. 4 or the structure of the model 800 corresponding to the above-mentioned FIG. 8 . limited.
  • the training phase describes the process of how the training device 220 obtains the trained model 201 by using the training set maintained in the database 230 .
  • the model 201 may be the structure of the model 400 corresponding to FIG. 4 or the structure of the model 800 corresponding to FIG. 8 .
  • the structures of the models are different, and the training methods of the models are slightly different. introduce.
  • the structure of the model is the structure corresponding to the model 400 .
  • FIG. 10 is a schematic flowchart of a model training method provided by an embodiment of the present application, which may specifically include the following steps:
  • the training device acquires a training sample, where the training sample is any degraded image in the constructed training set, wherein each degraded image in the training set is obtained from a clear image through image degradation processing.
  • the training device first obtains a training sample from the constructed training set, the training sample may be any degraded image in the constructed training set, and each degraded image is obtained through image degradation processing through a clear image.
  • the acquisition of each clear image can be obtained by the user from an open source large-scale dataset, such as the clear image can be obtained from the ImageNet dataset, since image enhancement tasks can be of different types, such as denoising, deraining, super-resolution reconstruction etc., therefore, different types of training sets can be constructed according to different image enhancement tasks.
  • image enhancement tasks can be of different types, such as denoising, deraining, super-resolution reconstruction etc., therefore, different types of training sets can be constructed according to different image enhancement tasks.
  • image degradation models can be used to synthesize multiple images from unsupervised clear images.
  • Various types of degraded images are used to obtain training sets corresponding to various image enhancement tasks. For example, for super-resolution tasks, downsampling clear images on unsupervised datasets results in low-resolution degraded images.
  • the purpose of constructing a training set through image degradation processing is to obtain a large training set, which is due to the general shortage of supervised data in image processing (for example, for super-resolution There are only 2000 images on the DIV2K dataset for the rate task), so this application proposes to use an unsupervised dataset to train the model based on an open source large-scale dataset (eg, ImageNet dataset).
  • an open source large-scale dataset eg, ImageNet dataset
  • this application can use the ImageNet dataset, which consists of more than 1M high-diversity color images. Training images are cropped into 48 ⁇ 48 blocks with 3 channels for training, of which over 10 million blocks are used to train the model proposed in this application. Then, this application generates damaged images with 6 degradation types: 2x, 3x, and 4x bicubic linear interpolation downsampled images, 30, 50 noise level Gaussian noise, and adding rain streaks.
  • the training sample may also be a real low-quality labeled image, and when the low-quality image is used as a training sample, a corresponding high-quality clear image must also exist.
  • the types of training samples are not limited here.
  • the model may be pre-trained using degraded images that have undergone image degradation processing, and then the model may be fine-tuned using real low-quality labeled images.
  • this application randomly selects a task from Nt image enhancement tasks for training, and each task uses its corresponding first The target neural network layer, the second target neural network layer and the first task code are pre-trained. After the model is pretrained, the model can then be fine-tuned for a specific task using the corresponding dataset for that task.
  • the parameters of the corresponding first target neural network layer, the second target neural network layer and the shared structure in the middle of the model will be updated, while the first target neural network layer and the second target neural network layer corresponding to other tasks will be updated. frozen.
  • the training device inputs the training sample into the model, and the selection module in the model determines the first target neural network layer corresponding to the training sample, and the first target neural network layer is the m first neural network layer in the model.
  • the selection module in the model determines the first target neural network layer corresponding to the training sample, and the first target neural network layer is the m first neural network layer in the model.
  • the training device After the training device obtains the training sample, it will input the training sample into the model, and then the selection module in the model will determine the first target neural network layer corresponding to the training sample. Since a training sample will have a corresponding label indicating which type of image enhancement task the training sample belongs to, the label is used to indicate which first neural network layer the training sample should use to extract features. Then, the selection module of the model can determine that the training sample belongs to the first image enhancement task according to the label of the training sample, and further determine the first target neural network layer corresponding to the first image enhancement task.
  • the first target neural network layer will perform feature extraction on the training sample to obtain a feature map (which may be referred to as a first feature map).
  • the obtained first feature map will be further input to the segmentation module of the model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), where n ⁇ 2.
  • the process of segmenting the first feature map by the segmenting module may specifically be as follows: firstly segment the first feature map to obtain n segments, and then segment the first feature map into n segments.
  • Each of the n segmentation blocks is extended into a feature block (ie, a first feature block) represented by a one-dimensional vector, so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may have the same size or different sizes. Do limit.
  • the transformer module in the model generates n second feature blocks corresponding to the n first feature blocks one-to-one according to the related information, where the related information is used to indicate any two first features in the n first feature blocks correlation between blocks.
  • the n first feature blocks are further input into the transformer module in the model for processing.
  • the transformer module generates relevant information based on the n first feature blocks.
  • the correlation information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates n second feature blocks corresponding to the n first feature blocks one-to-one according to the related information . That is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks.
  • the dimensions of the n first feature blocks input by the transformer module are consistent with the dimensions of the n second feature blocks output.
  • the transformer module including at least one encoder and at least one decoder as an example, how the transformer module generates a one-to-one correspondence with n first feature blocks based on relevant information Description of n second feature blocks: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one.
  • a correlation information is used to indicate the first correlation between any two of the n first feature blocks, and the dimensions of the n first feature blocks and the n third feature blocks input by the encoder keep the same; after that, the decoder generates second related information, and according to the second related information, generates n second feature blocks corresponding to the n third feature blocks one-to-one, and the second related information is used to indicate The second correlation degree between any two third feature blocks in the n third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the first task code is integrated into the second related information, the first task code acts as an input to the decoder, and the first task code is the corresponding identifier of the first image enhancement task, which can also be considered as the first task code.
  • Each image enhancement task corresponds to a task code. Since the input image corresponding to each image enhancement task will be input to the corresponding first neural network layer, through the task code, not only can Knowing which input image of the image enhancement task the n first feature blocks received by the transformer module are from, and also knowing which first neural network layer performs the feature extraction operation for the n first feature blocks.
  • the transformer module in the model After the transformer module in the model obtains n second feature blocks from n first feature blocks based on the relevant information, it will splicing and reorganizing the n second feature blocks according to the relative spatial positions through the reorganization module in the model, and obtain the same as the input.
  • the first feature map has the same dimension as the second feature map.
  • the reorganization module in the model splices and reorganizes the n second feature blocks to obtain a second feature map, and the second feature map is input into the second target neural network layer uniquely corresponding to the first target neural network layer.
  • the target neural network layer belongs to one of m second neural network layers in the model.
  • the second target neural network layer then decodes the received second feature map, thereby obtaining an enhanced image of the training sample (which may be referred to as a first enhanced image).
  • the training device trains the model according to the first enhanced image, the clear image, and the loss function, to obtain a trained model, where the clear image corresponds to the training sample.
  • the training device After the training device obtains the first enhanced image output via the model, it will train the model according to the first enhanced image, the clear image and the loss function to obtain a trained model.
  • the training sample is obtained from the clear image through image degradation processing, so it can be said that the clear image corresponds to the training sample.
  • I clean represents a clear image
  • I corrupted represents the degraded image corresponding to the clear image
  • f represents the image degradation transformation
  • the loss function of the model is trained on such a synthetic training set It can be expressed as formula (15):
  • L1 represents the L1 loss function
  • i represents the degraded image of task i
  • the training objective is to close the similarity between the clear image and the first enhanced image.
  • this application introduces a contrastive learning approach to learn generic functions for unseen tasks. Specifically, taking a clear image x j as input, the output patched features generated by the decoder in the transformer model are represented as The goal of contrastive learning is to minimize the distance between decoder output encodings of feature blocks from the same image, while maximizing the distance between them and different images.
  • the loss function of contrastive learning can be shown in formula (16):
  • the loss function of the model It can be shown by the following formula (17):
  • this application combines the contrast loss with the supervision loss as the final loss function of the training model
  • the trained model can be deployed on a target device, such as an edge device or an end-side device, such as a mobile phone, tablet, laptop, supervision system (such as cameras) and so on.
  • a target device such as an edge device or an end-side device, such as a mobile phone, tablet, laptop, supervision system (such as cameras) and so on.
  • the model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying visual tasks.
  • the model structure has Multiple first neural network layers and multiple second neural network layers, different first/second neural network layers correspond to different image enhancement tasks, so the model can be used to process different image enhancement tasks after training, and compared to Since most of the existing models dealing with low-level visual tasks are based on CNN (CNN, as an excellent feature extractor, can flex its muscles in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks).
  • CNN CNN
  • the transformer module can pay attention to global information, which can improve the image enhancement effect.
  • the structure of the model is the structure corresponding to the model 800 .
  • FIG. 11 is a schematic flowchart of another model training method provided by an embodiment of the present application, which may specifically include the following steps:
  • the training device obtains a training sample, where the training sample is any degraded image in the constructed training set, wherein each degraded image in the training set is obtained from a clear image through image degradation processing.
  • step 1101 is similar to the foregoing step 1001, and details are not described here.
  • the training device inputs the training sample into the model, and the first neural network layer in the model performs feature extraction on the training sample to obtain a first feature map.
  • the training device After the training device obtains the training sample, it will input the training sample into the model, and the first neural network layer in the model will perform feature extraction on the training sample to obtain a first feature map.
  • the obtained first feature map will be further input to the segmentation module of the model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), where n ⁇ 2.
  • the process of segmenting the first feature map by the segmenting module may specifically be as follows: firstly segment the first feature map to obtain n segments, and then segment the first feature map into n segments.
  • Each of the n segmentation blocks is extended into a feature block (ie, a first feature block) represented by a one-dimensional vector, so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may have the same size or different sizes. Do limit.
  • the transformer module in the model generates n second feature blocks corresponding to the n first feature blocks one-to-one according to the related information, where the related information is used to indicate any two first features in the n first feature blocks correlation between blocks.
  • the n first feature blocks are further input into the transformer module in the model for processing.
  • the transformer module generates relevant information based on the n first feature blocks.
  • the correlation information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates n second feature blocks corresponding to the n first feature blocks one-to-one according to the related information . That is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks.
  • the dimensions of the n first feature blocks input by the transformer module are consistent with the dimensions of the n second feature blocks output.
  • the transformer module including at least one encoder and at least one decoder as an example, how the transformer module generates a one-to-one correspondence with n first feature blocks based on relevant information Description of n second feature blocks: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one.
  • a correlation information is used to indicate the first correlation between any two of the n first feature blocks, and the dimensions of the n first feature blocks and the n third feature blocks input by the encoder keep the same; after that, the decoder generates second related information, and according to the second related information, generates n second feature blocks corresponding to the n third feature blocks one-to-one, and the second related information is used to indicate The second correlation degree between any two third feature blocks in the n third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the second related information incorporates the first task code
  • the first task code acts as an input to the decoder
  • the first task code is the corresponding identifier of the image enhancement task to which the input image belongs. Encoding, you can know which input image of the image enhancement task the n first feature blocks received by the transformer module are from.
  • the transformer module in the model After the transformer module in the model obtains n second feature blocks from n first feature blocks based on the relevant information, it will splicing and reorganizing the n second feature blocks according to the relative spatial positions through the reorganization module in the model, and obtain the same as the input.
  • the first feature map has the same dimension as the second feature map.
  • the reorganization module in the model splices and reorganizes the n second feature blocks to obtain a second feature map, which will be input into the second neural network layer, and then the second neural network layer will receive the second feature.
  • the image is decoded to obtain an enhanced image of the training sample (which may be referred to as a first enhanced image).
  • the training device trains the model according to the first enhanced image, the clear image, and the loss function, to obtain a trained model, where the clear image corresponds to the training sample.
  • step 1107 is similar to the foregoing step 1008, and details are not described here.
  • the trained model can be deployed on a target device, such as an edge device or an end-side device, such as a mobile phone, tablet, laptop, supervision system (such as , camera) and so on.
  • a target device such as an edge device or an end-side device, such as a mobile phone, tablet, laptop, supervision system (such as , camera) and so on.
  • the trained model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying vision tasks.
  • the model structure has a first neural network layer and a second neural network layer, which are used to deal with a specific image enhancement task.
  • CNN is an excellent feature.
  • the extractor can flex its muscles in high-level vision tasks, but it is difficult to pay attention to global information when dealing with low-level vision tasks).
  • the application stage describes the process of how the execution device 210 uses the mature model 201 to perform corresponding image enhancement processing on the real target image to be processed.
  • the trained model 201 obtained at the stage may be the structure of the model 400 corresponding to FIG. 4 or the structure of the model 800 corresponding to FIG. 8 .
  • the structures of the models are different, and the method for performing image enhancement based on the trained model 201 is also applicable. There are slight differences, which will be introduced separately below.
  • the structure of the trained model is the structure corresponding to the model 400 .
  • FIG. 12 is a schematic flowchart of an image enhancement method provided by an embodiment of the present application, which may specifically include the following steps:
  • the executing device acquires the target image to be processed.
  • the executing device acquires the target image to be processed, such as an image captured by a mobile phone through a camera, an image captured by a monitoring device through a camera, and the like.
  • the execution device inputs the target image into the trained model, and the first target neural network layer corresponding to the target image is determined by the selection module in the trained model, and the first target neural network layer is the trained model One of the m first neural network layers in .
  • a trained model is deployed on the execution device. After the execution device obtains the target image, it will input the target image into the trained model, and the selection module in the trained model will determine the first target corresponding to the target image.
  • a neural network layer where the first target neural network layer is one of m first neural network layers in the trained model.
  • the execution device will additionally send an instruction to the trained model, the instruction It is used to indicate which type of image enhancement task the target image belongs to, that is, in the inference stage, the selection module of the trained model determines that the target image belongs to the first image enhancement task according to the received instruction, and further determine the first target neural network layer corresponding to the first image enhancement task.
  • the first target neural network layer will perform feature extraction on the target image to obtain a feature map (which may be referred to as a first feature map).
  • the obtained first feature map will be further input into the segmentation module of the trained model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), n ⁇ 2.
  • the process of segmenting the first feature map by the segmenting module may specifically be as follows: firstly segment the first feature map to obtain n segments, and then segment the first feature map into n segments.
  • Each of the n segmentation blocks is extended into a feature block (ie, a first feature block) represented by a one-dimensional vector, so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may have the same size or different sizes. Do limit.
  • the transformer module in the trained model generates n second feature blocks corresponding to the n first feature blocks one-to-one according to relevant information, where the relevant information is used to indicate any two of the n first feature blocks The correlation between the first feature blocks.
  • the n first feature blocks are further input into the transformer module in the trained model for processing, and the transformer module is based on the n first features block, and generate related information, the related information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates a one-to-one correspondence with the n first feature blocks according to the related information
  • the n second feature blocks of that is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks. It should be noted here that the dimensions of the n first feature blocks input by the transformer module are consistent with the dimensions of the n second feature blocks output.
  • the transformer module including at least one encoder and at least one decoder as an example, how the transformer module generates a one-to-one correspondence with n first feature blocks based on relevant information Description of n second feature blocks: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one.
  • a correlation information is used to indicate the first correlation between any two of the n first feature blocks, and the dimensions of the n first feature blocks and the n third feature blocks input by the encoder keep the same; after that, the decoder generates second related information, and according to the second related information, generates n second feature blocks corresponding to the n third feature blocks one-to-one, and the second related information is used to indicate The second correlation degree between any two third feature blocks in the n third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the first task code is integrated into the second related information, the first task code acts as an input to the decoder, and the first task code is the corresponding identifier of the first image enhancement task, which can also be considered as the first task code.
  • Each image enhancement task corresponds to a task code. Since the input image corresponding to each image enhancement task will be input to the corresponding first neural network layer, through the task code, not only can Knowing which input image of the image enhancement task the n first feature blocks received by the transformer module are from, and also knowing which first neural network layer performs the feature extraction operation for the n first feature blocks.
  • the transformer module in the trained model After the transformer module in the trained model obtains n second feature blocks from the n first feature blocks based on the relevant information, the reorganization module in the trained model will perform the processing on the n second feature blocks according to their relative spatial positions. After splicing and recombination, a second feature map with the same dimension as the input first feature map is obtained.
  • the reorganization module in the trained model splices and reorganizes the n second feature blocks to obtain a second feature map, and the second feature map is input into the second target neural network layer uniquely corresponding to the first target neural network layer,
  • the second target neural network layer belongs to one of m second neural network layers in the trained model. Then the second target neural network layer decodes the received second feature map, thereby obtaining an enhanced image of the training target image (which may be referred to as a second enhanced image).
  • the trained model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying vision tasks.
  • the model structure has multiple first neural network layers and multiple second neural network layers, and different first/second neural network layers correspond to different image enhancement tasks, so the model can be used to process different image enhancement tasks after training, And compared with the existing models that deal with low-level visual tasks, most of them are based on CNN (CNN, as an excellent feature extractor, can flex its muscles in high-level visual tasks, but it is difficult to pay attention to global information when dealing with low-level visual tasks). With the help of the transformer module, the model can pay attention to the global information, which can improve the image enhancement effect.
  • CNN CNN
  • the transformer module the model can pay attention to the global information, which can improve the image enhancement effect.
  • the structure of the trained model is the structure corresponding to the model 800 .
  • FIG. 13 is another schematic flowchart of an image enhancement method provided by an embodiment of the present application, which may specifically include the following steps:
  • the executing device acquires the target image to be processed.
  • step 1301 is similar to the foregoing step 1201, and details are not described here.
  • the execution device inputs the target image into the trained model, and the first neural network layer in the trained model performs feature extraction on the target image to obtain a first feature map.
  • a trained model is deployed on the execution device. After the execution device obtains the target image, it will input the target image into the trained model, and the first neural network layer in the trained model will perform feature extraction on the target image. Get the first feature map.
  • the obtained first feature map will be further input into the segmentation module of the trained model, and the first feature map will be segmented by the segmentation module to obtain n feature blocks (which may be referred to as first feature blocks), n ⁇ 2.
  • the process of segmenting the first feature map by the segmenting module may specifically be as follows: firstly segment the first feature map to obtain n segments, and then segment the first feature map into n segments.
  • Each of the n segmentation blocks is extended into a feature block (ie, a first feature block) represented by a one-dimensional vector, so that n first feature blocks can be obtained.
  • the segmentation module performs segmentation on the first feature map, and the obtained n segmentation blocks may have the same size or different sizes. Do limit.
  • the transformer module in the trained model generates n second feature blocks corresponding to the n first feature blocks one-to-one according to relevant information, where the relevant information is used to indicate any two of the n first feature blocks The correlation between the first feature blocks.
  • the n first feature blocks are further input into the transformer module in the trained model for processing, and the transformer module is based on the n first features block, and generate related information, the related information is used to indicate the correlation between any two first feature blocks in the n first feature blocks, and then the transformer module generates a one-to-one correspondence with the n first feature blocks according to the related information
  • the n second feature blocks of that is to say, each first feature block, in addition to its own feature information, also fuses feature information of other first feature blocks according to the correlation between itself and other first feature blocks. It should be noted here that the dimensions of the n first feature blocks input by the transformer module are consistent with the dimensions of the n second feature blocks output.
  • the transformer module including at least one encoder and at least one decoder as an example, how the transformer module generates a one-to-one correspondence with n first feature blocks based on relevant information Description of n second feature blocks: First, the encoder generates first related information, and according to the first related information, generates n third feature blocks corresponding to the n first feature blocks one-to-one.
  • a correlation information is used to indicate the first correlation between any two of the n first feature blocks, and the dimensions of the n first feature blocks and the n third feature blocks input by the encoder keep the same; after that, the decoder generates second related information, and according to the second related information, generates n second feature blocks corresponding to the n third feature blocks one-to-one, and the second related information is used to indicate The second correlation degree between any two third feature blocks in the n third feature blocks, and the dimensions of the n third feature blocks input by the decoder are consistent with the dimensions of the n second feature blocks.
  • the second related information incorporates the first task code
  • the first task code acts as an input to the decoder
  • the first task code is the corresponding identifier of the image enhancement task to which the input image belongs. Encoding, it can be known that the n first feature blocks received by the transformer module are from the input image of the image enhancement task.
  • the transformer module in the trained model obtains n second feature blocks from n first feature blocks based on relevant information
  • the n second feature blocks are spliced according to their relative spatial positions through the reorganization module in the trained model. Recombination to obtain a second feature map with the same dimension as the input first feature map.
  • the reorganization module in the trained model splices and reorganizes n second feature blocks to obtain a second feature map, which will be input into the second neural network layer, and then the second neural network layer will receive the received
  • the second feature map is decoded to obtain an enhanced image of the training sample (which may be referred to as a second enhanced image).
  • the trained model combines the transformer module for processing natural language tasks and different neural network structures, breaking through the limitation that the transformer module can only be used for processing natural language tasks.
  • the model structure can be applied to the underlying vision tasks.
  • the model structure has a first neural network layer and a second neural network layer, which are used to deal with a specific image enhancement task.
  • CNN is an excellent feature.
  • the extractor can flex its muscles in high-level vision tasks, but it is difficult to pay attention to global information when dealing with low-level vision tasks).
  • model structure constructed in the embodiment of the present application and the trained model obtained by the training of the model can be applied to various image enhancement tasks.
  • the models trained in the embodiments of this application can be used in any field to perform image enhancement task processing (such as super-resolution reconstruction, denoising, dehazing, deraining, etc.).
  • An application scenario is introduced.
  • Camera photo inpainting is a very important technology, which has great use value in processing mobile phone imaging effects and other scenarios.
  • the main method of camera image inpainting is to use multiple convolutional neural network models for different image enhancement tasks.
  • Using the model structure constructed in this application, as shown in FIG. 14 can realize different types of image enhancement tasks through one model, and can achieve better effects than multiple task-specific convolutional neural network models.
  • the model trained in this application can be used for photo optimization of terminals (such as mobile phones, smart watches, personal computers, etc.).
  • terminals such as mobile phones, smart watches, personal computers, etc.
  • the model trained in this application can be applied to the mobile phone.
  • the trained model effectively retains the detailed information of the image pixels. After optimization The image quality of the mobile phone is also clearer than the image optimized by the existing neural network, which can bring a better user experience to the user and improve the quality of mobile phone products.
  • the trained model described in this application can not only be applied to the above-mentioned application scenarios, but also can be applied to various sub-fields in the field of artificial intelligence, as long as the fields and equipment of neural networks can be used,
  • the trained models provided in the embodiments of the present application can all be applied, which are not illustrated here.
  • Table 1 shows the comparison results between the present application and the best model based on CNN . It can be seen from Table 1 that the model trained by using the model constructed by this application and the training method can achieve performance that surpasses that of the CNN model on a variety of image enhancement tasks and on a variety of data sets. In addition, it should be noted that different superresolution ratios need to use different CNNs, and the model proposed in this application can use one model for different types of image enhancement tasks.
  • CNN ⁇ 3 34.72 30.66 29.31 29.03 this application ⁇ 3 34.81 30.85 29.38 29.38 CNN ⁇ 4 32.57 28.85 27.77 26.84 this application ⁇ 4 32.64 29.01 27.82 27.26
  • FIG. 15 is a schematic diagram of a training device provided by an embodiment of the application.
  • the training device 1500 may specifically include: an acquisition module 1501, an input module 1502, and a training module 1503, wherein the acquisition module 1501 is used for training samples , the training sample is any degraded image in the constructed training set, wherein each degraded image in the training set is obtained from a clear image through image degradation processing; the input module 1502 is used to deploy on the training device 1500 The model input the training sample, the training sample is processed by the model, and the first enhanced image of the training sample is obtained; the training module 1503 is used to deploy the training device according to the first enhanced image, the clear image and the loss function. The model on 1500 is trained to obtain a trained model, and the clear image corresponds to the training sample.
  • FIG. 16 is a schematic diagram of an execution device provided by an embodiment of the present application.
  • the execution device 1600 includes: an acquisition module 1601 and an input module 1602 , wherein the acquisition module 1601 For obtaining the target image to be processed; the input module 1602 is used to input the target image into the trained model deployed on the execution device 1600, and the trained model processes the target image to obtain the target image of the second enhanced image.
  • FIG. 17 is a schematic structural diagram of the training device provided by the embodiment of the present application.
  • the described training device 1500 is used to implement the functions of the training device 1500 in the embodiment corresponding to FIG. 15 .
  • the training device 1700 is implemented by one or more servers. Differences, which may include one or more central processing units (CPUs) 1722 and memory 1732, one or more storage media 1730 (eg, one or more mass storage devices) that store applications 1742 or data 1744.
  • the memory 1732 and the storage medium 1730 may be short-term storage or persistent storage.
  • the program stored in the storage medium 1730 may include one or more modules (not shown), and each module may include a series of instructions to operate on the training device 1700. Further, the central processing unit 1722 may be configured to communicate with the storage medium 1730 to execute a series of instruction operations in the storage medium 1730 on the training device 1700 .
  • Training device 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input and output interfaces 1758, and/or, one or more operating systems 1741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • operating systems 1741 such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the central processing unit 1722 is configured to execute the training method of the model executed by the training device in the embodiment corresponding to FIG. 10 or FIG. 11 .
  • FIG. 18 is a schematic structural diagram of the execution device provided by the embodiment of the present application.
  • the execution device 1800 may specifically be represented as various terminal devices, such as virtual Realistic VR devices, mobile phones, tablets, laptops, smart wearable devices, monitoring data processing devices or radar data processing devices, etc., are not limited here.
  • the execution device 1600 described in the embodiment corresponding to FIG. 16 may be deployed on the execution device 1800 to implement the functions of the execution device 1600 in the embodiment corresponding to FIG. 16 .
  • the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803, and a memory 1804 (wherein the number of processors 1803 in the execution device 1800 may be one or more, and one processor is taken as an example in FIG. 18 ) , wherein the processor 1803 may include an application processor 18031 and a communication processor 18032.
  • the receiver 1801, the transmitter 1802, the processor 1803, and the memory 1804 may be connected by a bus or otherwise.
  • Memory 1804 may include read-only memory and random access memory, and provides instructions and data to processor 1803 .
  • a portion of memory 1804 may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory 1804 stores processors and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations.
  • the processor 1803 controls the operation of the execution device 1800 .
  • various components of the execution device 1800 are coupled together through a bus system, where the bus system may include a power bus, a control bus, a status signal bus, and the like in addition to a data bus.
  • the various buses are referred to as bus systems in the figures.
  • the method disclosed in the above-mentioned embodiment corresponding to FIG. 12 or FIG. 13 of the present application may be applied to the processor 1803 or implemented by the processor 1803 .
  • the processor 1803 may be an integrated circuit chip, which has signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 1803 or an instruction in the form of software.
  • the above-mentioned processor 1803 may be a general-purpose processor, a digital signal processing (DSP), a microprocessor or a microcontroller, and may further include an application specific integrated circuit (ASIC), a field programmable Field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • ASIC application specific integrated circuit
  • FPGA field programmable Field-programmable gate array
  • the processor 1803 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments corresponding to FIG. 12 or FIG. 13 of the present application.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor.
  • the software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art.
  • the storage medium is located in the memory 1804, and the processor 1803 reads the information in the memory 1804, and completes the steps of the above method in combination with its hardware.
  • the receiver 1801 may be used to receive input numerical or character information, and to generate signal input related to performing the relevant settings and function control of the device 1800 .
  • the transmitter 1802 can be used to output digital or character information through the first interface; the transmitter 1802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1802 can also include a display device such as a display screen .
  • the processor 1803 is configured to perform image enhancement processing on the input target image by using the trained model to obtain a corresponding enhanced image.
  • the trained model may be obtained through the training method corresponding to FIG. 10 or FIG. 11 of the present application.
  • Embodiments of the present application further provide a computer-readable storage medium, where a program for performing signal processing is stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, it causes the computer to execute the programs described in the foregoing embodiments.
  • the steps performed by the training device, or the computer is caused to perform the steps performed by the execution device described in the embodiment shown in FIG. 16 .
  • the training device, execution device, etc. provided by the embodiments of the present application may be specifically a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or circuit etc.
  • the processing unit can execute the computer-executed instructions stored in the storage unit, so that the chip in the training device executes the steps performed by the training device described in the above-described embodiment, or, the chip in the execution device executes the steps shown in the aforementioned FIG. 16 .
  • the steps described in the embodiments are executed by the device.
  • the storage unit is a storage unit in the chip, such as a register, a cache, etc.
  • the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.
  • ROM Read-only memory
  • RAM random access memory
  • FIG. 19 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • the chip may be represented as a neural network processor NPU 200, and the NPU 200 is mounted as a co-processor to the main CPU (Host CPU), tasks are allocated by the Host CPU.
  • the core part of the NPU is the arithmetic circuit 2003, which is controlled by the controller 2004 to extract the matrix data in the memory and perform multiplication operations.
  • the arithmetic circuit 2003 includes multiple processing units (process engines, PEs). In some implementations, the arithmetic circuit 2003 is a two-dimensional systolic array. The arithmetic circuit 2003 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2003 is a general-purpose matrix processor.
  • the arithmetic circuit fetches the data corresponding to the matrix B from the weight memory 2002 and buffers it on each PE in the arithmetic circuit.
  • the arithmetic circuit fetches the data of matrix A and matrix B from the input memory 2001 to perform matrix operation, and stores the partial result or final result of the matrix in an accumulator 2008 .
  • Unified memory 2006 is used to store input data and output data.
  • the weight data is directly passed through the storage unit access controller (direct memory access controller, DMAC) 2005, and the DMAC is transferred to the weight memory 2002.
  • Input data is also transferred to unified memory 2006 via the DMAC.
  • the bus interface unit 2010 (bus interface unit, BIU for short) is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (Instruction Fetch Buffer, IFB) 2009.
  • the bus interface unit 2010 is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
  • the DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 2006 , the weight data to the weight memory 2002 , or the input data to the input memory 2001 .
  • the vector calculation unit 2007 includes a plurality of operation processing units, and further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on, if necessary. It is mainly used for non-convolutional/fully connected layer network computation in neural networks, such as Batch Normalization, pixel-level summation, and upsampling of feature planes.
  • the vector computation unit 2007 can store the processed output vectors to the unified memory 2006 .
  • the vector calculation unit 2007 may apply a linear function and/or a non-linear function to the output of the operation circuit 2003, such as linear interpolation of the feature plane extracted by the convolutional layer, such as a vector of accumulated values, to generate activation values.
  • the vector computation unit 2007 generates normalized values, pixel-level summed values, or both.
  • the vector of processed outputs can be used as activation input to the arithmetic circuit 2003, eg, for use in subsequent layers in a neural network.
  • the instruction fetch memory (instruction fetch buffer) 2009 connected to the controller 2004 is used to store the instructions used by the controller 2004;
  • Unified memory 2006, input memory 2001, weight memory 2002 and instruction fetch memory 2009 are all On-Chip memories. External memory is private to the NPU hardware architecture.
  • the processor mentioned in any one of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the method in the first aspect.
  • the device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be A physical unit, which can be located in one place or distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general purpose computer, special purpose computer, computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be retrieved from a website, computer, training device, or data
  • the center transmits to another website site, computer, training equipment or data center by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • wire eg, coaxial cable, fiber optic, digital subscriber line (DSL)
  • wireless eg, infrared, wireless, microwave, etc.
  • the computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a training device, a data center, or the like that includes an integration of one or more available media.
  • the usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了一种模型结构、模型训练方法、图像增强方法及设备，可应用于人工智能领域中的计算机视觉领域，该模型结构包括：选择模块、多个第一神经网络层、切分模块、transformer模块、重组模块及多个第二神经网络层，模型突破了transformer模块只能用于处理自然语言任务的局限，可应用在底层视觉任务中，该模型具备多个第一/二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，相比于现有处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务性能好，但在处理底层视觉任务时难以关注全局信息)，该模型借助transformer模块可关注到全局信息，提高了图像增强效果。

Description

一种模型结构、模型训练方法、图像增强方法及设备
本申请要求于2020年12月1日提交中国专利局、申请号为202011382775.1、申请名称为“一种模型结构、模型训练方法、图像增强方法及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机视觉领域，尤其涉及一种模型结构、模型训练方法、图像增强方法及设备。
背景技术
计算机视觉是各个应用领域(如制造业、检验、文档分析、医疗诊断，和军事等领域)中各种智能/自主系统中不可分割的一部分，它是一门关于如何运用照相机/摄像机和计算机来获取人们所需的被拍摄对象的数据与信息的学问。根据是否需要用到图像的语义信息，计算机视觉任务可分为底层视觉任务和高层视觉任务这两类，底层视觉任务一般是指像素级别的图像处理任务，不需要用到图像的语义信息，或者最多用到底层特征(如，图像的边缘、纹理等)，这些任务有图像增强(如，去噪、去模糊、去雨、超分辨重建等)、图像加密等。高层视觉任务则需要用到图像的语义信息，提取的特征是高层特征，比如目标定位、识别、检测、分类、分割，以及用到语义特征的图像生成。
现有的处理底层视觉任务的模型大多是基于卷积神经网络(convolutional neural networks，CNN)，以图像增强任务为例，如图1中的(a)子示意图所示，首先初始化一个CNN，之后按照图像增强任务的不同，制定各自对应的损失函数，利用训练数据对该CNN进行训练，在该CNN达到收敛状态后，结束训练，得到一个训练好的CNN，最后将得到的该训练好的CNN应用于各自指定的图像增强任务中。
CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息，并且每个图像增强任务对需要训练对应的CNN，如图1中的(b)子示意图所示，若有3个不同的图像增强任务(去噪、去雾、去雨)，就需要对应训练3个不同的CNN，不具备通用性。
发明内容
本申请实施例提供了一种模型结构、模型训练方法、图像增强方法及设备，将用于处理自然语言任务的transformer模块结合不同的神经网络结构得到一种新的模型结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
基于此，本申请实施例提供以下技术方案：
第一方面，本申请实施例首先提供一种模型结构，可用于人工智能领域中的计算机视觉领域，该模型的结构包括：选择模块、m个第一神经网络层、m个第二神经网络层、切分模块、重组模块以及transformer模块，每个第一神经网络层唯一对应一个第二神经网络层，每个第一神经网络层也可以称为头模块或头结构，每个第二神经网络层也可以称为尾模块或尾结构，其中，m≥2。选择模块，用于获取输入图像，并确定与所述输入图像对应的第一目标神经网络层，所述第一目标神经网络层为所述m个第一神经网络层中的一个。该模型的选择模块根据输入图像确定出与该输入图像对应的第一目标神经网络层后，会将该输入图像输入至该第一目标神经网络层，第一目标神经网络层，就用于对输入图像进行特征提取，得到特征图(可称为第一特征图)。得到的第一特征图会进一步输入至切分模块，该切分模块，就用于对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到transformer模块中进行处理，transformer模块，则用于根据相关信息，生成与n个第一特征块一一对应的n个第二特征块，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将发送给重组模块，该重组模块，就用于对n个第二特征块按照空间相对位置进行拼接重组，从而得到与输入的第一特征图维度一致的第二特征图，该重组模块的操作是切分模块的逆操作。重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至与第一目标神经网络层唯一对应的第二目标神经网络层中，该第二目标神经网络层属于m个第二神经网络层中的一个。该第二目标神经网络层，用于对第二特征图进行解码，得到输出图像。
在本申请上述实施方式中，将用于处理自然语言任务的transformer模块结合不同的神经网络结构得到一种新的模型结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
在第一方面的一种可能实现方式中，选择模块接收到输入图像后，会判断该输入图像应该由哪个第一神经网络层去做特征提取操作，具体地，选择模块用于先确定该输入图像属于哪一种类型的图像增强任务，再将该输入图像输入到对应该任务的第一神经网络层去。该输入图像所属的图像增强任务可称为第一图像增强任务，假设该第一图像增强任务对应的是第一目标神经网络层，那么选择模块还用于将接收到的该输入图像输入到第一目标神经网络层中。
在本申请上述实施方式中，具体阐述了如何模型的选择模块确定与输入图像对应的第 一目标神经网络层是通过第一图像增强任务识别的，具备可实现性。
在第一方面的一种可能实现方式中，当该模型是处于模型的训练阶段，那么输入图像就为训练集中的训练样本，此时每个训练样本都会有对应的标签指示该训练样本属于哪一类图像增强任务，该标签就用于指示该训练样本应该由哪个第一神经网络层去提取特征。那么该模型的选择模块就可根据该训练样本的标签以确定该训练样本属于第一图像增强任务。
在本申请上述实施方式中，具体阐述了当输入图像是训练样本时，选择模块如何确定与该训练样本对应的图像增强任务，具备灵活性。
在第一方面的一种可能实现方式中，当该模型是处于模型的推理阶段，那么输入图像就为真实的待处理的目标图像，在这个过程中，选择模块除了会接收到该输入图像，还会接收到部署该模型的设备发出的指令，该指令就是用于指示该目标图像是属于哪一类图像增强任务，也就是说，在推理阶段，该模型的选择模块是根据接收到的指令以确定该目标图像是属于第一图像增强任务。
在本申请上述实施方式中，具体阐述了当输入图像是待处理的目标图像时，选择模块如何确定与该目标图像对应的图像增强任务，具备灵活性。
在第一方面的一种可能实现方式中，transformer模块包括编码器和解码器，这种情况transformer模块基于相关信息生成与n个第一特征块一一对应的n个第二特征块可以是：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为第一图像增强任务的对应标识，也可以认为是第一目标神经网络层的对应标识，每个图像增强任务都对应有一个任务编码，由于每个图像增强任务对应的输入图像会输入对应的第一神经网络层，因此，通过该任务编码，不仅可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像，还可以知道这n个第一特征块是由哪个第一神经网络层进行的特征提取操作。
在本申请上述实施方式中，阐述了transformer模块具体是如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块，具备可实现性。
在第一方面的一种可能实现方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
在本申请上述实施方式中，阐述了切分模块如何对第一特征图进行切分的执行过程，具备可实现性。
在第一方面的一种可能实现方式中，切分模块对第一特征图进行切分，得到的n个切 分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
在本申请上述实施方式中，阐述了切分模块切分得到的n个切分块的尺寸可以相同，也可以不同，可根据需求预设，具备选择性。
本申请实施例第二方面还提供一种模型结构，该模型具体可以包括：第一神经网络层1、切分模块、transformer模块、重组模块以及第二神经网络层，其中，第一神经网络层也可以称为头模块或头结构，第二神经网络层也可以称为尾模块或尾结构。在本申请实施例中，由于第一神经网络层和第二神经网络层各自只有一个，因此在该模型中，就不存在选择模块。第一神经网络层，用于对输入图像进行特征提取，得到特征图(可称为第一特征图)，之后该第一特征图输入至切分模块，该切分模块，则用于对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到transformer模块中进行处理。transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示这n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，重组模块，用于对这n个第二特征块按照空间相对位置进行拼接重组，从而得到与输入的第一特征图维度一致的第二特征图。重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至第二神经网络层中，由该第二神经网络层对接收到的第二特征图进行解码，得到输出图像，该输出图像就是经过了模型处理后该输入图像的增强图像。
在本申请上述实施方式中，将用于处理自然语言任务的transformer模块结合不同的神经网络结构得到一种新的模型结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备一个第一神经网络层和一个第二神经网络层，用于处理一个特定的图像增强任务，相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
在第二方面的一种可能实现方式中，transformer模块包括编码器和解码器，这种情况transformer模块基于相关信息生成与n个第一特征块一一对应的n个第二特征块可以是：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块 一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为输入图像所属的图像增强任务的对应标识，通过该任务编码，可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像。
在本申请上述实施方式中，阐述了transformer模块具体是如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块，具备可实现性。
在第二方面的一种可能实现方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
在本申请上述实施方式中，阐述了切分模块如何对第一特征图进行切分的执行过程，具备可实现性。
在第二方面的一种可能实现方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
在本申请上述实施方式中，阐述了切分模块切分得到的n个切分块的尺寸可以相同，也可以不同，可根据需求预设，具备选择性。
本申请实施例第三方面提供一种模型的训练方法，该方法包括：训练设备首先从构建的训练集中获取训练样本，该训练样本可以是构建的该训练集中的任意一个退化图像，而每个退化图像又是经由一个清晰图像经过图像退化处理得到的。训练设备获取到训练样本后，会将该训练样本输入模型中，然后由模型中的选择模块确定与该训练样本对应的第一目标神经网络层。该第一目标神经网络层将会对该训练样本进行特征提取，得到特征图(可称为第一特征图)。得到的第一特征图会进一步输入至模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成 与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至与第一目标神经网络层唯一对应的第二目标神经网络层中，该第二目标神经网络层属于模型中m个第二神经网络层中的一个。然后该第二目标神经网络层对接收到的第二特征图进行解码，从而得到训练样本的增强图像(可称为第一增强图像)。训练设备得到经由模型输出的第一增强图像后，将根据该第一增强图像、清晰图像和损失函数对该模型进行训练，以得到训练后的模型。其中，该训练样本就是该清晰图像通过图像退化处理得到的，因此，可称为该清晰图像与该训练样本对应。
在本申请上述实施例中，具体阐述了如何对本申请构建的一种模型进行训练，得到训练后的模型。该模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
在第三方面的一种可能实现方式中，由于训练样本都会有对应的标签指示该训练样本属于哪一类图像增强任务，该标签就用于指示该训练样本应该由哪个第一神经网络层去提取特征。那么该模型的选择模块就可根据该训练样本的标签以确定该训练样本属于第一图像增强任务，并进一步确定与该第一图像增强任务对应的第一目标神经网络层。
在本申请上述实施方式中，具体阐述了选择模块如何确定与该训练样本对应的图像增强任务，具备灵活性。
在第三方面的一种可能实现方式中，transformer模块包括编码器和解码器，这种情况transformer模块基于相关信息生成与n个第一特征块一一对应的n个第二特征块可以是：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为第一图像增强任务的对应标识，也可以认为是第一目标神经网络层的对应标识，每个图像增强任务都对应有一个任务编码，由于每个图像增强任务对应的输入图像会输入对应的第一神经网络层，因此，通过该任务 编码，不仅可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像，还可以知道这n个第一特征块是由哪个第一神经网络层进行的特征提取操作。
在本申请上述实施方式中，阐述了transformer模块具体是如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块，具备可实现性。
在第三方面的一种可能实现方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
在本申请上述实施方式中，阐述了切分模块如何对第一特征图进行切分的执行过程，具备可实现性。
在第三方面的一种可能实现方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
在本申请上述实施方式中，阐述了切分模块切分得到的n个切分块的尺寸可以相同，也可以不同，可根据需求预设，具备选择性。
在第三方面的一种可能实现方式中，训练后的模型可部署在目标设备上，如，部署在边缘设备或端侧设备上，例如，手机、平板、笔记本电脑、监督系统(如，摄像头)等等。
本申请实施例第四方面还提供了一种模型的训练方法，该方法可以包括：训练设备获取训练样本，训练样本为构建的训练集中任意一个退化图像，其中，训练集中的每个退化图像由一个清晰图像经过图像退化处理得到。训练设备获取到训练样本后，会将该训练样本输入模型中，由模型中的第一神经网络层对训练样本进行特征提取，得到第一特征图。得到的第一特征图会进一步输入至模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至第二神经网络层中，然后该第二神经网络层对接收到的第二特征 图进行解码，从而得到训练样本的增强图像(可称为第一增强图像)。
在本申请上述实施例中，具体阐述了如何对本申请构建的另一种模型进行训练，得到训练后的模型。该训练后的模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备一个第一神经网络层和一个第二神经网络层，用于处理一个特定的图像增强任务，相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
在第四方面的一种可能实现方式中，transformer模块包括编码器和解码器，这种情况transformer模块基于相关信息生成与n个第一特征块一一对应的n个第二特征块可以是：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为输入图像所属的图像增强任务的对应标识，通过该任务编码，可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像。
在本申请上述实施方式中，阐述了transformer模块具体是如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块，具备可实现性。
在第四方面的一种可能实现方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
在本申请上述实施方式中，阐述了切分模块如何对第一特征图进行切分的执行过程，具备可实现性。
在第四方面的一种可能实现方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
在本申请上述实施方式中，阐述了切分模块切分得到的n个切分块的尺寸可以相同，也可以不同，可根据需求预设，具备选择性。
在第四方面的一种可能实现方式中，训练后的模型可部署在目标设备上，如，部署在边缘设备或端侧设备上，例如，手机、平板、笔记本电脑、监督系统(如，摄像头)等等。
本申请实施例第五方面提供了一种图像增强方法，该方法包括：执行设备(即上述所述的目标设备)获取待处理的目标图像，如，由手机通过摄像头拍摄到的图像，由监控设备通过摄像头拍摄下的图像等。该执行设备上部署有训练后的模型，执行设备获取到目标图像后，会将该目标图像输入训练后的模型，由该训练后的模型中的选择模块确定与该目标图像对应的第一目标神经网络层，该第一目标神经网络层为训练后的模型中m个第一神经网络层中的一个。该第一目标神经网络层将会对该目标图像进行特征提取，得到特征图(可称为第一特征图)。得到的第一特征图会进一步输入至该训练后的模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。训练后的模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到该训练后的模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。训练后的模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过该训练后的模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。训练后的模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至与第一目标神经网络层唯一对应的第二目标神经网络层中，该第二目标神经网络层属于该训练后的模型中m个第二神经网络层中的一个。然后该第二目标神经网络层对接收到的第二特征图进行解码，从而得到训目标图像的增强图像(可称为第二增强图像)。
在本申请上述实施例中，具体阐述了如何对本申请训练后的模型进行实际应用，从而得到目标图像对应的增强图像。该训练后的模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
在第五方面的一种可能实现方式中，由于真实的待处理的目标图像不具有标签，训练后的模型感知不到该目标图像对应哪种类型的图像增强任务，这时，执行设备会额外向该训练后的模型发送一个指令，该指令用于指示该目标图像是属于哪一类图像增强任务，也 就是说，在推理阶段，该训练后的模型的选择模块是根据接收到的指令以确定该目标图像是属于第一图像增强任务，并进一步确定与该第一图像增强任务对应的第一目标神经网络层。
在本申请上述实施方式中，具体阐述了选择模块如何确定与该目标图像对应的图像增强任务，具备灵活性。
在第五方面的一种可能实现方式中，transformer模块包括编码器和解码器，这种情况transformer模块基于相关信息生成与n个第一特征块一一对应的n个第二特征块可以是：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为第一图像增强任务的对应标识，也可以认为是第一目标神经网络层的对应标识，每个图像增强任务都对应有一个任务编码，由于每个图像增强任务对应的输入图像会输入对应的第一神经网络层，因此，通过该任务编码，不仅可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像，还可以知道这n个第一特征块是由哪个第一神经网络层进行的特征提取操作。
在本申请上述实施方式中，阐述了transformer模块具体是如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块，具备可实现性。
在第五方面的一种可能实现方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
在本申请上述实施方式中，阐述了切分模块如何对第一特征图进行切分的执行过程，具备可实现性。
在第五方面的一种可能实现方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
在本申请上述实施方式中，阐述了切分模块切分得到的n个切分块的尺寸可以相同，也可以不同，可根据需求预设，具备选择性。
本申请实施例第六方面提供了一种图像增强方法，该方法包括：执行设备(即上述所述的目标设备)获取待处理的目标图像，如，由手机通过摄像头拍摄到的图像，由监控设备通过摄像头拍摄下的图像等。该执行设备上部署有训练后的模型，执行设备获取到目标图像后，会将该目标图像输入训练后的模型，由该训练后的模型中的第一神经网络层对目标图像进行特征提取，得到第一特征图。得到的第一特征图会进一步输入至该训练后的模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。训练后的模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到该训练后的模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。训练后的模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过训练后的模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。训练后的模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至第二神经网络层中，然后该第二神经网络层对接收到的第二特征图进行解码，从而得到训练样本的增强图像(可称为第二增强图像)。
在本申请上述实施例中，具体阐述了如何对本申请训练后的模型进行实际应用，从而得到目标图像对应的增强图像。该训练后的模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备一个第一神经网络层和一个第二神经网络层，用于处理一个特定的图像增强任务，相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
在第六方面的一种可能实现方式中，transformer模块包括编码器和解码器，这种情况transformer模块基于相关信息生成与n个第一特征块一一对应的n个第二特征块可以是：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为输入图像所属的图像增强任务的对应标识，通过该任务编码，可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像。
在本申请上述实施方式中，阐述了transformer模块具体是如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块，具备可实现性。
在第六方面的一种可能实现方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
在本申请上述实施方式中，阐述了切分模块如何对第一特征图进行切分的执行过程，具备可实现性。
在第六方面的一种可能实现方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
在本申请上述实施方式中，阐述了切分模块切分得到的n个切分块的尺寸可以相同，也可以不同，可根据需求预设，具备选择性。
本申请实施例第七方面提供一种训练设备，该训练设备具有实现上述第三/四方面或第三/四方面任意一种可能实现方式的方法的功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。
本申请实施例第八方面提供一种执行设备，该训练设备具有实现上述第五/六方面或第五/六方面任意一种可能实现方式的方法的功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。
本申请实施例第九方面提供一种训练设备，可以包括存储器、处理器以及总线系统，其中，存储器用于存储程序，处理器用于调用该存储器中存储的程序以执行本申请实施例第三/四方面或第三/四方面任意一种可能实现方式的方法。
本申请实施例第十方面提供一种执行设备，可以包括存储器、处理器以及总线系统，其中，存储器用于存储程序，处理器用于调用该存储器中存储的程序以执行本申请实施例第五/六方面或第五/六方面任意一种可能实现方式的方法。
本申请第十一方面提供一种计算机可读存储介质，该计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机可以执行上述第三/四方面或第三/四方面任意一种可能实现方式的方法，或，使得计算机可以执行上述第五/六方面或第五/六方面任意一种可能实现方式的方法。
本申请实施例第十二方面提供了一种计算机程序，当其在计算机上运行时，使得计算机可以执行上述第三/四方面或第三/四方面任意一种可能实现方式的方法，或，使得计算机可以执行上述第五/六方面或第五/六方面任意一种可能实现方式的方法。
本申请实施例第十三方面提供了一种芯片，该芯片包括至少一个处理器和至少一个接口电路，该接口电路和该处理器耦合，至少一个接口电路用于执行收发功能，并将指令发送给至少一个处理器，至少一个处理器用于运行计算机程序或指令，其具有实现如上述第三/四方面或第三/四方面任意一种可能实现方式的方法的功能，或，其具有实现如上述第五/六方面或第五/六方面任意一种可能实现方式的方法的功能，该功能可以通过硬件实现，也可以通过软件实现，还可以通过硬件和软件组合实现，该硬件或软件包括一个或多个与上述功能相对应的模块。
附图说明
图1为基于CNN对底层视觉任务进行处理的一个示意图；
图2为transformer模块的标准结构的一个示意图；
图3为本申请实施例提供的人工智能主体框架的一种结构示意图；
图4为本申请实施例提供的模型的结构的一个示意图；
图5为本申请实施例提供的模型用于对输入图像进行图像增强处理的一个示意图；
图6为本申请实施例提供的transformer编码器的一个示意图；
图7为本申请实施例提供的transformer解码器的一个示意图；
图8为本申请实施例提供的模型的结构的另一示意图；
图9为本申请实施例提供的图像增强系统的一种系统架构图；
图10为本申请实施例提供的模型的训练方法的一种流程示意图；
图11为本申请实施例提供的模型的训练方法的另一种流程示意图；
图12为本申请实施例提供的图像增强方法的一种流程示意图；
图13为本申请实施例提供的图像增强方法的另一种流程示意图；
图14为本申请实施例提供的应用场景的一个示意图；
图15为本申请实施例提供的训练设备的一个示意图；
图16为本申请实施例提供的执行设备的一个示意图；
图17为本申请实施例提供的训练设备的另一示意图；
图18为本申请实施例提供的执行设备的另一示意图；
图19为本申请实施例提供的芯片的一种结构示意图。
具体实施方式
本申请实施例提供了一种模型结构、模型训练方法、图像增强方法及设备，将用于处理自然语言任务的transformer模块结合不同的神经网络结构得到一种新的模型结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模 型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
本申请实施例涉及了许多关于神经网络、模型等相关知识，为了更好地理解本申请实施例的方案，下面先对本申请实施例可能涉及的相关术语和概念进行介绍。应理解的是，相关的概念解释可能会因为本申请实施例的具体情况有所限制，但并不代表本申请仅能局限于该具体情况，在不同实施例的具体情况可能也会存在差异，具体此处不做限定。
(1)神经网络
神经网是一种模型，神经网络可以是由神经单元组成的，具体可以理解为具有输入层、隐含层、输出层的神经网络，一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。其中，具有很多层隐含层的神经网络则称为深度神经网络(deep neural network，DNN)。神经网络中的每一层的工作可以用数学表达式
Figure PCTCN2021131704-appb-000001
来描述，从物理层面，神经网络中的每一层的工作可以理解为通过五种对输入空间(输入向量的集合)的操作，完成输入空间到输出空间的变换(即矩阵的行空间到列空间)，这五种操作包括：1、升维/降维；2、放大/缩小；3、旋转；4、平移；5、“弯曲”。其中1、2、3的操作由
Figure PCTCN2021131704-appb-000002
完成，4的操作由“+b”完成，5的操作则由“a()”来实现。这里之所以用“空间”二字来表述是因为被分类的对象并不是单个事物，而是一类事物，空间是指这类事物所有个体的集合，其中，W是神经网络各层的权重矩阵，该矩阵中的每一个值表示该层的一个神经元的权重值。该矩阵W决定着上文所述的输入空间到输出空间的空间变换，即神经网络每一层的W控制着如何变换空间。训练神经网络的目的，也就是最终得到训练好的神经网络的所有层的权重矩阵。因此，神经网络的训练过程本质上就是学习控制空间变换的方式，更具体的就是学习权重矩阵。
(2)模型
在本申请实施例中，用于处理图像增强任务的模型，本质上都是神经网络或模型的一部分结构为神经网络。模型的应用一般包括训练和推理两个阶段，训练阶段用于根据训练集对模型进行训练，以得到训练后的模型；推理阶段用于将训练后的模型对真实的无标签实例(即真实待处理的目标图像)进行图像增强处理，而图像增强处理后得到的增强图像的质量是衡量一个模型训练的好坏的重要指标之一。
(3)卷积神经网络(convolutional neural networks，CNN)
CNN是一种带有卷积结构的神经网络。CNN包含了一个由卷积层和子采样层构成的特征抽取器。该特征抽取器可以看作是滤波器，卷积过程可以看作是使用一个可训练的滤波器与一个输入的图像或者卷积特征平面(feature map)做卷积。卷积层是指CNN中对输入信号进行卷积处理的神经元层。在CNN的卷积层中，一个神经元可以只与部分邻层神经 元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。这其中隐含的原理是：图像的某一部分的统计信息与其他部分是一样的。即意味着在某一部分学习的图像信息也能用在另一部分上。所以对于图像上的所有位置，都能使用同样的学习得到的图像信息。在同一卷积层中，可以使用多个卷积核来提取不同的图像信息，一般地，卷积核数量越多，卷积操作反映的图像信息越丰富。
卷积核可以以随机大小的矩阵的形式初始化，在CNN的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)损失函数(loss function)
在训练神经网络的过程中，因为希望神经网络的输出尽可能的接近真正想要预测的值，可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重矩阵(当然，在第一次更新之前通常会有初始化的过程，即为神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重矩阵让它预测低一些，不断的调整，直到神经网络能够预测出真正想要的目标值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么神经网络的训练就变成了尽可能缩小这个loss的过程。
(5)反向传播算法
在神经网络的训练过程中，可以采用误差反向传播(back propagation，BP)算法修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经网络模型中的参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如权重矩阵。
(6)自注意力模块和多头自注意力模块
自注意力模块是神经网络的一种结构，其特点是计算输入模块中的每个单位(自注意力模块一开始是用在自然语言处理中，此时每个单位指的是每个单词)之间的相关度，并按照相关度在输入单位之间抓取信息。
具体来说，对于一个输入单位，自注意力模块首先将其转换为3个向量
Figure PCTCN2021131704-appb-000003
之后再将这3个向量分别乘以3个权重矩阵得到3个新的向量q、k、v，这3个不同的权重矩阵可记为Q、K、V。对于一个输入单位i，计算该输入单位i与其他单位j之间的相关性可通过公式s ij＝q i·k j得到，之后，对该相关性s ij采取归一化操作，即首先除以
Figure PCTCN2021131704-appb-000004
其中d k是向量k的维度，然后再对除以
Figure PCTCN2021131704-appb-000005
后的相关性s ij执行softmax操作，得到操作后的相关性
Figure PCTCN2021131704-appb-000006
利用相关性s ij′，对每个输入单位的向量v进行点乘，相加即可得到对该输入单位的输出结果，计算公式为z i＝∑ js ij′·v j。z i为输入单位i的输出。类似地，针对其余 输入单位也进行如此类似操作。
在实际使用中，一般使用多头自注意力模块，即对于输入单位，首先将其切分成h块，分别输入h个上述自注意力模块中，得到h个输出z，再将z按照切分方式重新拼起来，经过一层全连接网络后，得到最终输出。上述过程可记为MSA
Figure PCTCN2021131704-appb-000007
(7)transformer模块
Transformer模块也可以称为transformer模型、transformer结构等，是一种基于自注意力模块的多层神经网络。目前主要是用于处理自然语言任务，transformer模块主要由层叠的多头自注意力模块(也可称为MSA模块)与前馈神经网络(feed forward neural networks，FFN)组成。transformer模块可进一步分成编码器与解码器(也可称为编码模块和解码模块)，其构成大致相似，也有所不同。
一个标准的transformer模块的组成结构如图2所示，其中，左边为编码器，右边为解码器，每个编码器可包括任意数量的编码子模块，每个编码子模块包括一个多头自注意力模块和一个前馈神经网络；类似地，每个解码器可包括任意数量的解码子模块，每个解码子模块包括两个多头自注意力模块和一个前馈神经网络。编码子模块的数量与解码子模块的数量可以不相同。目前transformer模块是用于处理自然语言任务，无法直接应用在计算机视觉任务中。也就是说，transformer模块的编码器和解码器的输入均是单词的编码。
下面结合附图，对本申请的实施例进行描述。本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
首先，对人工智能系统总体工作流程进行描述，请参见图3，图3示出的为人工智能主体框架的一种结构示意图，下面从“智能信息链”(水平轴)和“IT价值链”(垂直轴)两个维度对上述人工智能主题框架进行阐述。其中，“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到系统的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施
基础设施为人工智能系统提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。通过传感器与外部沟通；计算能力由智能芯片(CPU、NPU、GPU、ASIC、FPGA等硬件加速芯片)提供；基础平台包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。举例来说，传感器和外部沟通获取数据，这些数据提供给基础平台提供的分布式计算系统中的智能芯片进行计算。
(2)数据
基础设施的上一层的数据用于表示人工智能领域的数据来源。数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有系统的业务数据以及力、位移、液位、温度、湿度等感知数据。
(3)数据处理
数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能系统中，模拟人类的智能推理方式，依据推理控制策略，利用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用系统，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用
智能产品及行业应用指人工智能系统在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能终端、智能制造、智能交通、智能家居、智能医疗、智能摄像头、自动驾驶、智慧城市等。
本申请实施例可以应用在模型的网络结构优化设计上，而通过本申请优化过结构的模型具体可以应用在人工智能领域的各个细分领域中，如，计算机视觉领域中的图像处理领域、语义分析领域等等。
首先介绍本申请实施提供的模型的结构，在本申请实施例中，基于提供的模型是可处理多种图像增强任务还是处理单一图像增强任务可具有两种不同的模型结构，下面分别进行描述：
一、模型的结构中包括的第一神经网络层和第二神经网络层均为m个，m≥2。
请参阅图4，图4为本申请实施例提供的模型的结构的一个示意图，该模型400具体可以包括：选择模块401、m个第一神经网络层402、切分模块403、transformer模块404、重组模块405以及m个第二神经网络层406，每个第一神经网络层唯一对应一个第二神经网络层，每个第一神经网络层也可以称为头模块或头结构，每个第二神经网络层也可以称为尾模块或尾结构。
需要注意的是，在本申请实施例中，不同的第一神经网络层对应不同的图像增强任务，也就是每个图像增强任务均有一个与之对应的第一神经网络层，一个第一神经网络层为某一种特定类型的图像增强任务处理相应的输入图像，例如，2倍超分辨重建任务、3倍超分辨重建任务、去噪任务等，均有与之各自对应的一个第一神经网络层。
还需要注意的是，每个第一神经网络层的大小、深度、参数量等都可自行设置，只需要能够运行即可。此外，第一神经网络层的数量m也可根据用户需求设定，这取决于该模型400是用于处理哪些类型的图像增强任务，例如，假设希望该模型400可同时用于处理去噪、去雨、2倍超分辨重建这3种类型的图像增强任务，那么m＝3，且这3种类型的图像增强任务分别对应一个第一神经网络层和一个第二神经网络层；假设希望该模型400可同时用于处理去噪、去模糊、去雨、2倍超分辨重建、4倍超分辨重建这5种类型的图像增强任务，那么m＝5，且这5种类型的图像增强任务分别对应一个第一神经网络层和一个第二神经网络层，类似地，可基于用户的实际使用需求进行设定，此处不予赘述。
该模型400中的选择模块401用于获取输入图像，并确定与该输入图像对应的第一目标神经网络层4021，该第一目标神经网络层4021为模型400中m个第一神经网络层402中的一个。也就是说，选择模块401接收到输入图像后，会判断该输入图像应该由哪个第一神经网络层去做特征提取操作，具体地，选择模块401会先确定该输入图像属于哪一种类型的图像增强任务，再将该输入图像输入到对应该任务的第一神经网络层去。该输入图像所属的图像增强任务可称为第一图像增强任务，假设该第一图像增强任务对应的是第一目标神经网络层4021，那么选择模块401就可确定将接收到的该输入图像输入到第一目标神经网络层4021中。
需要说明的是，在本申请的一些实施方式中，由于在训练阶段和推理阶段，输入图像是不一样，那么选择模块401针对不同阶段的输入图像确定该输入图像属于哪一类图像增强任务的方式也略有不同，下面分别进行阐述：
a、训练阶段时，输入图像为训练集中的训练样本。
在本申请的一些实施方式中，当该模型400是处于模型的训练阶段，那么输入图像就为训练集中的训练样本，此时每个训练样本都会有对应的标签指示该训练样本属于哪一类图像增强任务，该标签就用于指示该训练样本应该由哪个第一神经网络层去提取特征。那么该模型400的选择模块401就可根据该训练样本的标签以确定该训练样本属于第一图像增强任务。
b、推理阶段时，输入图像为待处理的目标图像。
在本申请的一些实施方式中，当该模型400是处于模型的推理阶段，那么输入图像就为真实的待处理的目标图像，在这个过程中，选择模块401除了会接收到该输入图像，还会接收到部署该模型400的设备发出的指令，该指令就是用于指示该目标图像是属于哪一类图像增强任务，也就是说，在推理阶段，该模型400的选择模块401是根据接收到的指令以确定该目标图像是属于第一图像增强任务。
该模型400的选择模块401根据输入图像确定出与该输入图像对应的第一目标神经网络层4021后，会将该输入图像输入至该第一目标神经网络层4021，该第一目标神经网络层4021将会对该输入图像进行特征提取，得到特征图(可称为第一特征图)。得到的第一特征图会进一步输入至切分模块403，由该切分模块403对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。
需要说明的是，在本申请的一些实施方式中，切分模块403对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
还需要说明的是，在本申请的一些实施方式中，切分模块403对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的 尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
切分模块403得到n个第一特征块后，将这n个第一特征块进一步输入到transformer模块404中进行处理。
transformer模块404基于这n个第一特征块，生成相关信息，该相关信息用于指示这n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块404根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。这里需要注意的是，transformer模块404输入的n个第一特征块的维度和输出的n个第二特征块的维度保持一致。
需要说明的是，在本申请的一些实施方式中，以transformer模块404包括至少一个编码器和至少一个解码器为例，对transformer模块404如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块进行说明：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。
这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为第一图像增强任务的对应标识，也可以认为是第一目标神经网络层的对应标识，每个图像增强任务都对应有一个任务编码，由于每个图像增强任务对应的输入图像会输入对应的第一神经网络层，因此，通过该任务编码，不仅可以知道transformer模块404接收到的n个第一特征块是来自于什么图像增强任务的输入图像，还可以知道这n个第一特征块是由哪个第一神经网络层进行的特征提取操作。
还需要说明的是，在本申请的一些实施方式中，该第一任务编码可以是编码器向解码器发送的，然后该第一任务编码再作为输入作用于该解码器，该第一任务编码也可以是在第一目标神经网络层被触发接收到输入图像时，通过部署该模型400的设备发送的指令接收到该第一任务编码，然后该第一任务编码再作为输入作用于该解码器，具体本申请对第一任务编码的获取方式不做限定。此外，还需要说明的是，每个任务编码可以是根据图像增强任务自行标记的，也可以是模型自己学习得到的，具体此处不做限定。
transformer模块404基于相关信息由n个第一特征块得到n个第二特征块后，将通过重组模块405对这n个第二特征块按照空间相对位置进行拼接重组，从而得到与输入的第一特征图维度一致的第二特征图，该重组模块405的操作是切分模块403的逆操作，此处不予赘述。这里需要注意的是，第二特征图的尺寸与第一特征图的尺寸要保持一致。
重组模块405将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至与第一目标神经网络层4021唯一对应的第二目标神经网络层4061中，该第二目标神经网络层4061属于m个第二神经网络层406中的一个。然后该第二目标神经网络层4061对接收到的第二特征图进行解码，从而得到输出图像，该输出图像就是经过了模型400处理后该输入图像的增强图像。
需要注意的是，与第一神经网络层类似，每个第二神经网络层的大小、深度、参数量等也都可自行设置，只需要能够运行即可。此外，第二神经网络层的数量m需与第一神经网络层的数量保持一致。
还需要说明的是，在本申请的一些实施方式中，transformer模块404的结构除了可以是如图2中所示的包括编码器和解码器的标准结构外，还可以对其结构进行微调，以得到调整后的transformer模块404的结构，例如，调整后的transformer模块404的结构可以是只包括编码器，也可以是只包括解码器。若transformer模块404的结构只包括编码器，那么transformer模块404至少应该包括2个编码器，其中至少一个编码器用于承担原本由解码器承担的操作；若transformer模块404的结构只包括解码器，那么transformer模块404至少应该包括2个解码器，其中至少一个解码器用于承担原本由编码器承担的操作。
需要说明的是，在本申请实施例中，由于m个第一神经网络层402和m个第二神经网络层406分别位于模型的头部和尾部，因此，在本申请的一些实施方式中，m个第一神经网络层402也可以简称为多头结构，m个第二神经网络层402也可以简称为多尾结构，每个第一神经网络层根据其对应的图像增强任务可称为“XX头”，如图5所示，模型包括有4个第一神经网络层，且这4个第一神经网络层各自对应的图像增强任务分别为：去噪、去雨、2倍超分辨率重建、4倍超分辨率重建，那么4个第一神经网络层可分别简称为“去噪头”、“去雨头”、“2倍超分头”和“4倍超分头”，类似地，这4个第一神经网络层也各自唯一对应一个第二神经网络层，共4个第二神经网络层，这4个第二神经网络层也可分别简称为“去噪尾”、“去雨尾”、“2倍超分尾”和“4倍超分尾”，类似地，若有其他的图像增强任务，可按照上述方式得到对应的第一神经网络层的简称，具体此处不予赘述。
为便于理解上述模型400的工作流程，下面以transformer模型为图2中的标准结构为例，对上述模型400对输入图像的具体处理过程进行介绍，请参阅图5，图5为本申请实施例提供的模型400用于对输入图像进行图像增强处理的一个示意图：
为了适应不同的图像增强任务，本申请使用多头结构分别处理每个任务，每个任务都有与之对应的头模块。假设模型的初始输入图像为
Figure PCTCN2021131704-appb-000008
其中，C是输入图像的通道数，例如，当输入黑白图像时C＝1，当输入彩色图象时C可以为3，是指RGB的三原色，H×W分别为初始输入图像x的尺寸(即高度和宽度)，初始输入图像x基于其所属的图形增强任务输入多头结构中的目标头结构(假设输入的是去噪头)，该目标头结构生成具有C个通道且尺寸与初始输入图像x的尺寸相同的特征图
Figure PCTCN2021131704-appb-000009
(即第一特征图)，可以将其表示为f H＝H i(x)，其中，H i(i＝{1,…,N t})表示第i个图像增强任务对应的头结构，而N t表示图像增强任务的种类数(即有几种图像增强任务)。
之后切分模块(图5中未示意出)将特征图
Figure PCTCN2021131704-appb-000010
进行切分，每个特征块都可视 为一个“单词”的编码，具体而言，将特征
Figure PCTCN2021131704-appb-000011
切分并整形为一系列特征块
Figure PCTCN2021131704-appb-000012
其中
Figure PCTCN2021131704-appb-000013
N表示的是分块的数量(即输入序列的长度)，这里需要注意的是，N的最大值是由transformer模型的具体结构决定的，切分模块切得的特征块f pi的数量不能超过N的最大值，此外，特征块f pi的尺寸也可以通过预设P的大小来决定。在本申请实施例中，每个特征块f pi的尺寸大小都是一致的，实际上在本申请的一些实施方式中，特征块f pi的尺寸大小也可以不一致，具体此处不做限定。此外，为了维护每个特征块的位置信息，本申请为每个特征块f pi都添加了可学习的位置编码
Figure PCTCN2021131704-appb-000014
(在一些实施方式中，位置编码也可以自行设定)，把每个特征块f pi与对应位置的位置编码相加，得到E pi+f pi，之后再将每个E pi+f pi输入到transformer编码器中。
在本申请实施例中，transformer模块中的transformer编码器的结构可如图6所示，图6中(a)子示意图示意的是transformer编码器中的一个编码子模块，该编码子模块具有一个多头自注意力模块(可记为MSA模块)和一个前馈神经网络(可记为FFN)，而transformer编码器中可以具备多个这样的编码子模块(可根据需要自行设置数量)，如图6中(b)子示意图示意的就是一个transformer编码器包括多个编码子模块。
下面基于图6所示的transformer编码器中的各个编码子模块，对transformer编码器的处理流程进行介绍。
transformer编码器的第一个编码子模块的输入可以表示为公式(1)所述的形式：
y 0＝[E p1+f p1,E p2+f p2,…,E pN+f pN]   (1)
其中，y 0表示第一个编码子模块的输入，
Figure PCTCN2021131704-appb-000015
是切分模块切分后的得到的特征块，
Figure PCTCN2021131704-appb-000016
是对应特征块f pi的位置编码。编码子模块处理每个特征块f pi得到的输出
Figure PCTCN2021131704-appb-000017
与输入的特征块f pi的大小相同。一个编码子模块的计算公式(2)如下：
q i＝k i＝v i＝LN(y i-1)   (2)
其中，LN表示层归一化(归一化操作的一种)，y i-1为当前编码子模块的输入，对于第一个编码子模块，其输入为上述y 0，对于之后的编码子模块，第i个编码子模块的输入就是第i-1个编码子模块的输出y i-1，q i,k i,v i是将输入y i-1转化为三个向量，并作为当前编码子模块中MSA模块(即多头自注意力模块)的输入，当前编码子模块的MSA模块的输出如公式(3)所示：
y′ i＝MSA(q i,k i,v i)+y i-1   (3)
其中，y′ i是当前编码子模块中MSA模块的输出，y′ i之后作为当前编码子模块的FFN(即前馈神经网络)的输入部分，如下述公式(4)所示：
y i＝FFN(LN(y′ i))+y′ i，i＝1,…,m   (4)
其中，y i是第i个编码子模块的输出，上式中m表示transformer编码器中的层数(即共有m个编码子模块)。transformer编码器最后一个编码子模块的输出则为y m(在解码器中记作z 0)，如下述公式(5)所示：
Figure PCTCN2021131704-appb-000018
类似地，在本申请实施例中，transformer模块中的transformer解码器的结构可如图7所示，transformer解码器与transformer编码器具有类似的体系，图7中(a)子示意图示意的是transformer解码器中的一个解码子模块，该解码子模块具有2个多头自注意力模块(可分别记为MSA1模块和MSA2模块)和一个前馈神经网络(可记为FFN)，而transformer解码器中可以具备多个这样的解码子模块(可根据需要自行设置数量)，如图7中(b)子示意图示意的就是一个transformer解码器包括多个解码子模块。
下面基于图7所示的transformer解码器中的各个解码子模块，对transformer解码器的处理流程进行介绍。
在本申请实施例中，与transformer模块用于处理自然语言任务不同的地方在于：本申请将对特定图像增强任务的任务编码作为transformer解码器的其中一个输入。这些任务编码
Figure PCTCN2021131704-appb-000019
能够对不同图像增强任务的特征进行编码，需要注意的是，任务编码可以预先设定好，也可以学习得到，具体此处不做限定。transformer解码器的第一个解码子模块的输入也就是transformer编码器的最后一个编码子模块的输出y m，可以表示为公式(6)所述的形式：
Figure PCTCN2021131704-appb-000020
对于解码子模块的MSA1模块，其输入的三个变量q i,k i,v i可如公式(7)所示：
q i＝k i＝LN(z i-1)+E t,v i＝LN(z i-1)   (7)
其中，E t是任务编码，被用来计算q i、k i向量，而v i则与此无关，z i-1为当前解码子模块的输入，对于第一个解码子模块，其输入为上述的z 0，对于之后的解码子模块，第i个解码子模块的输入就是第i-1个解码子模块的输出z i-1，这三个向量q i,k i,v i之后被送入解码子模块的MSA1模块中，根据下述公式(8)得到MSA1模块的输出z′ i：
z′ i＝MSA(q i,k i,v i)+z i-1   (8)
对于第一个解码子模块，其输入为编码模块的输出z 0，对于第i(i≥2)个解码子模块，其输入是上层(即第i-1个)解码子模块的输出z i-1。对于解码子模块中的MSA2模块，其输入的三个向量q′ i、k′ i、v′ i的计算方式如公式(9)所示：
q′ i＝LN(z′ i)+E t,k′ i＝v′ i＝LN(z 0)   (9)
其中，计算q′ i向量是根据MSA1模块的输出z′ i计算得到，而计算k′ i、v′ i向量则是利用transformer编码器的输出z 0，由此得到MSA2模块的输入q′ i,k′ i,v′ i，从而MSA2模块的输出z″ i可通过如下公式(10)计算得到：
z″ i＝MSA(q′ i,k′ i,v′ i)+z′ i  (10)
之后，MSA2模块的输出z″ i作为FFN的输入，通过如下公式(11)得到第i个解码子 模块的输出z i：
z i＝FFN(LN(z″ i))+z″ i,i＝1,…,n   (11)
对于共有n个解码子模块的transformer解码器来说，其最终输出z n可记为如下公式(12)的表达形式：
Figure PCTCN2021131704-appb-000021
其中，
Figure PCTCN2021131704-appb-000022
表示解码器中每个特征块的输出。然后将已解码的N个大小为P 2×C的特征块通过重组模块重新整形为大小为C×H×W的特征f D(即第二特征图)，最后，重组模块会将该特征f D输入至与处理输入图像的头结构对应的尾结构，例如，假设对输入图像进行特征提取的是“去噪头”，那么该特征f D就会被输入至“去噪尾”，由该尾结构对特征f D进行解码，得到输出图像，该输出图像就是经过了模型处理后该输入图像的增强图像。尾结构的计算公式(13)如下所示：
f T＝T i(f D)   (13)
其中，T i(i＝{1,…,N t})表示第i个图像增强任务的尾结构，N t表示图像增强任务的种类数。输出f T是大小为3×H′×W′的结果图像。H′和W′是输出图像的大小，由具体的图像增强任务决定。例如，对于2倍超分辨率重建任务，H′＝2H,W＝2W。
在本申请上述实施方式中，将用于处理自然语言任务的transformer模块结合不同的神经网络结构得到一种新的模型结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
二、模型的结构中包括的第一神经网络层和第二神经网络层均为1个。
请参阅图8所示，图8为本申请实施例提供的模型的结构的另一示意图，该模型800具体可以包括：第一神经网络层801、切分模块802、transformer模块803、重组模块804以及第二神经网络层805，其中，第一神经网络层801也可以称为头模块或头结构，第二神经网络层805也可以称为尾模块或尾结构。在本申请实施例中，由于第一神经网络层801和第二神经网络层805各自只有一个，因此在该模型800中，就不存在选择模块。
需要注意的是，在本申请实施例中，第一神经网络层801只对应一种类型的图像增强任务，该第一神经网络层801为某一种特定类型的图像增强任务处理相应的输入图像。
还需要注意的是，第一神经网络层801和第二神经网络层805的大小、深度、参数量等都可自行设置，只需要能够运行即可。
在本申请实施例中，第一神经网络层801，用于对输入图像进行特征提取，得到特征图(可称为第一特征图)，之后该第一特征图输入至切分模块802，由该切分模块802对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。类似地，在模型800中，切分模块802对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
还需要说明的是，在本申请的一些实施方式中，切分模块802对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。在得到的n个切分块的尺寸均相同的情况下，后续transformer模块中的可以通过一个自注意力模块对这n个切分块进行处理，减少了计算量；在得到的n个切分块的尺寸不同的情况下，后续transformer模块就需要通过多个自注意力模块对这n个切分块进行处理，有几种不同的尺寸(如，x种不同的尺寸)，transformer模块中就至少需要配置对应的x个自注意力模块，但这种切分尺寸不同的好处在于：针对需要更多细节特征的区域(如，天空中飞行的鸟)，切分模块可以切成更多数量的小尺寸切分块，而针对不需要太多细节特征的区域(如，天空)，则切分模块可以切成少数几个大尺寸切分块，从而具备灵活性。
切分模块802得到n个第一特征块后，将这n个第一特征块进一步输入到transformer模块803中进行处理。
transformer模块803基于这n个第一特征块，生成相关信息，该相关信息用于指示这n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块803根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。这里需要注意的是，transformer模块803输入的n个第一特征块的维度和输出的n个第二特征块的维度保持一致。
需要说明的是，在本申请的一些实施方式中，以transformer模块803包括至少一个编码器和至少一个解码器为例，对transformer模块803如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块进行说明：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。
这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为输入图像所属的图像增强任务的对应标识，通过该任务编码，可以知道transformer模块803接收到的n个第一特征块是来自于什么图像增强任务的输入图像。
还需要说明的是，在本申请的一些实施方式中，该第一任务编码可以是编码器向解码 器发送的，然后该第一任务编码再作为输入作用于该解码器，该第一任务编码也可以是在第一目标神经网络层被触发接收到输入图像时，通过部署该模型800的设备发送的指令接收到该第一任务编码，然后该第一任务编码再作为输入作用于该解码器，具体本申请对第一任务编码的获取方式不做限定。此外，还需要说明的是，每个任务编码可以是根据图像增强任务自行标记的，也可以是模型自己学习得到的，具体此处不做限定。
transformer模块803基于相关信息由n个第一特征块得到n个第二特征块后，将通过重组模块804对这n个第二特征块按照空间先对位置进行拼接重组，从而得到与输入的第一特征图维度一致的第二特征图，该重组模块804的操作是切分模块802的逆操作，此处不予赘述。这里需要注意的是，第二特征图的尺寸与第一特征图的尺寸要保持一致。
重组模块804将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至第二神经网络层805中，由该第二神经网络层805对接收到的第二特征图进行解码，得到输出图像，该输出图像就是经过了模型800处理后该输入图像的增强图像。
需要说明的是，在本申请实施例中，在模型800的训练阶段，输入图像指的是训练集中的训练样本；在模型800的推理阶段，输入图像指的则是真实的待处理的目标图像。
还需要说明的是，在本申请实施例中，模型800除了不具备模型400的选择模块401之外，区别在于模型800的第一神经网络层和第二神经网络层均只有一个，模型800中各个模块的处理过程与上述模型400类型，具体地，可参阅图4中模型400对输入图像进行图像增强处理的对应实施方式，具体此处不予赘述。
在本申请上述实施方式中，将用于处理自然语言任务的transformer模块结合不同的神经网络结构得到一种新的模型结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备一个第一神经网络层和一个第二神经网络层，用于处理一个特定的图像增强任务，相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
需要说明的是，图4对应所述的模型400以及图8对应所述的模型800需要先进行训练，训练后才能部署在目标设备上对目标图像执行图像增强任务，接下来对图像增强系统的架构进行介绍，请参阅图9，图9为本申请实施例提供的图像增强系统的一种系统架构图，在图9中，图像增强系统200包括执行设备210、训练设备220、数据库230、客户设备240、数据存储系统250和数据采集设备260，执行设备210中包括计算模块211。其中，数据采集设备260用于获取用户需要的开源的大规模数据集(即训练集)，并将训练集存入数据库230中，训练设备220基于数据库230中的维护的训练集对本申请提供的模型201进行训练，训练得到的训练后的模型201再在执行设备210(执行设备也可称为目标设备)上进行运用。执行设备210可以调用数据存储系统250中的数据、代码等，也可以将数据、指令等存入数据存储系统250中。数据存储系统250可以置于执行设备210中，也可以为数据存储系统250相对执行设备210是外部存储器。
经由训练设备220训练得到的训练后的模型201可以应用于不同的系统或设备(即执 行设备210)中，具体可以是边缘设备或端侧设备，例如，手机、平板、笔记本电脑、监督系统(如，摄像头)等等。在图9中，执行设备210配置有I/O接口212，与外部设备进行数据交互，“用户”可以通过客户设备240向I/O接口212输入数据。如，客户设备240可以是监控系统的摄像设备，通过该摄像设备拍摄的图像作为输入数据输入至执行设备210的计算模块211，由计算模块211对输入的图像进行图像增强处理得到增强后的图像，得到的增强图像可以再输出至摄像设备进行显示或存储，或得到的增强图像可以直接在执行设备210的显示界面(若有)进行显示或存储；此外，在本申请的一些实施方式中，客户设备240也可以集成在执行设备210中，如，当执行设备210为手机时，则可以直接通过该手机获取到待处理的目标图像(如，可以通过该手机的摄像头拍摄到的图像)或者接收其他设备(如，另一个手机)发送的目标图像，再由该手机内的计算模块211对该目标图像进行图像增强后得出增强图像，并直接将该增强图像呈现在手机的显示界面或存储在该手机内。此处对执行设备210与客户设备240的产品形态不做限定。
值得注意的，图9仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图9中，数据存储系统250相对执行设备210是外部存储器，在其它情况下，也可以将数据存储系统250置于执行设备210中；在图9中，客户设备240相对执行设备210是外部设备，在其他情况下，客户设备240也可以集成在执行设备210中。
需要说明的是，本申请实施例所述的模型201的训练可以是在云侧实现，例如，可以由云侧的训练设备220(该训练设备220可设置在一个或多个服务器或者虚拟机上)获取训练集，并根据训练集内的多组训练样本对模型201进行训练，得到训练后的模型201，之后，该训练后的模型201再发送给执行设备210进行应用，例如，发送给执行设备210进行图像超分辨重建、去噪、去雨等图像增强任务，示例性地，图9对应的系统架构中所述，就是由训练设备220对模型201进行训练，训练后的模型201再发送给执行设备210进行使用；上述实施例所述的模型201的训练也可以是在终端侧实现，即训练设备220可以是位于终端侧，例如，可以由终端设备(如，手机、智能手表等)、轮式移动设备(如，自动驾驶车辆、辅助驾驶车辆等)等获取训练集，并根据训练集内的多组训练样本对模型201进行训练，得到训练后的模型201，该训练后的模型201就可以直接在该终端设备使用，也可以由该终端设备发送给其他的设备进行使用。具体本申请实施例对模型201在哪个设备(云侧或终端侧)上进行训练或应用不做限定。
还需注意的是，在图9对应的实施例中，模型201的模型结构可以是上述图4对应的模型400的结构，也可以是上述图8对应的模型800的结构，具体此处不做限定。
接下来分别从模型的训练阶段和模型的推理阶段，对本申请实施例提供的模型的训练方法和图像增强方法的具体实现流程进行描述。
A、训练阶段
本申请实施例中，训练阶段描述的是训练设备220如何利用数据库230中维护的训练集得到训练后的模型201的过程。由于在本申请实施例中，模型201即可以是图4对应的模型400的结构，也可以是图8对应的模型800的结构，模型的结构不同，模型的训练方 法略有不同，下面分别进行介绍。
(1)模型的结构为模型400对应的结构。
请参阅图10，图10为本申请实施例提供的模型的训练方法的一种流程示意图，具体可以包括如下步骤：
1001、训练设备获取训练样本，该训练样本为构建的训练集中任意一个退化图像，其中，该训练集中的每个退化图像由一个清晰图像经过图像退化处理得到。
训练设备首先从构建的训练集中获取训练样本，该训练样本可以是构建的该训练集中的任意一个退化图像，而每个退化图像又是经由一个清晰图像经过图像退化处理得到的。每个清晰图像的获取可以是用户从开源的大规模数据集中得到，如清晰图像可以是从ImageNet数据集中得到，由于图像增强任务可以有不同类型，如，去噪、去雨、超分辨率重建等，因此，可以根据不同的图像增强任务构建不同类型的训练集，在本申请实施例中，为了以监督的方式训练模型，可以使用不同的图像退化模型从非监督的清晰图像中合成了多种类型的退化图像，从而得到了对应各种不同图像增强任务的训练集。例如，对于超分辨率任务，对非监督数据集上的清晰图像进行下采样得到低分辨率的退化图像。
需要说明的是，在本申请实施例中，通过图像退化处理构建训练集的目的是为了获得大型训练集，这是由于在图像处理中有监督的数据量的通常不足(例如，用于超分辨率任务的DIV2K数据集上只有2000张图像)，因此本申请提出基于开源的大规模数据集(如，ImageNet数据集)，使用无监督的数据集来对模型进行训练。
为便于理解，下面举例进行示意：本申请可使用ImageNet数据集，该数据集由超过1M的高多样性彩色图像组成。训练图像被裁剪为具有3个通道的48×48的块进行训练，其中有超过1000万个块用于训练本申请提出的模型。然后，本申请生成具有6种退化类型的损坏图像：分别为2倍，3倍，4倍的双三次线性插值下采样图像，30、50噪声级高斯噪声和添加雨条纹。对于超分辨率重建任务，退化模型是采用f sr双三次插值，分别进行2倍，3倍，4倍的下采样来得到不同倍率超分任务所需要的图像；对于降噪，退化模型为f noise(I)＝I+η，其中η是高斯噪声，我们添加30、50噪声级高斯噪声来得到训练样本；对于去雨任务，退化模型为f rain(I)＝I+r，其中r是降雨条纹，即在纯净图像上加入降雨条纹来得到去雨任务的训练样本。
需要说明的是，在本申请的一些实施方式中，训练样本也可以是真实的低质量的带有标签的图像，该低质量图像作为训练样本时，也需对应存在一个高质量的清晰图像。具体此处对训练样本的类型不做限定。
还需要说明的是，在本申请的一些实施方式中，可以先利用经过图像退化处理的退化图像对模型进行预训练，然后再利用真实的低质量的带有标签的图像对模型再进行微调。具体来说，对于每个训练批次(每个批次对应一个图像增强任务类型)，本申请从N t个图像增强任务中随机选择一个任务进行训练，每个任务都采用其对应的第一目标神经网络层、第二目标神经网络层和第一任务编码进行预训练。在对模型进行预训练之后，可以再使用该任务的相应数据集对模型进行微调以应用于特定任务。在微调阶段，相应的第一目标神经网络层、第二目标神经网络层和模型中间的共享结构的参数会更新，而其他任务对应的 第一目标神经网络层和第二目标神经网络层则会被冻结。
1002、训练设备将训练样本输入模型中，由模型中的选择模块确定与训练样本对应的第一目标神经网络层，该第一目标神经网络层为该模型中m个第一神经网络层中的一个。
训练设备获取到训练样本后，会将该训练样本输入模型中，然后由模型中的选择模块确定与该训练样本对应的第一目标神经网络层。由于训练样本都会有对应的标签指示该训练样本属于哪一类图像增强任务，该标签就用于指示该训练样本应该由哪个第一神经网络层去提取特征。那么该模型的选择模块就可根据该训练样本的标签以确定该训练样本属于第一图像增强任务，并进一步确定与该第一图像增强任务对应的第一目标神经网络层。
在本申请实施例中，模型中的选择模块的执行过程可参阅上述图4对应的实施例中选择模块401，此处不予赘述。
1003、通过该第一目标神经网络层对该训练样本进行特征提取，得到第一特征图。
该第一目标神经网络层将会对该训练样本进行特征提取，得到特征图(可称为第一特征图)。
在本申请实施例中，模型中的第一目标神经网络层的执行过程可参阅上述图4对应的实施例中第一目标神经网络层4021，此处不予赘述。
1004、通过该模型中的切分模块对第一特征图进行切分，得到n个第一特征块。
得到的第一特征图会进一步输入至模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。
需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
还需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。
在本申请实施例中，模型中的切分模块的执行过程可参阅上述图4对应的实施例中切分模块403，此处不予赘述。
1005、由该模型中的transformer模块根据相关信息生成与n个第一特征块一一对应的n个第二特征块，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度。
模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。这里需要注意的是，transformer模块输入的n个第一特征块的维度和输出的n个第二特征块的维度保持一致。
需要说明的是，在本申请的一些实施方式中，以transformer模块包括至少一个编码器和至少一个解码器为例，对transformer模块如何基于相关信息，生成与n个第一特征块一 一对应的n个第二特征块进行说明：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为第一图像增强任务的对应标识，也可以认为是第一目标神经网络层的对应标识，每个图像增强任务都对应有一个任务编码，由于每个图像增强任务对应的输入图像会输入对应的第一神经网络层，因此，通过该任务编码，不仅可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像，还可以知道这n个第一特征块是由哪个第一神经网络层进行的特征提取操作。
在本申请实施例中，模型中的transformer模块的执行过程可参阅上述图4对应的实施例中transformer模块404，此处不予赘述。
1006、通过该模型中的重组模块对n个第二特征块进行拼接重组，得到第二特征图。
模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。
在本申请实施例中，模型中的重组模块的执行过程可参阅上述图4对应的实施例中重组模块405，此处不予赘述。
1007、通过第二目标神经网络层对第二特征图进行解码，得到训练样本的第一增强图像，该第二目标神经网络层与第一目标神经网络层对应，且第二目标神经网络层为该模型中m个第二神经网络层中的一个。
模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至与第一目标神经网络层唯一对应的第二目标神经网络层中，该第二目标神经网络层属于模型中m个第二神经网络层中的一个。然后该第二目标神经网络层对接收到的第二特征图进行解码，从而得到训练样本的增强图像(可称为第一增强图像)。
在本申请实施例中，模型中的第二目标神经网络层的执行过程可参阅上述图4对应的实施例中第二目标神经网络层4061，此处不予赘述。
1008、训练设备根据第一增强图像、清晰图像和损失函数对该模型进行训练，得到训练后的模型，该清晰图像与该训练样本对应。
训练设备得到经由模型输出的第一增强图像后，将根据该第一增强图像、清晰图像和损失函数对该模型进行训练，以得到训练后的模型。其中，该训练样本就是该清晰图像通过图像退化处理得到的，因此，可称为该清晰图像与该训练样本对应。
需要说明的是，在本申请实施例中，退化图像与对应的清晰图像之间的关系可如公式(14)所示：
I corrupted＝f(I clean)   (14)
其中，I clean表示清晰图像，I corrupted表示与该清晰图像对应的退化图像，f表示图像退化变换，在这样的合成的训练集上训练模型的损失函数
Figure PCTCN2021131704-appb-000023
可表示为公式(15)：
Figure PCTCN2021131704-appb-000024
其中，L 1表示L1损失函数，而
Figure PCTCN2021131704-appb-000025
表示任务i的退化图像，损失函数
Figure PCTCN2021131704-appb-000026
的训练目标为拉近该清晰图像与该第一增强图像之间的相似度。
还需要说明的是，在本申请的一些实施方式中，由于图像退化模型的多样性，我们无法为所有图像增强任务合成退化图像。因此，本申请引入了对比学习方法来学习未见任务的通用功能。具体来说，将清晰图像x j作为输入，transformer模型中解码器生成的输出修补特征表示为
Figure PCTCN2021131704-appb-000027
对比学习的目标是最小化来自同一图像的特征块的解码器输出编码之间的距离，同时最大化它们与不同图像之间的距离。对比学习的损失函数可如公式(16)所示：
Figure PCTCN2021131704-appb-000028
其中，
Figure PCTCN2021131704-appb-000029
表示余弦相似度。此外，为了使模型保持原始图像结构，在本申请的一些实施方式中，模型的损失函数
Figure PCTCN2021131704-appb-000030
可以由下述公式(17)所示：
Figure PCTCN2021131704-appb-000031
也就是本申请将对比损失与监督损失相结合，作为训练模型的最终损失函数
Figure PCTCN2021131704-appb-000032
还需要说明的是，在本申请的一些实施方式中，训练后的模型可部署在目标设备上，如，部署在边缘设备或端侧设备上，例如，手机、平板、笔记本电脑、监督系统(如，摄像头)等等。
在本申请上述实施例中，具体阐述了如何对本申请构建的一种模型进行训练，得到训练后的模型。该模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
(2)模型的结构为模型800对应的结构。
请参阅图11，图11为本申请实施例提供的模型的训练方法的另一种流程示意图，具体可以包括如下步骤：
1101、训练设备获取训练样本，该训练样本为构建的训练集中任意一个退化图像，其 中，该训练集中的每个退化图像由一个清晰图像经过图像退化处理得到。
本申请实施例中，步骤1101与上述步骤1001类似，此处不予赘述。
1102、训练设备将该训练样本输入模型中，由模型中的第一神经网络层对训练样本进行特征提取，得到第一特征图。
训练设备获取到训练样本后，会将该训练样本输入模型中，由模型中的第一神经网络层对训练样本进行特征提取，得到第一特征图。
在本申请实施例中，模型中的第一神经网络层的执行过程可参阅上述图8对应的实施例中第一神经网络层801，此处不予赘述。
1103、通过该模型中的切分模块对第一特征图进行切分，得到n个第一特征块。
得到的第一特征图会进一步输入至模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。
需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
还需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。
在本申请实施例中，模型中的切分模块的执行过程可参阅上述图8对应的实施例中切分模块802，此处不予赘述。
1104、由该模型中的transformer模块根据相关信息生成与n个第一特征块一一对应的n个第二特征块，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度。
模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。这里需要注意的是，transformer模块输入的n个第一特征块的维度和输出的n个第二特征块的维度保持一致。
需要说明的是，在本申请的一些实施方式中，以transformer模块包括至少一个编码器和至少一个解码器为例，对transformer模块如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块进行说明：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第 二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为输入图像所属的图像增强任务的对应标识，通过该任务编码，可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像。
在本申请实施例中，模型中的transformer模块的执行过程可参阅上述图8对应的实施例中transformer模块803，此处不予赘述。
1105、通过该模型中的重组模块对n个第二特征块进行拼接重组，得到第二特征图。
模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。
在本申请实施例中，模型中的重组模块的执行过程可参阅上述图8对应的实施例中重组模块804，此处不予赘述。
1106、通过该模型中的第二神经网络层对第二特征图进行解码，得到训练样本的第一增强图像。
模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至第二神经网络层中，然后该第二神经网络层对接收到的第二特征图进行解码，从而得到训练样本的增强图像(可称为第一增强图像)。
在本申请实施例中，模型中的第二神经网络层的执行过程可参阅上述图8对应的实施例中第二神经网络层805，此处不予赘述。
1107、训练设备根据第一增强图像、清晰图像和损失函数对该模型进行训练，得到训练后的模型，该清晰图像与该训练样本对应。
本申请实施例中，步骤1107与上述步骤1008类似，此处不予赘述。
需要说明的是，在本申请的一些实施方式中，训练后的模型可部署在目标设备上，如，部署在边缘设备或端侧设备上，例如，手机、平板、笔记本电脑、监督系统(如，摄像头)等等。
在本申请上述实施例中，具体阐述了如何对本申请构建的另一种模型进行训练，得到训练后的模型。该训练后的模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备一个第一神经网络层和一个第二神经网络层，用于处理一个特定的图像增强任务，相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
B、推理阶段
本申请实施例中，应用阶段描述的是执行设备210如何利用成熟的模型201对真实的待处理的目标图像进行对应的图像增强处理的过程，类似地，由于在本申请实施例中，经过训练阶段得到的训练后的模型201即可以是图4对应的模型400的结构，也可以是图8对应的模型800的结构，模型的结构不同，基于该训练后的模型201执行图像增强的方法 也略有不同，下面分别进行介绍。
(1)训练后的模型的结构为模型400对应的结构。
请参阅图12，图12为本申请实施例提供的图像增强方法的一种流程示意图，具体可以包括如下步骤：
1201、执行设备获取待处理的目标图像。
执行设备(即上述所述的目标设备)获取待处理的目标图像，如，由手机通过摄像头拍摄到的图像，由监控设备通过摄像头拍摄下的图像等。
1202、执行设备将该目标图像输入训练后的模型，由该训练后的模型中的选择模块确定与该目标图像对应的第一目标神经网络层，该第一目标神经网络层为训练后的模型中m个第一神经网络层中的一个。
该执行设备上部署有训练后的模型，执行设备获取到目标图像后，会将该目标图像输入训练后的模型，由该训练后的模型中的选择模块确定与该目标图像对应的第一目标神经网络层，该第一目标神经网络层为训练后的模型中m个第一神经网络层中的一个。
由于真实的待处理的目标图像不具有标签，训练后的模型感知不到该目标图像对应哪种类型的图像增强任务，这时，执行设备会额外向该训练后的模型发送一个指令，该指令用于指示该目标图像是属于哪一类图像增强任务，也就是说，在推理阶段，该训练后的模型的选择模块是根据接收到的指令以确定该目标图像是属于第一图像增强任务，并进一步确定与该第一图像增强任务对应的第一目标神经网络层。
在本申请实施例中，训练后的模型中的选择模块的执行过程可参阅上述图4对应的实施例中选择模块401，此处不予赘述。
1203、通过该第一目标神经网络层对目标图像进行特征提取，得到第一特征图。
该第一目标神经网络层将会对该目标图像进行特征提取，得到特征图(可称为第一特征图)。
在本申请实施例中，训练后的模型中的第一目标神经网络层的执行过程可参阅上述图4对应的实施例中第一目标神经网络层4021，此处不予赘述。
1204、通过该训练后的模型中的切分模块对第一特征图进行切分，得到n个第一特征块。
得到的第一特征图会进一步输入至该训练后的模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。
需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
还需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。
在本申请实施例中，训练后的模型中的切分模块的执行过程可参阅上述图4对应的实施例中切分模块403，此处不予赘述。
1205、由该训练后的模型中的transformer模块根据相关信息生成与n个第一特征块一 一对应的n个第二特征块，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度。
训练后的模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到该训练后的模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。这里需要注意的是，transformer模块输入的n个第一特征块的维度和输出的n个第二特征块的维度保持一致。
需要说明的是，在本申请的一些实施方式中，以transformer模块包括至少一个编码器和至少一个解码器为例，对transformer模块如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块进行说明：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为第一图像增强任务的对应标识，也可以认为是第一目标神经网络层的对应标识，每个图像增强任务都对应有一个任务编码，由于每个图像增强任务对应的输入图像会输入对应的第一神经网络层，因此，通过该任务编码，不仅可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像，还可以知道这n个第一特征块是由哪个第一神经网络层进行的特征提取操作。
在本申请实施例中，训练后的模型中的transformer模块的执行过程可参阅上述图4对应的实施例中transformer模块404，此处不予赘述。
1206、通过该训练后的模型中的重组模块对n个第二特征块进行拼接重组，得到第二特征图。
训练后的模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过该训练后的模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。
在本申请实施例中，训练后的模型中的重组模块的执行过程可参阅上述图4对应的实施例中重组模块405，此处不予赘述。
1207、通过第二目标神经网络层对第二特征图进行解码，得到目标图像的第二增强图像，该第二目标神经网络层与第一目标神经网络层对应，且第二目标神经网络层为该训练后的模型中m个第二神经网络层中的一个。
训练后的模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二 特征图输入至与第一目标神经网络层唯一对应的第二目标神经网络层中，该第二目标神经网络层属于该训练后的模型中m个第二神经网络层中的一个。然后该第二目标神经网络层对接收到的第二特征图进行解码，从而得到训目标图像的增强图像(可称为第二增强图像)。
在本申请实施例中，训练后的模型中的第二目标神经网络层的执行过程可参阅上述图4对应的实施例中第二目标神经网络层4061，此处不予赘述。
在本申请上述实施例中，具体阐述了如何对本申请训练后的模型进行实际应用，从而得到目标图像对应的增强图像。该训练后的模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备多个第一神经网络层和多个第二神经网络层，不同的第一/二神经网络层对应不同的图像增强任务，从而该模型训练好后可用于处理不同的图像增强任务，并且相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
(2)训练后的模型的结构为模型800对应的结构。
请参阅图13，图13为本申请实施例提供的图像增强方法的另一种流程示意图，具体可以包括如下步骤：
1301、执行设备获取待处理的目标图像。
本申请实施例中，步骤1301与上述步骤1201类似，此处不予赘述。
1302、执行设备将目标图像输入训练后的模型，由该训练后的模型中的第一神经网络层对目标图像进行特征提取，得到第一特征图。
该执行设备上部署有训练后的模型，执行设备获取到目标图像后，会将该目标图像输入训练后的模型，由该训练后的模型中的第一神经网络层对目标图像进行特征提取，得到第一特征图。
在本申请实施例中，训练后的模型中的第一神经网络层的执行过程可参阅上述图8对应的实施例中第一神经网络层801，此处不予赘述。
1303、通过该训练后的模型中的切分模块对第一特征图进行切分，得到n个第一特征块。
得到的第一特征图会进一步输入至该训练后的模型的切分模块，由该切分模块对该第一特征图进行切分，得到n个特征块(可称为第一特征块)，n≥2。
需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分的过程具体可以是：首先对第一特征图进行切分，得到n个切分块，然后将这n个切分块中的每个切分块延展为一维向量表示的特征块(即第一特征块)，这样就可以得到n个第一特征块。
还需要说明的是，在本申请的一些实施方式中，切分模块对第一特征图进行切分，得到的n个切分块可以是尺寸均相同，也可以尺寸不相同，具体此处不做限定。
在本申请实施例中，训练后的模型中的切分模块的执行过程可参阅上述图8对应的实施例中切分模块802，此处不予赘述。
1304、由该训练后的模型中的transformer模块根据相关信息生成与n个第一特征块一一对应的n个第二特征块，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度。
训练后的模型中的切分模块得到n个第一特征块后，将这n个第一特征块进一步输入到该训练后的模型中的transformer模块进行处理，transformer模块基于这n个第一特征块，生成相关信息，该相关信息用于指示n个第一特征块中任意两个第一特征块之间的相关度，然后transformer模块根据该相关信息生成与n个第一特征块一一对应的n个第二特征块。也就是说，每个第一特征块，除了具有自身的特征信息外，还根据自身与其他第一特征块之间的相关度，融合了其他第一特征块的特征信息。这里需要注意的是，transformer模块输入的n个第一特征块的维度和输出的n个第二特征块的维度保持一致。
需要说明的是，在本申请的一些实施方式中，以transformer模块包括至少一个编码器和至少一个解码器为例，对transformer模块如何基于相关信息，生成与n个第一特征块一一对应的n个第二特征块进行说明：首先，通过编码器生成第一相关信息，并根据该第一相关信息，生成与这n个第一特征块一一对应的n个第三特征块，该第一相关信息用于指示n个第一特征块中任意两个第一特征块之间的第一相关度，并且编码器输入的n个第一特征块的维度与n个第三特征块的维度保持一致；之后，通过解码器生成第二相关信息，并根据该第二相关信息，生成与这n个第三特征块一一对应的n个第二特征块，该第二相关信息用于指示该n个第三特征块中任意两个第三特征块之间的第二相关度，并且解码器输入的n个第三特征块的维度与n个第二特征块的维度保持一致。这里需要注意的是，第二相关信息中融合了第一任务编码，该第一任务编码作为输入作用于解码器，该第一任务编码为输入图像所属的图像增强任务的对应标识，通过该任务编码，可以知道transformer模块接收到的n个第一特征块是来自于什么图像增强任务的输入图像。
在本申请实施例中，训练后的模型中的transformer模块的执行过程可参阅上述图8对应的实施例中transformer模块803，此处不予赘述。
1305、通过该训练后的模型中的重组模块对n个第二特征块进行拼接重组，得到第二特征图。
训练后的模型中的transformer模块基于相关信息由n个第一特征块得到n个第二特征块后，将通过训练后的模型中的重组模块对n个第二特征块按照空间相对位置进行拼接重组，得到与输入的第一特征图维度一致的第二特征图。
在本申请实施例中，训练后的模型中的重组模块的执行过程可参阅上述图8对应的实施例中重组模块804，此处不予赘述。
1306、通过该训练后的模型中的第二神经网络层对第二特征图进行解码，得到目标图像的第二增强图像。
训练后的模型中的重组模块将n个第二特征块拼接重组得到第二特征图，会将该第二特征图输入至第二神经网络层中，然后该第二神经网络层对接收到的第二特征图进行解码，从而得到训练样本的增强图像(可称为第二增强图像)。
在本申请实施例中，模型中的第二神经网络层的执行过程可参阅上述图8对应的实施 例中第二神经网络层805，此处不予赘述。
在本申请上述实施例中，具体阐述了如何对本申请训练后的模型进行实际应用，从而得到目标图像对应的增强图像。该训练后的模型结合了用于处理自然语言任务的transformer模块和不同的神经网络结构，突破了transformer模块只能用于处理自然语言任务的局限，该模型结构可应用在底层视觉任务中，该模型结构具备一个第一神经网络层和一个第二神经网络层，用于处理一个特定的图像增强任务，相比于现有的处理底层视觉任务的模型大多是基于CNN方式(CNN作为优良的特征提取器在高层视觉任务上能够大展拳脚，但是在处理底层视觉任务时难以关注全局信息)，该模型借助于transformer模块可关注到全局信息，从而可提高图像增强效果。
需要说明的是，本申请实施例构建的模型结构以及该模型经过训练得到的经过训练后的模型可以应用在多种图像增强任务中，在实际应用中，由于智能摄像头、智慧城市、智能终端等领域中都可以用到本申请实施例中训练好的模型来进行图像增强任务处理(如，超分辨率重建、去噪、去雾、去雨等)，下面将对多个落地到产品的多个应用场景进行介绍。
(1)相机图片的修复
相机照片修复是一项非常重要的技术，在处理手机成像效果等场景中具有重大的使用价值，目前相机图像修复的主要方法是采用多个针对不同图像增强任务的卷积神经网络模型进行的，使用本申请构建的模型结构，如图14所示，能够通过一个模型实现不同类型的图像增强任务，并且能够实现比多个特定任务的卷积神经网络模型更好的效果。
(2)手机拍照优化
本申请训练好的模型可用于终端(如，手机、智能手表、个人电脑等)的拍照优化，以终端为手机为例，当用户使用手机拍照时，自动抓取人脸、动物等目标，可以帮助手机自动对焦、美化等。若手机与被拍摄对象距离较远时，手机拍摄到的图像可能不太清楚，因此本申请训练好的模型就可应用于手机，该训练好的模型有效保留了图像像素的细节信息，优化后的图像画质也比现有神经网络优化的图像更清晰，可以给用户带来更好的用户体验，提升手机产品品质。
需要说明的是，本申请所述的训练好的模型不仅可以应用于上述所述的应用场景中，还可以应用在人工智能领域的各个细分领域中，只要能使用神经网络的领域和设备，都可应用本申请实施例提供的训练好的模型，此处不再举例示意。
为了对本申请实施例所带来的有益效果有更为直观的认识，以下对本申请实施例所带来的技术效果作进一步的对比，表1展示了本申请与基于CNN的最好模型的对比结果。从表1中可以看出，使用本申请所构建的模型以及训练方法训练得到的模型能够在多种图像增强任务上、以及多种数据集上均取得超越CNN模型的性能。此外，需要说明的是，不同的超分倍率需要使用不同的CNN，而本申请所提出模型则可以用一个模型适用于不同类型的图像增强任务。
表1、本申请构建的模型和CNN模型在超分辨率重建任务上的PSNR结果
超分方法 超分倍率 Set5数据集 Set14数据集 B100数据集 Urban100数据集
CNN ×2 38.24 34.07 32.41 33.23
本申请 ×2 38.37 34.43 32.48 33.76
CNN ×3 34.72 30.66 29.31 29.03
本申请 ×3 34.81 30.85 29.38 29.38
CNN ×4 32.57 28.85 27.77 26.84
本申请 ×4 32.64 29.01 27.82 27.26
在上述实施例的基础上，为了更好的实施本申请实施例的上述方案，下面还提供用于实施上述方案的相关设备。具体参阅图15，图15为本申请实施例提供的一种训练设备的示意图，该训练设备1500具体可以包括：获取模块1501、输入模块1502、训练模块1503，其中，获取模块1501用于训练样本，所述训练样本为构建的训练集中任意一个退化图像，其中，所述训练集中的每个退化图像由一个清晰图像经过图像退化处理得到；输入模块1502，用于向部署于该训练设备1500上的模型输入该训练样本，由该模型对该训练样本进行处理，得到该训练样本的第一增强图像；训练模块1503，用于根据第一增强图像、清晰图像和损失函数对部署于该训练设备1500上的模型进行训练，得到训练后的模型，所述清晰图像与所述训练样本对应。
需要说明的是，在本申请实施例中，训练设备1500上部署的模型具体的执行过程可参阅上述图4对应实施例所述的模型400或上述图8对应实施例所述的模型800，此处不予赘述。
还需要说明的是，训练设备1500中各模块/单元之间的信息交互、执行过程等内容，与本申请中图10或图11对应的方法实施例基于同一构思，具体内容可参见本申请前述所示的方法实施例中的叙述，此处不再赘述。
本申请实施例还提供一种执行设备，请参阅图16，图16为本申请实施例提供的一种执行设备的示意图，执行设备1600包括：获取模块1601、输入模块1602，其中，获取模块1601用于获取待处理的目标图像；输入模块1602用于将所述目标图像输入部署于该执行设备1600上的训练后的模型，由该训练后的模型对该目标图像进行处理，得到该目标图像的第二增强图像。
需要说明的是，在本申请实施例中，执行设备1600上部署的模型具体的执行过程可参阅上述图4对应实施例所述的模型400或上述图8对应实施例所述的模型800，此处不予赘述。
还需要说明的是，执行设备1600中各模块/单元之间的信息交互、执行过程等内容，与本申请中图12或图13对应的方法实施例基于同一构思，具体内容可参见本申请前述所示的方法实施例中的叙述，此处不再赘述。
接下来介绍本申请实施例提供的另一种训练设备，请参阅图17，图17为本申请实施例提供的训练设备的一种结构示意图，训练设备1700上可以部署有图15对应实施例中所描述的训练设备1500，用于实现图15对应实施例中训练设备1500的功能，具体的，训练设备1700由一个或多个服务器实现，训练设备1700可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(central processing units，CPU)1722和存储器1732，一个或一个以上存储应用程序1742或数据1744的存储介质1730(例如一个或一个以上海量存储设备)。其中，存储器1732和存储介质1730可以是短暂存储或持久存储。存储在存储介质1730的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包 括对训练设备1700中的一系列指令操作。更进一步地，中央处理器1722可以设置为与存储介质1730通信，在训练设备1700上执行存储介质1730中的一系列指令操作。
训练设备1700还可以包括一个或一个以上电源1726，一个或一个以上有线或无线网络接口1750，一个或一个以上输入输出接口1758，和/或，一个或一个以上操作系统1741，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
本申请实施例中，中央处理器1722，用于执行图10或图11对应实施例中的训练设备执行的模型的训练方法。
需要说明的是，中央处理器1722执行上述各个步骤的具体方式，与本申请中图10或图11对应的方法实施例基于同一构思，其带来的技术效果也与本申请上述实施例相同，具体内容可参见本申请前述所示的方法实施例中的叙述，此处不再赘述。
接下来介绍本申请实施例提供的一种执行设备，请参阅图18，图18为本申请实施例提供的执行设备的一种结构示意图，执行设备1800具体可以表现为各种终端设备，如虚拟现实VR设备、手机、平板、笔记本电脑、智能穿戴设备、监控数据处理设备或者雷达数据处理设备等，此处不做限定。其中，执行设备1800上可以部署有图16对应实施例中所描述的执行设备1600，用于实现图16对应实施例中执行设备1600的功能。具体的，执行设备1800包括：接收器1801、发射器1802、处理器1803和存储器1804(其中执行设备1800中的处理器1803的数量可以一个或多个，图18中以一个处理器为例)，其中，处理器1803可以包括应用处理器18031和通信处理器18032。在本申请的一些实施例中，接收器1801、发射器1802、处理器1803和存储器1804可通过总线或其它方式连接。
存储器1804可以包括只读存储器和随机存取存储器，并向处理器1803提供指令和数据。存储器1804的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。存储器1804存储有处理器和操作指令、可执行模块或者数据结构，或者它们的子集，或者它们的扩展集，其中，操作指令可包括各种操作指令，用于实现各种操作。
处理器1803控制执行设备1800的操作。具体的应用中，执行设备1800的各个组件通过总线系统耦合在一起，其中总线系统除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都称为总线系统。
本申请上述图12或图13对应实施例揭示的方法可以应用于处理器1803中，或者由处理器1803实现。处理器1803可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器1803中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器1803可以是通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器，还可进一步包括专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。该处理器1803可以实现或者执行本申请图12或图13对应的实施例中公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块 组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1804，处理器1803读取存储器1804中的信息，结合其硬件完成上述方法的步骤。
接收器1801可用于接收输入的数字或字符信息，以及产生与执行设备1800的相关设置以及功能控制有关的信号输入。发射器1802可用于通过第一接口输出数字或字符信息；发射器1802还可用于通过第一接口向磁盘组发送指令，以修改磁盘组中的数据；发射器1802还可以包括显示屏等显示设备。
本申请实施例中，在一种情况下，处理器1803，用于通过训练后的模型对输入的目标图像进行图像增强处理，得到对应的增强图像。该训练后的模型可以是经过本申请图10或图11对应的训练方法得到，具体内容可参见本申请前述所示的方法实施例中的叙述，此处不再赘述。
本申请实施例中还提供一种计算机可读存储介质，该计算机可读存储介质中存储有用于进行信号处理的程序，当其在计算机上运行时，使得计算机执行如前述所示实施例描述的训练设备所执行的步骤，或者，使得计算机执行如前述图16所示实施例描述的执行设备所执行的步骤。
本申请实施例提供的训练设备、执行设备等具体可以为芯片，芯片包括：处理单元和通信单元，所述处理单元例如可以是处理器，所述通信单元例如可以是输入/输出接口、管脚或电路等。该处理单元可执行存储单元存储的计算机执行指令，以使训练设备内的芯片执行上述所示实施例描述的训练设备所执行的步骤，或者，使得执行设备内的芯片执行如前述图16所示实施例描述的执行设备所执行的步骤。
可选地，所述存储单元为所述芯片内的存储单元，如寄存器、缓存等，所述存储单元还可以是所述无线接入设备端内的位于所述芯片外部的存储单元，如只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)等。
具体的，请参阅图19，图19为本申请实施例提供的芯片的一种结构示意图，所述芯片可以表现为神经网络处理器NPU 200，NPU 200作为协处理器挂载到主CPU(Host CPU)上，由Host CPU分配任务。NPU的核心部分为运算电路2003，通过控制器2004控制运算电路2003提取存储器中的矩阵数据并进行乘法运算。
在一些实现中，运算电路2003内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路2003是二维脉动阵列。运算电路2003还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路2003是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器2002中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器2001中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)2008中。
统一存储器2006用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控 制器(direct memory access controller，DMAC)2005，DMAC被搬运到权重存储器2002中。输入数据也通过DMAC被搬运到统一存储器2006中。
总线接口单元2010(bus interface unit，简称BIU)，用于AXI总线与DMAC和取指存储器(Instruction Fetch Buffer，IFB)2009的交互。
总线接口单元2010，用于取指存储器2009从外部存储器获取指令，还用于存储单元访问控制器2005从外部存储器获取输入矩阵A或者权重矩阵B的原数据。
DMAC主要用于将外部存储器DDR中的输入数据搬运到统一存储器2006或将权重数据搬运到权重存储器2002中或将输入数据数据搬运到输入存储器2001中。
向量计算单元2007包括多个运算处理单元，在需要的情况下，对运算电路的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。主要用于神经网络中非卷积/全连接层网络计算，如Batch Normalization(批归一化)，像素级求和，对特征平面进行上采样等。
在一些实现中，向量计算单元2007能将经处理的输出的向量存储到统一存储器2006。例如，向量计算单元2007可以将线性函数和/或非线性函数应用到运算电路2003的输出，例如对卷积层提取的特征平面进行线性插值，再例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元2007生成归一化的值、像素级求和的值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路2003的激活输入，例如用于在神经网络中的后续层中的使用。
控制器2004连接的取指存储器(instruction fetch buffer)2009，用于存储控制器2004使用的指令；
统一存储器2006，输入存储器2001，权重存储器2002以及取指存储器2009均为On-Chip存储器。外部存储器私有于该NPU硬件架构。
其中，上述任一处提到的处理器，可以是一个通用中央处理器，微处理器，ASIC，或一个或多个用于控制上述第一方面方法的程序执行的集成电路。
另外需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件的方式来实现，当然也可以通过专用硬件包括专用集成电路、专用CPU、专用存储器、专用元器件等来实现。一般情况下，凡由计算机程序完成的功能都可以很容易地用相应的硬件来实现，而且，用来实现同一功能的具体硬件结构也可以是多种多样的，例如模拟电路、数字电路或专用电路等。但是，对本申请而言更多情况下软件程序实现是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在可读取的存储介质中， 如计算机的软盘、U盘、移动硬盘、ROM、RAM、磁碟或者光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，训练设备，或者网络设备等)执行本申请各个实施例所述的方法。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。
所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、训练设备或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、训练设备或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存储的任何可用介质或者是包含一个或多个可用介质集成的训练设备、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(solid state disk，SSD))等。

Claims (38)

  1. 一种模型的结构，其特征在于，包括：
    选择模块、m个第一神经网络层、m个第二神经网络层、切分模块、重组模块以及transformer模块，其中，一个第一神经网络层唯一对应一个第二神经网络层，m≥2；
    所述选择模块，用于获取输入图像，并确定与所述输入图像对应的第一目标神经网络层，所述第一目标神经网络层为所述m个第一神经网络层中的一个；
    所述第一目标神经网络层，用于对所述输入图像进行特征提取，得到第一特征图；
    所述切分模块，用于对所述第一特征图进行切分，得到n个第一特征块，n≥2；
    所述transformer模块，用于根据相关信息，生成与所述n个第一特征块一一对应的n个第二特征块，所述相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的相关度；
    所述重组模块，用于对所述n个第二特征块进行拼接重组，得到第二特征图；
    第二目标神经网络层，用于对所述第二特征图进行解码，得到输出图像，所述第二目标神经网络层与所述第一目标神经网络层对应，且所述第二目标神经网络层为所述m个第二神经网络层中的一个。
  2. 根据权利要求1所述的结构，其特征在于，不同的第一神经网络层对应不同的图像增强任务，所述选择模块，具体用于：
    获取输入图像，并确定所述输入图像属于第一图像增强任务；
    确定与所述第一图像增强任务对应的第一目标神经网络层。
  3. 根据权利要求2所述的结构，其特征在于，所述输入图像为训练集中的训练样本，所述选择模块，具体还用于：
    获取所述训练样本，并根据所述训练样本的标签以确定所述训练样本属于所述第一图像增强任务。
  4. 根据权利要求2所述的结构，其特征在于，所述输入图像为待处理的目标图像，所述选择模块，具体还用于：
    获取所述目标图像，并根据接收到的指令以确定所述目标图像属于第一图像增强任务。
  5. 根据权利要求2-4中任一项所述的结构，其特征在于，所述transformer模块包括编码器和解码器；
    所述编码器，用于生成第一相关信息，并根据所述第一相关信息，生成与所述n个第一特征块一一对应的n个第三特征块，所述第一相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的第一相关度；
    所述解码器，用于生成第二相关信息，并根据所述第二相关信息，生成与所述n个第三特征块一一对应的n个所述第二特征块，所述第二相关信息中包括第一任务编码，所述第一任务编码为所述第一图像增强任务的对应标识，所述第二相关信息用于指示所述n个第三特征块中任意两个第三特征块之间的第二相关度。
  6. 根据权利要求1-5中任一项所述的结构，其特征在于，所述切分模块，具体用于：
    对所述第一特征图进行切分，得到n个切分块；
    将所述n个切分块中的每个切分块各自延展为一维向量表示的第一特征块，得到所述n个第一特征块。
  7. 根据权利要求6所述的结构，其特征在于，所述n个切分块的尺寸相同。
  8. 一种模型结构，其特征在于，包括：
    第一神经网络层、第二神经网络层、切分模块、重组模块以及transformer模块；
    所述第一神经网络层，用于对输入图像进行特征提取，得到第一特征图；
    所述切分模块，用于对所述第一特征图进行切分，得到n个第一特征块，n≥2；
    所述transformer模块，用于根据相关信息，生成与所述n个第一特征块一一对应的n个第二特征块，所述相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的相关度；
    所述重组模块，还用于对所述n个第二特征块进行拼接重组，得到第二特征图；
    所述第二神经网络层，用于对所述第二特征图进行解码，得到输出图像。
  9. 根据权利要求8所述的结构，其特征在于，所述transformer模块包括编码器和解码器；
    所述编码器，用于生成第一相关信息，并根据所述第一相关信息，生成与所述n个第一特征块一一对应的n个第三特征块，所述第一相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的第一相关度；
    所述解码器，用于生成第二相关信息，并根据所述第二相关信息，生成与所述n个第三特征块一一对应的n个所述第二特征块，所述第二相关信息中包括第一任务编码，所述第一任务编码为所述第一图像增强任务的对应标识，所述第二相关信息用于指示所述n个第三特征块中任意两个第三特征块之间的第二相关度。
  10. 根据权利要求8-9中任一项所述的结构，其特征在于，所述切分模块，具体用于：
    对所述第一特征图进行切分，得到n个切分块；
    将所述n个切分块中的每个切分块各自延展为一维向量表示的第一特征块，得到所述n个第一特征块。
  11. 根据权利要求10所述的结构，其特征在于，所述n个切分块的尺寸相同。
  12. 一种模型的训练方法，其特征在于，所述模型包括选择模块、m个第一神经网络层、m个第二神经网络层、切分模块、重组模块以及transformer模块，所述方法包括：
    获取训练样本，所述训练样本为构建的训练集中任意一个退化图像，其中，所述训练集中的每个退化图像由一个清晰图像经过图像退化处理得到；
    将所述训练样本输入所述模型中，由所述选择模块确定与所述训练样本对应的第一目标神经网络层，所述第一目标神经网络层为所述m个第一神经网络层中的一个；
    通过所述第一目标神经网络层对所述训练样本进行特征提取，得到第一特征图；
    通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块，n≥2；
    通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块，所述相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的相关度；
    通过所述重组模块对所述n个第二特征块进行拼接重组，得到第二特征图；
    通过第二目标神经网络层对所述第二特征图进行解码，得到所述训练样本的第一增强图像，所述第二目标神经网络层与所述第一目标神经网络层对应，且所述第二目标神经网络层为所述m个第二神经网络层中的一个；
    根据所述第一增强图像、清晰图像和损失函数对所述模型进行训练，得到训练后的模型，所述清晰图像与所述训练样本对应。
  13. 根据权利要求12所述的方法，其特征在于，不同的第一神经网络层对应不同的图像增强任务，所述由所述选择模块确定与所述训练样本对应的第一目标神经网络层包括：
    由所述选择模块根据所述训练样本的标签确定所述训练样本属于第一图像增强任务，并确定与所述第一图像增强任务对应的第一目标神经网络层。
  14. 根据权利要求12-13中任一项所述的方法，其特征在于，所述transformer模块包括编码器和解码器，所述通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块包括：
    通过所述编码器生成第一相关信息，并根据所述第一相关信息，生成与所述n个第一特征块一一对应的n个第三特征块，所述第一相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的第一相关度；
    通过所述解码器生成第二相关信息，并根据所述第二相关信息，生成与所述n个第三特征块一一对应的n个所述第二特征块，所述第二相关信息中包括第一任务编码，所述第一任务编码为所述第一图像增强任务的对应标识，所述第二相关信息用于指示所述n个第三特征块中任意两个第三特征块之间的第二相关度。
  15. 根据权利要求12-14中任一项所述的方法，其特征在于，所述通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块包括：
    通过所述切分模块对所述第一特征图进行切分，得到n个切分块，并将所述n个切分块中的每个切分块各自延展为一维向量表示的第一特征块，得到所述n个第一特征块。
  16. 根据权利要求15所述的方法，其特征在于，所述n个切分块的尺寸相同。
  17. 根据权利要求12-16中任一项所述的方法，其特征在于，所述方法还包括：
    将所述训练后的模型部署在目标设备上。
  18. 一种模型的训练方法，其特征在于，所述模型包括第一神经网络层、第二神经网络层、切分模块、重组模块以及transformer模块，所述方法包括：
    获取训练样本，所述训练样本为构建的训练集中任意一个退化图像，其中，所述训练集中的每个退化图像由一个清晰图像经过图像退化处理得到；
    将所述训练样本输入所述模型中，通过所述第一神经网络层对所述训练样本进行特征提取，得到第一特征图；
    通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块，n≥2；
    通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块，所述相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的相关度；
    通过所述重组模块对所述n个第二特征块进行拼接重组，得到第二特征图；
    通过所述第二神经网络层对所述第二特征图进行解码，得到所述训练样本的第一增强图像；
    根据所述第一增强图像、清晰图像和损失函数对所述模型进行训练，得到训练后的模型，所述清晰图像与所述训练样本对应。
  19. 根据权利要求18所述的方法，其特征在于，所述transformer模块包括编码器和解码器，所述通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块包括：
    通过所述编码器生成第一相关信息，并根据所述第一相关信息，生成与所述n个第一特征块一一对应的n个第三特征块，所述第一相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的第一相关度；
    通过所述解码器生成第二相关信息，并根据所述第二相关信息，生成与所述n个第三特征块一一对应的n个所述第二特征块，所述第二相关信息中包括第一任务编码，所述第一任务编码为所述第一图像增强任务的对应标识，所述第二相关信息用于指示所述n个第三特征块中任意两个第三特征块之间的第二相关度。
  20. 根据权利要求18-19中任一项所述的方法，其特征在于，所述通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块包括：
    通过所述切分模块对所述第一特征图进行切分，得到n个切分块，并将所述n个切分块中的每个切分块各自延展为一维向量表示的第一特征块，得到所述n个第一特征块。
  21. 根据权利要求20所述的方法，其特征在于，所述n个切分块的尺寸相同。
  22. 根据权利要求18-21中任一项所述的方法，其特征在于，所述方法还包括：
    将所述训练后的模型部署在目标设备上。
  23. 一种图像增强方法，其特征在于，包括：
    获取待处理的目标图像；
    将所述目标图像输入训练后的模型中，所述训练后的模型包括选择模块、m个第一神经网络层、m个第二神经网络层、切分模块、重组模块以及transformer模块；
    由所述选择模块确定与所述目标图像对应的第一目标神经网络层，所述第一目标神经网络层为所述m个第一神经网络层中的一个；
    通过所述第一目标神经网络层对所述目标图像进行特征提取，得到第一特征图；
    通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块，n≥2；
    通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块，所述相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的相关度；
    通过所述重组模块对所述n个第二特征块进行拼接重组，得到第二特征图；
    通过第二目标神经网络层对所述第二特征图进行解码，得到所述目标图像的第二增强图像，所述第二目标神经网络层与所述第一目标神经网络层对应，且所述第二目标神经网络层为所述m个第二神经网络层中的一个。
  24. 根据权利要求23所述的方法，其特征在于，不同的第一神经网络层对应不同的图像增强任务，所述由所述选择模块确定与所述目标图像对应的第一目标神经网络层包括：
    由所述选择模块根据接收到的指令确定所述目标图像属于第一图像增强任务，并确定与所述第一图像增强任务对应的第一目标神经网络层。
  25. 根据权利要求23-24中任一项所述的方法，其特征在于，所述transformer模块包括编码器和解码器，所述通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块包括：
    通过所述编码器生成第一相关信息，并根据所述第一相关信息，生成与所述n个第一特征块一一对应的n个第三特征块，所述第一相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的第一相关度；
    通过所述解码器生成第二相关信息，并根据所述第二相关信息，生成与所述n个第三特征块一一对应的n个所述第二特征块，所述第二相关信息中包括第一任务编码，所述第一任务编码为所述第一图像增强任务的对应标识，所述第二相关信息用于指示所述n个第三特征块中任意两个第三特征块之间的第二相关度。
  26. 根据权利要求23-25中任一项所述的方法，其特征在于，所述通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块包括：
    通过所述切分模块对所述第一特征图进行切分，得到n个切分块，并将所述n个切分块中的每个切分块各自延展为一维向量表示的第一特征块，得到所述n个第一特征块。
  27. 根据权利要求26所述的方法，其特征在于，所述n个切分块的尺寸相同。
  28. 一种图像增强方法，其特征在于，包括：
    获取待处理的目标图像；
    将所述目标图像输入训练后的模型中，所述训练后的模型包括第一神经网络层、第二神经网络层、切分模块、重组模块以及transformer模块；
    通过所述第一神经网络层对所述目标图像进行特征提取，得到第一特征图；
    通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块，n≥2；
    通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块，所述相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的相关度；
    通过所述重组模块对所述n个第二特征块进行拼接重组，得到第二特征图；
    通过所述第二神经网络层对所述第二特征图进行解码，得到所述目标图像的第二增强图像。
  29. 根据权利要求28所述的方法，其特征在于，所述transformer模块包括编码器和解码器，所述通过所述transformer模块根据相关信息生成与所述n个第一特征块一一对应的n个第二特征块包括：
    通过所述编码器生成第一相关信息，并根据所述第一相关信息，生成与所述n个第一特征块一一对应的n个第三特征块，所述第一相关信息用于指示所述n个第一特征块中任意两个第一特征块之间的第一相关度；
    通过所述解码器生成第二相关信息，并根据所述第二相关信息，生成与所述n个第三特征块一一对应的n个所述第二特征块，所述第二相关信息中包括第一任务编码，所述第一任务编码为所述第一图像增强任务的对应标识，所述第二相关信息用于指示所述n个第三特征块中任意两个第三特征块之间的第二相关度。
  30. 根据权利要求28-29中任一项所述的方法，其特征在于，所述通过所述切分模块对所述第一特征图进行切分，得到n个第一特征块包括：
    通过所述切分模块对所述第一特征图进行切分，得到n个切分块，并将所述n个切分块中的每个切分块各自延展为一维向量表示的第一特征块，得到所述n个第一特征块。
  31. 根据权利要求30所述的方法，其特征在于，所述n个切分块的尺寸相同。
  32. 一种训练设备，所述设备具有实现权利要求12-17中任一项所述方法的功能，或，所述设备具有实现权利要求18-22中任一项所述方法的功能，所述功能通过硬件或通过硬件执行相应的软件实现，所述硬件或所述软件包括一个或多个与所述功能相对应的模块。
  33. 一种执行设备，所述设备具有实现权利要求23-27中任一项所述方法的功能，或，所述设备具有实现权利要求28-31中任一项所述方法的功能，所述功能通过硬件或通过硬件执行相应的软件实现，所述硬件或所述软件包括一个或多个与所述功能相对应的模块。
  34. 一种训练设备，包括处理器和存储器，所述处理器与所述存储器耦合，其特征在于，
    所述存储器，用于存储程序；
    所述处理器，用于执行所述存储器中的程序，使得所述训练设备执行如权利要求12-17中任一项所述的方法，或，使得所述训练设备执行如权利要求18-22中任一项所述的方法。
  35. 一种执行设备，包括处理器和存储器，所述处理器与所述存储器耦合，其特征在于，
    所述存储器，用于存储程序；
    所述处理器，用于执行所述存储器中的程序，使得所述执行设备执行如权利要求23-27中任一项所述的方法，或，使得所述执行设备执行如权利要求28-31中任一项所述的方法。
  36. 一种计算机可读存储介质，包括程序，当其在计算机上运行时，使得计算机执行如权利要求12-31中任一项所述的方法。
  37. 一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行如权利要求12-31中任一项所述的方法。
  38. 一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行如权利要求12-31中任一项所述的方法。
PCT/CN2021/131704 2020-12-01 2021-11-19 一种模型结构、模型训练方法、图像增强方法及设备 WO2022116856A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21899890.4A EP4242917A4 (en) 2020-12-01 2021-11-19 MODEL STRUCTURE, MODEL TRAINING METHOD, AND IMAGE ENHANCEMENT METHOD AND DEVICE
US18/203,337 US20230306719A1 (en) 2020-12-01 2023-05-30 Model structure, method for training model, image enhancement method, and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011382775.1 2020-12-01
CN202011382775.1A CN112529150B (zh) 2020-12-01 2020-12-01 一种模型结构、模型训练方法、图像增强方法及设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/203,337 Continuation US20230306719A1 (en) 2020-12-01 2023-05-30 Model structure, method for training model, image enhancement method, and device

Publications (1)

Publication Number Publication Date
WO2022116856A1 true WO2022116856A1 (zh) 2022-06-09

Family

ID=74995778

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/131704 WO2022116856A1 (zh) 2020-12-01 2021-11-19 一种模型结构、模型训练方法、图像增强方法及设备

Country Status (4)

Country Link
US (1) US20230306719A1 (zh)
EP (1) EP4242917A4 (zh)
CN (1) CN112529150B (zh)
WO (1) WO2022116856A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115018711A (zh) * 2022-07-15 2022-09-06 成都运荔枝科技有限公司 一种用于仓库调度的图像超分辨率重建方法
CN116055895A (zh) * 2023-03-29 2023-05-02 荣耀终端有限公司 图像处理方法及其相关设备
CN116258658A (zh) * 2023-05-11 2023-06-13 齐鲁工业大学(山东省科学院) 基于Swin Transformer的图像融合方法

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112529150B (zh) * 2020-12-01 2024-06-14 华为技术有限公司 一种模型结构、模型训练方法、图像增强方法及设备
CN113112599A (zh) * 2021-03-29 2021-07-13 上海氢枫能源技术有限公司 一种基于vr技术的加氢站远程诊断方法、系统及电子设备
CN113378902B (zh) * 2021-05-31 2024-02-23 深圳神目信息技术有限公司 一种基于优化视频特征的视频抄袭检测方法
CN117529725A (zh) * 2021-06-28 2024-02-06 华为技术有限公司 图像处理方法及装置
CN114418868A (zh) * 2021-12-21 2022-04-29 云南联合视觉科技有限公司 图像增强方法、装置、服务器及存储介质
CN114494022B (zh) * 2022-03-31 2022-07-29 苏州浪潮智能科技有限公司 模型训练方法、超分辨率重建方法、装置、设备及介质
CN115565177B (zh) * 2022-08-16 2023-06-20 北京百度网讯科技有限公司 文字识别模型训练、文字识别方法、装置、设备及介质
CN115661460B (zh) * 2022-11-03 2023-07-14 广东工业大学 一种具有对比机制的相似度感知框架的医学图像分割方法
CN116137023B (zh) * 2023-04-20 2023-06-20 中国民用航空飞行学院 基于背景建模和细节增强的低照度图像增强方法
CN116309924B (zh) * 2023-05-25 2023-08-18 之江实验室 一种模型训练的方法、图像展示的方法及装置
CN117952865A (zh) * 2024-03-25 2024-04-30 中国海洋大学 一种基于循环生成对抗网络的单幅图像去雾方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109102483A (zh) * 2018-07-24 2018-12-28 厦门美图之家科技有限公司 图像增强模型训练方法、装置、电子设备及可读存储介质
EP3582142A1 (en) * 2018-06-15 2019-12-18 Université de Liège Image classification using neural networks
CN110866140A (zh) * 2019-11-26 2020-03-06 腾讯科技（深圳）有限公司 图像特征提取模型训练方法、图像搜索方法及计算机设备
WO2020154537A1 (en) * 2019-01-23 2020-07-30 Google Llc Convolutional neural networks with soft kernel selection
CN112529150A (zh) * 2020-12-01 2021-03-19 华为技术有限公司 一种模型结构、模型训练方法、图像增强方法及设备

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111950700A (zh) * 2020-07-06 2020-11-17 华为技术有限公司 一种神经网络的优化方法及相关设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3582142A1 (en) * 2018-06-15 2019-12-18 Université de Liège Image classification using neural networks
CN109102483A (zh) * 2018-07-24 2018-12-28 厦门美图之家科技有限公司 图像增强模型训练方法、装置、电子设备及可读存储介质
WO2020154537A1 (en) * 2019-01-23 2020-07-30 Google Llc Convolutional neural networks with soft kernel selection
CN110866140A (zh) * 2019-11-26 2020-03-06 腾讯科技（深圳）有限公司 图像特征提取模型训练方法、图像搜索方法及计算机设备
CN112529150A (zh) * 2020-12-01 2021-03-19 华为技术有限公司 一种模型结构、模型训练方法、图像增强方法及设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4242917A4

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115018711A (zh) * 2022-07-15 2022-09-06 成都运荔枝科技有限公司 一种用于仓库调度的图像超分辨率重建方法
CN115018711B (zh) * 2022-07-15 2022-10-25 成都运荔枝科技有限公司 一种用于仓库调度的图像超分辨率重建方法
CN116055895A (zh) * 2023-03-29 2023-05-02 荣耀终端有限公司 图像处理方法及其相关设备
CN116055895B (zh) * 2023-03-29 2023-08-22 荣耀终端有限公司 图像处理方法及其装置、芯片系统和存储介质
CN116258658A (zh) * 2023-05-11 2023-06-13 齐鲁工业大学(山东省科学院) 基于Swin Transformer的图像融合方法

Also Published As

Publication number Publication date
CN112529150A (zh) 2021-03-19
EP4242917A1 (en) 2023-09-13
CN112529150B (zh) 2024-06-14
EP4242917A4 (en) 2024-04-10
US20230306719A1 (en) 2023-09-28

Similar Documents

Publication Publication Date Title
WO2022116856A1 (zh) 一种模型结构、模型训练方法、图像增强方法及设备
US11593916B2 (en) Image super-resolution method and apparatus
WO2021018163A1 (zh) 神经网络的搜索方法及装置
WO2021043168A1 (zh) 行人再识别网络的训练方法、行人再识别方法和装置
WO2022017245A1 (zh) 一种文本识别网络、神经网络训练的方法以及相关设备
WO2019228358A1 (zh) 深度神经网络的训练方法和装置
WO2022042713A1 (zh) 一种用于计算设备的深度学习训练方法和装置
WO2021238366A1 (zh) 一种神经网络构建方法以及装置
WO2022179492A1 (zh) 一种卷积神经网络的剪枝处理方法、数据处理方法及设备
CN113066017B (zh) 一种图像增强方法、模型训练方法及设备
WO2022022288A1 (zh) 一种图像处理方法以及装置
CN113065645B (zh) 孪生注意力网络、图像处理方法和装置
US11983903B2 (en) Processing images using self-attention based neural networks
CN113705769A (zh) 一种神经网络训练方法以及装置
CN110222718B (zh) 图像处理的方法及装置
KR20220137076A (ko) 이미지 프로세싱 방법 및 관련된 디바이스
CN112598597A (zh) 一种降噪模型的训练方法及相关装置
WO2023231794A1 (zh) 一种神经网络参数量化方法和装置
CN113065635A (zh) 一种模型的训练方法、图像增强方法及设备
WO2021227787A1 (zh) 训练神经网络预测器的方法、图像处理方法及装置
CN113011562A (zh) 一种模型训练方法及装置
WO2024002211A1 (zh) 一种图像处理方法及相关装置
WO2022179588A1 (zh) 一种数据编码方法以及相关设备
WO2022111387A1 (zh) 一种数据处理方法及相关装置
CN114359289A (zh) 一种图像处理方法及相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21899890

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021899890

Country of ref document: EP

Effective date: 20230609

NENP Non-entry into the national phase

Ref country code: DE