CN112381733B - Image recovery-oriented multi-scale neural network structure searching method and network application - Google Patents

Image recovery-oriented multi-scale neural network structure searching method and network application Download PDF

Info

Publication number
CN112381733B
CN112381733B CN202011269424.XA CN202011269424A CN112381733B CN 112381733 B CN112381733 B CN 112381733B CN 202011269424 A CN202011269424 A CN 202011269424A CN 112381733 B CN112381733 B CN 112381733B
Authority
CN
China
Prior art keywords
image
layer
neural network
module
scale neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011269424.XA
Other languages
Chinese (zh)
Other versions
CN112381733A (en
Inventor
彭玺
缑元彪
李伯运
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN202011269424.XA priority Critical patent/CN112381733B/en
Publication of CN112381733A publication Critical patent/CN112381733A/en
Application granted granted Critical
Publication of CN112381733B publication Critical patent/CN112381733B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a multi-scale neural network structure searching method and network application for image recovery, wherein the multi-scale neural network structure searching method for image recovery comprises the steps of determining the number S of cells and the number of columns of each cell in a multi-scale neural network, and each column of the cells is only provided with a parallel module or a fusion module; constructing a super network to be searched on a multi-scale search space, wherein the super network comprises a combination module, 1 fusion module and 1 convolution layer, wherein the combination module is formed by sequentially connecting two parallel modules, an S pair of combination modules formed by serially connecting a transition module and cells; and determining a finally selected module of each cell column in the super network based on a search strategy of gradient optimization to obtain the multi-scale neural network.

Description

Image recovery-oriented multi-scale neural network structure searching method and network application
Technical Field
The invention relates to a neural network structure method, in particular to a multi-scale neural network structure searching method and network application for image recovery.
Background
Due to the complexity of the environment and the quality of the acquisition equipment, digital images are often degraded due to various noise pollution, and image recovery aims to recover clean and clear images from the degraded images.
At present, a Neural network is generally implemented by using a network Architecture Search (Neural Architecture Search), which is an automated Architecture engineering and aims to automatically find an ideal Neural network Architecture from a Search space by using Search strategies such as an evolutionary algorithm, reinforcement learning, gradient optimization and the like.
The E-CAE adopts an evolutionary algorithm to initialize a network structure population through coding a network structure into a corresponding genotype, and carries out iterative crossover, mutation and selection, thereby searching the number and the size of convolution kernels of a convolution layer on a standard convolution self-encoder architecture and judging whether jump connection is used. HiNAS adopts a gradient optimization-based method and a layering strategy, and constructs a to-be-searched super network by using cells as search units through continuous relaxation represented by a structure and a multi-candidate path strategy, thereby simultaneously searching the internal structure and the external convolution kernel number of the cells.
Although network structure search techniques based on E-CAE and hisnas achieve excellent performance, they often face the problem of explosion of structure combinations, requiring the use of large amounts of computing resources to achieve this result. For example, E-CAE needs to operate 384GPU hours on a Tesla P100 GPU, where GPU hours represent the number of hours a single GPU needs to operate.
To alleviate the problems with E-CAE, recent focus has shifted to gradient-based approaches, e.g., hisnas still needs to compute 16.5GPU hours on Tesla V100 GPU. In addition, most existing network structure search technologies are originally designed for classification tasks, and do not take the characteristics of image restoration tasks into consideration.
Disclosure of Invention
Aiming at the defects in the prior art, the image recovery-oriented multi-scale neural network structure searching method and the network application solve the problem that the image recovery task is not considered in the existing network structure searching method.
In order to achieve the purpose of the invention, the invention adopts the technical scheme that:
in a first aspect, a multi-scale neural network structure search method facing image recovery is provided, which includes:
s1, determining the number S of cells in the multi-scale neural network and the number of columns of each cell, wherein each column of the cells only has a parallel module or a fusion module;
s2, constructing a super network to be searched on a multi-scale search space, wherein the super network comprises two parallel modules, an S pair of combination modules formed by serially connecting a transition module and a cell, 1 fusion module and 1 convolution layer which are sequentially connected;
s3, determining a finally selected module of each cell column in the super network based on a search strategy of gradient optimization to obtain a multi-scale neural network; the method of searching for each column of finally selected modules for each cell includes:
s31, initializing current iteration q to 1, initializing iteration e, structure updating threshold t, super network weight parameter θ and super network structure parameter { a }p，af}；
S32, acquiring a data set consisting of a pair of degraded images and a pair of corresponding clean images, and dividing the data set into a training set and a verification set, wherein the degraded images and the corresponding clean images are respectively used as network input and target output;
s33, judging whether the current iteration number q is larger than a structure updating threshold value t, if so, entering a step S34, and otherwise, entering a step S35;
s34, based on the loss function, using the data in the verification set to update the structural parameter { a } of the super networkp，afBefore proceeding to step S35;
s35, updating the weight parameter theta of the super network by using the data in the training set based on the loss function, and then entering the step S36;
s36, determining whether the current iteration number q is greater than the iteration number e, if so, entering step S37, otherwise, making q equal to q +1, and then returning to step S33;
s37, judging the structural parameter a of the parallel module of each column in each cellpWhether or not it is greater than the structural parameter a of the fusion modulefIf so, selecting the parallel module corresponding to the column, otherwise, selecting the fusion module corresponding to the column.
In a second aspect, an application of a multi-scale neural network is provided, where the multi-scale neural network is obtained by using a multi-scale neural network structure search method oriented to image recovery, and the method includes:
training the multi-scale neural network, and then inputting the degraded image into the trained multi-scale neural network to obtain a clean image; the method for training the multi-scale neural network comprises the following steps:
acquiring a data set consisting of a degraded image and a clean image, taking the degraded image as the input of a network structure, and taking the corresponding clean image as the target output of the network structure;
based on loss functions
Figure BDA0002777225460000031
Updating the weight parameters of the multi-scale neural network by adopting the data in the training set;
and when the weight parameters of the multi-scale neural network are converged, obtaining the trained multi-scale neural network.
The invention has the beneficial effects that: according to the technical scheme, a super network to be searched is constructed in a multi-scale search space, and a search strategy based on gradient optimization is utilized to directly search a network structure on a specific image recovery task.
The multi-scale search space of the scheme is composed of some modularized operations instead of basic operators, namely, basic operations in network modules instead of each module are searched, so that the space to be searched is greatly reduced, the search time is further reduced, and the search efficiency is improved.
When the module of each column of cells is determined, by introducing the structure updating threshold value, the network weight parameter has a better initial value when the network structure is searched, and further the condition that the structure parameter and the weight parameter are together trapped in a local optimal interval in the alternating optimization process to cause the search of a suboptimal network structure is avoided.
Drawings
Fig. 1 is a flowchart of a multi-scale neural network structure search method for image restoration.
FIG. 2 is a flow chart of a method of searching for a module finally selected for each column of each cell.
FIG. 3 is an example of a search space according to the present scheme; (a) the structure of the parallel module, the transition module and the fusion module is shown schematically; (b) the structure of the cell is shown, and each column comprises a parallel module or a fusion module; (c) the structure of the super network is shown schematically.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.
Referring to fig. 1, fig. 1 shows a flow chart of a multi-scale neural network structure search method for image recovery; as shown in fig. 1, the method S includes steps S1 to S3.
In step S1, the number S of cells in the multi-scale neural network and the number of columns per cell are determined, each column of cells having only parallel modules or fusion modules.
In step S2, a super-network to be searched is constructed on a multi-scale search space, where the super-network includes a combination module, 1 fusion module and 1 convolution layer, which are connected in series by a transition module and a cell in S pair, and two parallel modules are connected in sequence.
In one embodiment of the invention, the parallel module is used for keeping the resolution of the features on each parallel path unchanged through operation, and comprises at least one parallel volume block, wherein the number of the volume blocks is equal to the resolution number of the image of the input neural network;
the transition module is used for adding a new parallel path with low resolution, and simultaneously keeping the characteristics of the existing parallel path unchanged, and comprises null operation blocks and sampling blocks, wherein the number of the null operation blocks is equal to the number of the resolutions input into the transition module, and the minimum resolution is downsampled by the downsampling block;
the fusion module is used for fusing the features on each parallel path with the features on each path respectively through downsampling, upsampling and/or null operation on the input resolution, and comprises a null operation block, an upsampling block and a downsampling block.
Taking fig. 3 as an example, a network structure is described, taking the case of three parallel branches in (a) as an example, first, a transition module in a super network converts 2 parallel branches into 3 parallel branches, then, a parallel module and a fusion module containing 3 parallel branches are combined into a cell, refer to (b) in fig. 3, and finally, after the cell is connected to the transition module, refer to (c) in fig. 3, through this process, a network part containing 3 parallel branches in the super network is constructed.
In implementation, the scheme preferably selects the 1 st layer of the lower sampling block as a convolution layer and the 2 nd layer as a batch normalization layer, and then repeats the structures of the 1 st to 2 nd layers
Figure BDA0002777225460000051
Second, 2R1+1 layer is a linear rectification function layer, where SresFor the input resolution of the downsampled block, TresIs the target resolution of the output;
the 1 st layer of the rolling block is a rolling layer, the 2 nd layer is a batch normalization layer, and the 3 rd layer is a linear rectification function layer; the 4 th layer is a convolution layer, the 5 th layer is a batch normalization layer, and the 6 th layer is a linear rectification function layer;
the 1 st layer of the up-sampling block is a convolution layer, the 2 nd layer is a batch normalization layer, the 3 rd layer is a linear rectification function layer, and then the structures of the 1-3 layers are repeated
Figure BDA0002777225460000061
Then, wherein S'resFor the input resolution of the upsampled block, TresIs the target resolution of the output.
In step S3, determining a finally selected module for each column of each cell in the super network based on a search strategy of gradient optimization to obtain a multi-scale neural network;
wherein the method of searching for each column of finally selected modules for each cell comprises:
s31, initializing current iteration number q to 1, initializing iteration number e, structure updating threshold t, super network weight parameter θ and super network structure parameter { a }p，af}；
S32, acquiring a data set consisting of a pair of degraded images and a pair of corresponding clean images, and dividing the data set into a training set and a verification set, wherein the degraded images and the corresponding clean images are respectively used as network input and target output;
when the method is implemented, the preferable image recovery comprises image rain removal, image denoising and image defogging;
when the image recovery task is image rain removal, the degraded image is a shot natural image containing rain, and the clean image is a shot natural image without rain in a corresponding scene;
when the image recovery task is image denoising, the degraded image is a shot natural image containing noise, and the clean image is a shot natural image without noise in a corresponding scene;
when the image recovery task is image defogging, the degraded image is a shot foggy natural image, and the clean image is a shot fogless natural image in a corresponding scene.
The multi-scale neural network obtained by the searching in the scheme can be applied to all target objects in the nature, and the method can be adopted to recover as long as noise, rainwater or fog exist in the target objects, namely, the noise, the rainwater or the fog in the image is eliminated.
S33, judging whether the current iteration number q is larger than a structure updating threshold value t, if so, entering a step S34, and otherwise, entering a step S35;
s34, based on the loss function, using the data in the verification set to update the structural parameter { a } of the super networkp，afStep S35 is then entered;
s35, updating the weight parameter theta of the super network by using the data in the training set based on the loss function, and then entering the step S36;
in further detail, step S34 is based on the loss function, and uses the adaptive moment estimation optimizer to update the network structure parameters by using the data in the verification set; step S35 is to update the network weight parameters based on the loss function by using the stochastic gradient descent optimizer and the cosine annealing strategy and using the data in the training set.
According to the scheme, different network parameters are updated by comparing the iteration times with the structure updating threshold value t, so that when the network structure is searched, the network weight parameters have a better initial value, and the condition that the structure parameters and the weight parameters are trapped in a local optimal interval together in the alternate optimization process to cause the searching of a suboptimal network structure is avoided.
In implementation, the preferred calculation formula of the loss function in the scheme is as follows:
Figure BDA0002777225460000071
Figure BDA0002777225460000072
Figure BDA0002777225460000073
wherein λ is1And λ2Are all weight coefficients;
Figure BDA0002777225460000074
is a real clean image
Figure BDA0002777225460000075
And clean image f (x) of network recoveryi) Recovery loss in between; i is an image pixel index; n is the number of image pixels;
Figure BDA0002777225460000076
loss for structural regularization; n is { ap，afThe number of structural parameters in the structure; a is { ap，afStructural parameters in (1);
Figure BDA0002777225460000077
is a micro-loss term used to measure the complexity of the model; cpAnd CfThe complexity of the parallel module and the fusion module respectively.
A in loss function losspAll columns per cell are indicated
Figure BDA0002777225460000081
Set of (a)fAll columns per cell are indicated
Figure BDA0002777225460000082
A collection of (a).
The complexity of the parallel and fusion modules may be predefined by module size, time cost, FLOPs, etc. In the scheme, the complexity and the loss item of the model are measured by adopting the module size
Figure BDA0002777225460000083
The scheme can control the balance between the performance and the size of the model, and has high value in wide application scenes including resource-limited scenes.
S36, determining whether the current iteration number q is greater than the iteration number e, if so, entering step S37, otherwise, making q equal to q +1, and then returning to step S33;
s37, judging the structure parameter a of the parallel module of each column in each cellpWhether or not it is greater than the structural parameter a of the fusion modulefIf yes, corresponding row is selectedAnd taking a parallel module, otherwise, selecting a fusion module corresponding to the column.
In one embodiment of the invention, the output of each column of cells within the super network is:
Figure BDA0002777225460000084
wherein, yj+1The output of column j +1 for the cell;
Figure BDA0002777225460000085
the outputs of the parallel module and the fusion module of the jth row of the cell respectively;
Figure BDA0002777225460000086
the structural parameters of the parallel module and the fusion module in the j-th column of the cell are respectively.
The scheme also provides an application of the multi-scale neural network obtained by adopting the multi-scale neural network structure searching method facing the image recovery, which comprises the following steps:
training the multi-scale neural network, and then inputting the degraded image into the trained multi-scale neural network to obtain a clean image; the method for training the multi-scale neural network comprises the following steps:
acquiring a data set consisting of a degraded image and a clean image, taking the degraded image as the input of a network structure, and taking the corresponding clean image as the target output of the network structure;
based on loss functions
Figure BDA0002777225460000091
Updating the weight parameters of the multi-scale neural network by adopting the data in the training set;
and when the weight parameters of the multi-scale neural network are converged, obtaining the trained multi-scale neural network.
In order to verify the effectiveness of the multi-scale neural network obtained by the searching in the scheme, the multi-scale neural network obtained by the searching is applied to two image recovery tasks, namely image denoising and image rain removing.
In image denoising, the input of the network is an image containing noise, and the output of the network is a corresponding clean image; in the aspect of image rain removal, the input of the network is an image containing rain, and the output of the network is a corresponding clean image. Both tasks follow the following experimental setup: the constructed super-network comprises 3 cells, each cell comprises 4 columns, each column is composed of parallel modules and fusion modules which are relaxed continuously, namely S is 3, Ni＝4，i∈{1,…,S}。
The super network comprises 4 parallel paths corresponding to 4 different feature resolutions from high to low, and the number of corresponding feature channels is 32, 64, 128 and 256. In the training process, when the weight parameter of the hyper-network is updated, a random gradient descent optimizer with momentum of 0.9 and weight attenuation of 0.0003 is adopted, and a cosine annealing strategy is adopted to automatically attenuate the learning rate from 0.025 to 0.001. When the structural parameters of the hyper-network are updated, an adaptive moment estimation (Adam) optimizer is adopted, the learning rate is set to be 0.0003, and the weight attenuation is set to be 0.001.
The iteration number of the training super network is e-10,000, and 32 image blocks of 64 × 64 are randomly cut out from the verification set and the training set respectively in each iteration and are respectively used for updating the network structure parameters and the network weight parameters. In order to avoid that the structure parameters and the weight parameters jointly fall into a local optimal interval in the alternating optimization process, only the weight parameters are updated in the previous 1,000 iterations, namely the weight parameters and the structure parameters are updated alternately after the 1,000 th iteration.
For fair comparison, λ is set1＝0.1，λ20, i.e. as with other solutions, the problem of model complexity is not taken into account. In the evaluation index, the peak signal-to-noise ratio (PSNR) of the image and the Structural Similarity (SSIM) of the image are used for evaluation, and the larger the value, the better the value.
Experiment one: the performance of the scheme on image denoising is evaluated by using the BSD500 data set. The BSD500 is a data set consisting of 500 natural images. The training set comprises 200 images, the verification set comprises 100 images, and the test set comprises 200 images. In the experiment, a training set and a verification set are adopted to train the super-network and derive and search to obtain a multi-scale neural network, then the training set and the verification set are utilized to train and verify the multi-scale neural network, finally the converged multi-scale neural network is applied to a test set, and the peak signal-to-noise ratio (PSNR) and the Structural Similarity (SSIM) are calculated.
In order to verify the superiority of the present solution, it is compared with 6 solutions in the prior art in the case where the noise standard deviation σ is 30,50, and 70, respectively. The 6 schemes are block matching and three-dimensional filtering (BM3D), weighted kernel norm minimization (WNNM), memory network (MemNet), non-local recursive network (NLRN), evolutionary convolutional self-encoder (E-CAE), hierarchical neural network structure search (HiNAS), and the test results are shown in Table 1.
Table 1 test results of this and 6 solutions with a standard deviation of noise σ of 30,50, and 70
Figure BDA0002777225460000101
As can be seen from table 1, the recovery effect of this solution is better than that of the other 6 technical solutions. Specifically, when the noise standard deviation σ is 30, the PSNR and SSIM of the scheme are higher than E-CAE by 1.44 and 0.0384, respectively. The corresponding indices were also improved by 0.53 and 0.0028, respectively, compared to HiNAS. Similar results can also be observed when σ is 50 and σ is 70, and the results of table 1 verify the validity of the image restoration of the present scheme.
Experiment two: the Rain removal performance of the scheme on the image is evaluated by using the Rain800 data set. Rain800 is a composite data set of 800 images, containing 700 training images and 100 test images. In the experiment, 100 training images are arbitrarily taken out from 700 training images as a verification set, the rest 600 training images are taken as a training set, a training set and the verification set are adopted to train a hyper-network and derive and search to obtain a multi-scale neural network, then the training set and the verification set are used to train and verify the multi-scale neural network, finally the converged multi-scale neural network is applied to a test set, and the peak signal-to-noise ratio (PSNR) and the Structural Similarity (SSIM) are calculated.
In order to verify the superiority of the scheme, the scheme is compared with 8 technical schemes in the prior art, namely a discrimination sparse coding method (DSC), a layer prior method (LP), a depth detail network (detailsNet), a depth joint rain detection and removal (JORDER, JORDER-R), a cyclic compression-excitation context aggregation network (SCAN), a residual cyclic compression-excitation context aggregation network (RESCAN), a hierarchical neural network structure search (HiNAS), and the comparison result is shown in a table 2.
Table 2 test results of peak snr and structural similarity for this and 8 schemes
Index \ method DSC LP DetailsNet JORDER JORDER-R SCAN RESCAN HiNAS This scheme
PSNR 18.56 20.46 21.16 22.24 22.29 23.45 24.09 26.31 26.58
SSIM 0.5996 0.7297 0.7320 0.7763 0.7922 0.8112 0.8410 0.8685 0.8530
As can be seen from table 2, the performance of this scheme and the hisnas is significantly better than that of other comparative schemes. For example, compared with the optimal technical scheme except HiNAS, the PSNR and SSIM indexes of the scheme are respectively 2.49 and 0.012 higher. Although the scheme is slightly lower than HiNAS in SSIM index, but higher than HiNAS in PSNR index, the test results in Table 2 again demonstrate the effectiveness of the scheme in image recovery.

Claims (6)

1. The image recovery-oriented multi-scale neural network structure searching method is characterized by comprising the following steps:
s1, determining the number S of cells in the multi-scale neural network and the number of columns of each cell, wherein each column of the cells only has a parallel module or a fusion module;
s2, constructing a super network to be searched on a multi-scale search space, wherein the super network comprises two parallel modules, an S pair of combination modules formed by serially connecting a transition module and a cell, 1 fusion module and 1 convolution layer which are sequentially connected;
s3, determining a finally selected module of each cell column in the super network based on a search strategy of gradient optimization to obtain a multi-scale neural network; the method of searching for each column of finally selected modules for each cell includes:
s31, initializing current iteration q to 1, initializing iteration e, structure updating threshold t, super network weight parameter θ and super network structure parameter { a }p，af}；
S32, acquiring a data set consisting of a degraded image and a corresponding clean image, and dividing the data set into a training set and a verification set, wherein the degraded image and the corresponding clean image are respectively used as a network input and a target output;
s33, judging whether the current iteration number q is larger than a structure updating threshold value t, if so, entering a step S34, and otherwise, entering a step S35;
s34, updating the structural parameter { a ] of the hyper-network by using the data in the verification set based on the loss functionp，afStep S35 is then entered;
s35, updating the weight parameter theta of the super network by using the data in the training set based on the loss function, and then entering the step S36;
s36, determining whether the current iteration number q is greater than the iteration number e, if so, entering step S37, otherwise, making q equal to q +1, and then returning to step S33;
s37, judging the structural parameter a of the parallel module of each column in each cellpWhether or not it is greater than the structural parameter a of the fusion modulefIf so, selecting a parallel module corresponding to the row, otherwise, selecting a fusion module corresponding to the row;
the parallel module is used for keeping the resolution of the features on each parallel path unchanged through operation, and comprises at least one parallel rolling block, and the number of the rolling blocks is equal to the resolution number of the images input to the neural network;
the transition module is used for adding a new parallel path with low resolution, and simultaneously keeping the characteristics of the existing parallel path unchanged, and comprises null operation blocks and sampling blocks, wherein the number of the null operation blocks is equal to the number of the resolutions input into the transition module, and the minimum resolution is downsampled by the downsampling block;
the fusion module is used for fusing the features on each parallel path with the features on each path respectively through downsampling, upsampling and/or null operation on the input resolution, and comprises a null operation block, an upsampling block and a downsampling block.
2. The image restoration-oriented multi-scale neural network structure searching method according to claim 1, wherein the output of each column of cells in the super network is:
Figure FDA0003620669200000021
wherein, yj+1The output of column j +1 for the cell;
Figure FDA0003620669200000022
the outputs of the parallel module and the fusion module of the jth row of the cell respectively;
Figure FDA0003620669200000023
the structural parameters of the parallel module and the fusion module in the j-th column of the cell are respectively.
3. The image restoration-oriented multi-scale neural network structure searching method according to claim 1, wherein the loss function is calculated by the formula:
Figure FDA0003620669200000024
Figure FDA0003620669200000025
Figure FDA0003620669200000031
wherein λ is1And λ2Are all weight coefficients;
Figure FDA0003620669200000032
is a real clean image
Figure FDA0003620669200000033
And clean image f (x) of network recoveryi) Recovery loss in between; i is an image pixel index; n is the number of image pixels;
Figure FDA0003620669200000034
loss of structural regularization; n is { ap，afThe number of structural parameters in the structure; a is { ap，afStructural parameters in (1);
Figure FDA0003620669200000035
is a micro-loss term used to measure the complexity of the model; cpAnd CfThe complexity of the parallel module and the fusion module respectively.
4. The image restoration-oriented multi-scale neural network structure searching method as claimed in claim 1, wherein the layer 1 of the downsampling block is a convolution layer, the layer 2 is a batch normalization layer, and then the structures of the layer 1-2 are repeated
Figure FDA0003620669200000036
Second, 2R1+1 layer is a linear rectification function layer, where SresFor the input resolution of the downsampled block, TresIs the target resolution of the output;
the 1 st layer of the rolling block is a rolling layer, the 2 nd layer is a batch normalization layer, and the 3 rd layer is a linear rectification function layer; the 4 th layer is a convolution layer, the 5 th layer is a batch normalization layer, and the 6 th layer is a linear rectification function layer;
the 1 st layer of the up-sampling block is a convolution layer, the 2 nd layer is a batch normalization layer, the 3 rd layer is a linear rectification function layer, and then the structures of the 1-3 layers are repeated
Figure FDA0003620669200000037
Then, wherein S'resFor the input resolution of the upsampled block, TresIs the target resolution of the output.
5. The image recovery-oriented multi-scale neural network structure searching method as claimed in claim 1, wherein the image recovery comprises image rain removal, image de-noising and image defogging;
when the image recovery task is image rain removal, the degraded image is a shot natural image containing rain, and the clean image is a shot natural image without rain in a corresponding scene;
when the image recovery task is image denoising, the degraded image is a shot natural image containing noise, and the clean image is a shot natural image without noise in a corresponding scene;
when the image recovery task is image defogging, the degraded image is a shot foggy natural image, and the clean image is a shot fogless natural image in a corresponding scene.
6. An application method of a multi-scale neural network, wherein the multi-scale neural network is obtained by adopting the multi-scale neural network structure searching method facing image recovery as claimed in any one of claims 1 to 5, and the method is characterized by comprising the following steps:
training the multi-scale neural network, and then inputting the degraded image into the trained multi-scale neural network to obtain a clean image; the method for training the multi-scale neural network comprises the following steps:
acquiring a data set consisting of a degraded image and a clean image, taking the degraded image as the input of a network structure, and taking the corresponding clean image as the target output of the network structure;
based on loss functions
Figure FDA0003620669200000041
Updating the weight parameters of the multi-scale neural network by adopting the data in the training set;
and when the weight parameters of the multi-scale neural network are converged, obtaining the trained multi-scale neural network.
CN202011269424.XA 2020-11-13 2020-11-13 Image recovery-oriented multi-scale neural network structure searching method and network application Active CN112381733B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011269424.XA CN112381733B (en) 2020-11-13 2020-11-13 Image recovery-oriented multi-scale neural network structure searching method and network application

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011269424.XA CN112381733B (en) 2020-11-13 2020-11-13 Image recovery-oriented multi-scale neural network structure searching method and network application

Publications (2)

Publication Number Publication Date
CN112381733A CN112381733A (en) 2021-02-19
CN112381733B true CN112381733B (en) 2022-07-01

Family

ID=74582212

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011269424.XA Active CN112381733B (en) 2020-11-13 2020-11-13 Image recovery-oriented multi-scale neural network structure searching method and network application

Country Status (1)

Country Link
CN (1) CN112381733B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112802012A (en) * 2021-03-09 2021-05-14 中南大学湘雅医院 Pathological image detection method, pathological image detection device, computer equipment and storage medium
CN113298239B (en) * 2021-07-06 2024-04-16 东南大学 Neural component searching method for generating image super-resolution network
CN117058000B (en) * 2023-10-10 2024-02-02 苏州元脑智能科技有限公司 Neural network architecture searching method and device for image super-resolution

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106708943A (en) * 2016-11-22 2017-05-24 安徽睿极智能科技有限公司 Image retrieval reordering method and system based on arrangement fusion
CN108364262A (en) * 2018-01-11 2018-08-03 深圳大学 A kind of restored method of blurred picture, device, equipment and storage medium
CN109003240A (en) * 2018-07-16 2018-12-14 安徽理工大学 A kind of image de-noising method based on multiple dimensioned parallel C NN
AU2019100352A4 (en) * 2019-04-04 2019-05-09 Duan, Zicheng Mr A Flower-Image Search System Based on Deep Learning
CN110175671A (en) * 2019-04-28 2019-08-27 华为技术有限公司 Construction method, image processing method and the device of neural network
CN110599409A (en) * 2019-08-01 2019-12-20 西安理工大学 Convolutional neural network image denoising method based on multi-scale convolutional groups and parallel
CN111125416A (en) * 2019-12-27 2020-05-08 郑州轻工业大学 Image retrieval method based on multi-feature fusion
CN111860495A (en) * 2020-06-19 2020-10-30 上海交通大学 Hierarchical network structure searching method and device and readable storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106708943A (en) * 2016-11-22 2017-05-24 安徽睿极智能科技有限公司 Image retrieval reordering method and system based on arrangement fusion
CN108364262A (en) * 2018-01-11 2018-08-03 深圳大学 A kind of restored method of blurred picture, device, equipment and storage medium
CN109003240A (en) * 2018-07-16 2018-12-14 安徽理工大学 A kind of image de-noising method based on multiple dimensioned parallel C NN
AU2019100352A4 (en) * 2019-04-04 2019-05-09 Duan, Zicheng Mr A Flower-Image Search System Based on Deep Learning
CN110175671A (en) * 2019-04-28 2019-08-27 华为技术有限公司 Construction method, image processing method and the device of neural network
CN110599409A (en) * 2019-08-01 2019-12-20 西安理工大学 Convolutional neural network image denoising method based on multi-scale convolutional groups and parallel
CN111125416A (en) * 2019-12-27 2020-05-08 郑州轻工业大学 Image retrieval method based on multi-feature fusion
CN111860495A (en) * 2020-06-19 2020-10-30 上海交通大学 Hierarchical network structure searching method and device and readable storage medium

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
CCF A类会议NeurIPS 2020论文收录结果出炉：我院彭玺教授指导学生共有两篇论文入选;huangxj;《https://sw.scu.edu.cn/info/1036/11934.htm》;20200930;1-5 *
Evolutionary Neural Architecture Search for Image Restoration;Gerard Jacques van Wyk;《2019 International Joint Conference on Neural Networks (IJCNN)》;20190930;1-8 *
MS-NAS: Multi-Scale Neural Architecture Search for Medical Image Segmentation;Xingang Yan;《https://arxiv.org/abs/2007.06151v1》;20200713;1-4 *
NeurIPS 2020｜图像处理（超分、去噪等）、底层视觉相关论文汇总;佚名;《https://blog.csdn.net/AI_Conf/article/details/109228700》;20201022;1-3 *
Self-Guided Network for Fast Image Denoising;Shuhang Gu;《2019 IEEE/CVF International Conference on Computer Vision (ICCV)》;20200227;2511-2520 *
图像序列预测及其超参数优化方法研究;彭程;《中国优秀硕士学位论文全文数据库 (信息科技辑)》;20200215;I138-1645 *
基于卷积神经网络的低分辨率细胞图像分割算法研究;刘仪;《中国优秀硕士学位论文全文数据库 (信息科技辑)》;20190815;I138-753 *

Also Published As

Publication number Publication date
CN112381733A (en) 2021-02-19

Similar Documents

Publication Publication Date Title
CN112381733B (en) Image recovery-oriented multi-scale neural network structure searching method and network application
CN110532859B (en) Remote sensing image target detection method based on deep evolution pruning convolution net
CN112991354B (en) High-resolution remote sensing image semantic segmentation method based on deep learning
CN111951207B (en) Image quality enhancement method based on deep reinforcement learning and semantic loss
CN111179187B (en) Single image rain removing method based on cyclic generation countermeasure network
CN111832484A (en) Loop detection method based on convolution perception hash algorithm
CN113269224B (en) Scene image classification method, system and storage medium
CN112634171B (en) Image defogging method and storage medium based on Bayesian convolutional neural network
CN112651360B (en) Skeleton action recognition method under small sample
CN112967327A (en) Monocular depth method based on combined self-attention mechanism
CN112241939A (en) Light-weight rain removing method based on multi-scale and non-local
CN113780389B (en) Deep learning semi-supervised dense matching method and system based on consistency constraint
CN112418229A (en) Unmanned ship marine scene image real-time segmentation method based on deep learning
CN116542865A (en) Multi-scale real-time defogging method and device based on structural re-parameterization
CN116863379A (en) Video prediction defense method based on space-time self-attention single-step disturbance
CN115953902A (en) Traffic flow prediction method based on multi-view space-time diagram convolution network
CN113222016B (en) Change detection method and device based on cross enhancement of high-level and low-level features
CN117011699A (en) GAN model-based crop identification model of high-resolution remote sensing image and identification method thereof
CN114220019B (en) Lightweight hourglass type remote sensing image target detection method and system
CN115937693A (en) Road identification method and system based on remote sensing image
CN115375966A (en) Image countermeasure sample generation method and system based on joint loss function
He et al. Feature aggregation convolution network for haze removal
CN113283400B (en) Skeleton action identification method based on selective hypergraph convolutional network
CN117315324B (en) Lightweight class detection method and system for Mars rugged topography
CN116091787B (en) Small sample target detection method based on feature filtering and feature alignment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant