CN114677306A - Context aggregation image rain removing method based on edge information guidance - Google Patents

Context aggregation image rain removing method based on edge information guidance Download PDF

Info

Publication number
CN114677306A
CN114677306A CN202210319123.6A CN202210319123A CN114677306A CN 114677306 A CN114677306 A CN 114677306A CN 202210319123 A CN202210319123 A CN 202210319123A CN 114677306 A CN114677306 A CN 114677306A
Authority
CN
China
Prior art keywords
information
image
rain
decoder
dnc
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210319123.6A
Other languages
Chinese (zh)
Other versions
CN114677306B (en
Inventor
王军
左慧园
潘在宇
韩淑雨
李玉莲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China University of Mining and Technology CUMT
Original Assignee
China University of Mining and Technology CUMT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China University of Mining and Technology CUMT filed Critical China University of Mining and Technology CUMT
Priority to CN202210319123.6A priority Critical patent/CN114677306B/en
Publication of CN114677306A publication Critical patent/CN114677306A/en
Application granted granted Critical
Publication of CN114677306B publication Critical patent/CN114677306B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a context aggregation image rain removing method based on edge information guidance, which aims to solve the problem that the rain removing method ignores texture information and edge information of an image at the present stage and is characterized in that a multi-scale information network is designed, wherein the multi-scale information network comprises an upper branch image rain removing network for acquiring rain removing information of a rough adjustment image and a lower branch edge information detection network for acquiring edge information of the image, and the multi-scale information network comprises a context aggregation module which is used for aggregating and processing the context information, guiding the rain removing information of the rough adjustment image by using the aggregated and processed information, and enhancing the representation capability of the upper branch image rain removing network on the detail information of the image. Experimental results show that the method can complete the rain removal of the image and enable the image to obtain richer texture information and edge information.

Description

Context aggregation image rain removing method based on edge information guidance
Technical Field
The invention belongs to the field of image processing and deep learning, and particularly relates to a context aggregation image rain removing method based on edge information guidance.
Background
The image edge is an important feature of an image, and is a discontinuity of distribution of characteristics (such as pixel gray scale, texture and the like) in the image, most information of the image is concentrated in the edge part of the image, and the edge structure and the characteristics of an image are often important parts for determining the image characteristics. At present, deep learning already obtains excellent performance on an image rain removing task, but edge information is often ignored in the process of removing rain from an image, and the edge information in the image is also removed in the process of removing rain stripes and raindrops, so that the original image cannot be completely restored due to the loss of some important edge information. Therefore, it is also important to recover the edge information while removing rain.
Most of the existing image rain removing methods usually omit the restoration of image edge information, or directly use a backbone network to process image rain removing and image detail restoration, although the image rain removing method based on deep learning is mature day by day, the important edge information restoration is still solved while the rain is removed. The invention provides a context aggregation image rain removing method based on edge information guiding, which designs a multi-scale information network, wherein the multi-scale information network comprises an upper branch image rain removing network for obtaining rain removing information of a coarse adjustment image and a lower branch edge information detection network for obtaining edge information of the image, and comprises a context aggregation module, the context aggregation module is used for aggregating and processing the context information, guiding the rain removing information of the coarse adjustment image by using the aggregated information, and enhancing the representation capability of the upper branch image rain removing network on the detail information of the image. Experimental results show that the method enables the image to obtain richer texture information and edge information while completing the rain removal of the image, and obtains richer edge information and better rain removal effect while the resolution is not lost.
Disclosure of Invention
The invention aims to provide a contextual image rain removing method based on edge information guidance, which can achieve the purpose of removing rain and simultaneously recovering edge information.
The technical solution for realizing the purpose of the invention is as follows: a contextual image rain removing method based on edge information guidance comprises the following steps:
step 1, selecting N images in Rain data of a Rain removing synthetic image Rain of Rain200L, wherein N is more than 100 and less than 10000, carrying out normalization processing, taking the images with uniform size, namely height multiplied by width of h multiplied by w as a training sample set S, and turning to step 2;
step 2, constructing a multi-scale information network, wherein the multi-scale information network comprises an encoder Enc _ P, a first decoder Dnc _ R, a second decoder Dnc _ E, an image output layer and three context aggregation modules EGCAkAnd k is 1,2,3, and the process goes to step 3;
step 3, training the multi-scale information network by using the training sample set S to obtain the trained multi-scale information network:
step 3-1, inputting the training sample set S into the encoder Enc _ P, extracting image characteristic information of the training sample set S, and correspondingly obtaining rain removal information and image edge information of a coarse-adjustment image by respectively utilizing the first decoder Dnc _ R and the second decoder Dnc _ E;
step 3-2, utilizing three context aggregation modules EGCAkPerforming context aggregation processing on the rain removing information and the image edge information of the coarse adjustment image to obtain aggregated information
Figure BDA0003570966620000021
Step 3-3, utilizing the information after the polymerization treatment
Figure BDA0003570966620000022
Guiding the rain removing information of the rough-adjusted image to obtain edge information-guided rain removing information of the image, and sending the edge information-guided rain removing information of the image into an image inputGoing out of the layer to obtain a rain removing image, further obtaining a trained multi-scale information network, and turning to step 4;
step 4, reselecting M images in Rain removing data of the Rain synthetic image Rain200L, wherein M is more than 100 and less than 10000, unifying the size of the images into hxw through normalization processing to form a test sample set T, and turning to step 5;
and step 5, inputting the rain-containing images in the test sample set T into the trained multi-scale information network to obtain a rain-removing image, so that the image has richer texture information and edge information while the rain-removing information is removed, and the rain-removing result is more realistic.
Compared with the prior art, the invention has the advantages that:
(1) the existing image rain removing method usually causes the loss of image details while removing information such as rain lines, raindrops and the like, so that the rain removing result graph has deviation with the original image.
(2) The existing image rain removing method directly uses a main network to process image rain removing and image detail repairing, and the result is poor. The invention utilizes the context aggregation module to aggregate and process the rain removing information of the rough adjustment image obtained by the rain removing network of the upper branch image and the edge information of the image obtained by the rain removing network of the lower branch, and utilizes the aggregated information to enhance the representation capability of the rain removing network of the upper branch image to the detail information of the image, thereby completing the rain removing of the image and simultaneously enabling the image to obtain richer texture information and edge information.
Drawings
FIG. 1 is a flowchart of a method for removing rain from a context aggregation image guided based on edge information according to the present invention.
FIG. 2 is a model diagram of a method for removing rain from a context aggregation image guided based on edge information according to the present invention.
FIG. 3 is a diagram of the results of comparison experiments of two semi-supervised image rain-removing algorithms SIRR and Syn2Real on a synthesized domain rain-containing image sample and the method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
With reference to fig. 1 and fig. 2, a method for removing rain based on context aggregation image guided by edge information includes the following steps:
step 1, selecting N images in a Rain database of a Rain removing synthetic image Rain200L, wherein N is more than 100 and less than 10000, carrying out normalization processing, taking the images with uniform size, namely height multiplied by width multiplied by h multiplied by w as a training sample set S, and turning to step 2.
Step 2, constructing a multi-scale information network, wherein the multi-scale information network comprises an encoder Enc _ P, a first decoder Dnc _ R, a second decoder Dnc _ E, an image output layer and three context aggregation modules EGCAkAnd k is 1,2 and 3, and the specific formula is as follows:
1) the encoder Enc _ P has four convolutional blocks, defined as E1、E2、E3、E4The encoder Enc _ P network is defined as follows:
Figure BDA0003570966620000031
wherein,
Figure BDA0003570966620000032
representing image feature information extracted via an mth convolution block in the encoder, S represents a training sample set input to the encoder,
Figure BDA0003570966620000033
the size of the image characteristic information, i.e. height x width x channel number, is hm×wm×cmWherein h ism＝h/2m-1，wm＝w/2m-1，cm＝32×2m-1。
2) The first decoder Dnc _ R includes three convolutional blocks, each defined as Dr1、Dr2、Dr3The image feature information extracted by the encoder Enc _ P is input into the first decoder Dnc _ R to obtain the coarse image degrain information, and the network of the first decoder Dnc _ R is defined as follows:
Figure BDA0003570966620000041
wherein,
Figure BDA0003570966620000042
representing the coarse image degrain information acquired after the ith convolution block of the first decoder Dnc _ R,
Figure BDA0003570966620000043
the size of the image characteristic information, i.e. height x width x channel number, is hi×wi×ciWherein h isi＝h/2i-1，wi＝w/2i-1，ci＝32×2i-1。
3) The second decoder Dnc _ E includes three convolution blocks, each defined as De1、De2、De3The image feature information extracted by the encoder Enc _ P is input to the second decoder Dnc _ E to obtain the image edge information, and the network of the second decoder Dnc _ E is defined as follows:
Figure BDA0003570966620000044
wherein,
Figure BDA0003570966620000045
representing the image edge information obtained via the jth convolutional block in the second decoder Dnc _ E,
Figure BDA0003570966620000046
the size of the image characteristic information, i.e. height x width x channel number, is hj×wj×cjWherein h isj＝h/2j-1，wj＝w/2j-1，cj＝32×2j-1。
4) Three context aggregation module EGCAkK is 1,2,3, i.e. EGCA1、EGCA2、EGCA3。
And (5) turning to the step 3.
Step 3, training the multi-scale information network by using the training sample set S to obtain the trained multi-scale information network:
step 3-1, inputting the training sample set S into the encoder Enc _ P, extracting image feature information of the training sample set S, and then correspondingly obtaining rain removal information and image edge information of the coarse-tuned image by using the first decoder Dnc _ R and the second decoder Dnc _ E, which are specifically as follows:
the encoder Enc _ P is used for extracting image feature information of a training sample set S, the encoder Enc _ P and the first decoder Dnc _ R jointly construct an upper branch image rain removal network for obtaining coarse adjustment image rain removal information, the encoder Enc _ P and the second decoder Dnc _ E jointly construct a lower branch edge information detection network for obtaining image edge information, the image feature information extracted by the same encoder aims at enabling an upper branch and a lower branch to share a weight, and the image rain removal and edge information detection process is facilitated:
the extraction process of the image characteristic information is specifically developed as follows:
when m is equal to 1, the compound is,
Figure BDA0003570966620000051
when m is equal to 2, the compound is,
Figure BDA0003570966620000052
when the m is equal to 3, the compound has the following characteristics,
Figure BDA0003570966620000053
when m is 4, the compound is shown in the specification,
Figure BDA0003570966620000054
wherein E ismDenotes the signal extracted via the mth convolution block in the encoderAnd e, operating, wherein m is 1,2,3 and 4.
The process of acquiring the rain removing information of the rough adjusted image by using the upper branch image rain removing network is specifically developed as follows:
when the value of i is 1, the value of i,
Figure BDA0003570966620000055
when the value of i is equal to 2,
Figure BDA0003570966620000056
when the value of i is 3, the value of i,
Figure BDA0003570966620000057
wherein D isri(. x) denotes the operation of obtaining the rain information of the coarse image by the i-th convolution block of the first decoder Dnc _ R, i being 1,2, 3.
The process of acquiring the image edge information by using the lower branch edge information detection network specifically expands as follows:
when j is equal to 1, the value of j,
Figure BDA0003570966620000058
when the j is 2, the sum of the j,
Figure BDA0003570966620000059
when j is 3, the number of the adjacent groups is 3,
Figure BDA00035709666200000510
wherein D isej(×) denotes an operation of acquiring image edge information via the jth convolution block in the second decoder Dnc _ E, where j is 1,2, 3.
Step 3-2, utilizing three context aggregation modules EGCAkPerforming context aggregation processing on the rain removing information and the image edge information of the coarse adjustment image to obtain aggregated information
Figure BDA00035709666200000511
The method comprises the following specific steps:
firstly, the size of the image characteristic information is hi×wi×ciCoarse adjustment of image rain removal information
Figure BDA00035709666200000512
And the size of the image characteristic information is hj×wj×cjImage edge information
Figure BDA00035709666200000513
Respectively carrying out three convolution kernels with the size of 1 multiplied by 1 to obtain three image characteristic information with the sizes of hi×wi×ci/2、hi×wi×ciH and 2j×wj×cj[ 2 ] of
Figure BDA00035709666200000514
Convolution of image information, i.e.
Figure BDA00035709666200000515
A second step of
Figure BDA0003570966620000061
The convolution image information is subjected to image characteristic recombination transformation to obtain the image characteristic information with the size of hi×wi×ciFirst recombined image information of/2
Figure BDA0003570966620000062
And image feature information size of cj/2×hj×wjSecond reconstructed image information of
Figure BDA0003570966620000063
Namely, it is
Figure BDA0003570966620000064
Thirdly, the first recombined image information is processed
Figure BDA0003570966620000065
And second reconstructed image information
Figure BDA0003570966620000066
Matrix multiplication is carried out to obtain preliminary image characteristic information Feture1kThe size of the image feature information is (h)i×wj)×(hj×wj) I.e. by
Figure BDA0003570966620000067
Figure BDA0003570966620000068
Represents a matrix multiplication;
fourthly, the preliminary image characteristic information Feture1 is processedkAfter being processed by the normalization layer, the mixture is mixed with
Figure BDA0003570966620000069
Matrix multiplication is carried out to obtain final image characteristic information Feture2kThe size of the image characteristic information is hi×wi×ci2, i.e. that
Figure BDA00035709666200000610
Fifthly, the final characteristic diagram information Feture2kAfter a convolution kernel with the size of 1 multiplied by 1, the information after the polymerization treatment is obtained
Figure BDA00035709666200000611
Image characteristic information size is hi×wi×ciI.e. by
Figure BDA00035709666200000612
Step 3-3, utilizing the information after the polymerization treatment
Figure BDA00035709666200000613
Guiding the rain removing information of the rough-adjusted image to obtain edge information guided image rain removing information, sending the edge information guided image rain removing information to an image output layer to obtain a rain removing image, and further obtaining a trained multi-scale information network, wherein the method specifically comprises the following steps:
defining guiding coarse adjustment image rain removal information as edge information guiding image rain removal information
Figure BDA00035709666200000614
Namely, it is
Figure BDA00035709666200000615
Wherein,
Figure BDA0003570966620000071
representing the rain removal information of the coarse image obtained by the ith convolution block in the first decoder,
Figure BDA0003570966620000072
representing the image edge information obtained by the jth convolutional block in the second decoder,
Figure BDA0003570966620000073
and the information aggregated by the k-th context aggregation module is represented.
Using information after aggregation processing
Figure BDA0003570966620000074
The specific process of guiding the rain removal information of the coarse adjustment image is as follows:
when k is equal to 1, the first step is carried out,
Figure BDA0003570966620000075
when k is equal to 2, the number of the bits is increased,
Figure BDA0003570966620000076
when k is 3, the number of the groups is 3,
Figure BDA0003570966620000077
then will be
Figure BDA0003570966620000078
And (4) obtaining a rain removing image after passing through the image output layer, further obtaining a trained multi-scale information network, and turning to the step 4.
And 4, reselecting M images in Rain removing data of the Rain200L synthetic image, wherein M is more than 100 and less than 10000, unifying the size of the images into h multiplied by w through normalization processing to form a test sample set T, and turning to the step 5.
And step 5, inputting the rain-containing images in the test sample set T into the trained multi-scale information network to obtain a rain-removing image, so that the image has richer texture information and edge information while the rain-removing information is removed, and the rain-removing result is more realistic.
Example 1
With reference to fig. 1 and fig. 2, a method for removing rain based on context aggregation image guided by edge information according to the present invention includes the following steps:
step 1, selecting 1800 images in Rain removing data of a Rain removing image synthesized by Rain200L, carrying out normalization processing, taking the images with the size of 256 multiplied by 256 as a training sample set S, and turning to step 2.
Step 2, constructing a multi-scale information network, wherein the multi-scale information network comprises an encoder Enc _ P, a first decoder Dnc _ R, a second decoder Dnc _ E, an image output layer and three context aggregation modules EGCA1、EGCA2、EGCA3The method comprises the following steps:
1) the encoder Enc _ P has four convolutional blocks, defined as E1、E2、E3、E4The encoder Enc _ P network is defined as follows:
Figure BDA0003570966620000081
wherein,
Figure BDA0003570966620000082
representing image feature information extracted via an mth convolution block in the encoder, S represents a training sample set input to the encoder,
Figure BDA0003570966620000083
has a size of 256 × 256 × 32,
Figure BDA0003570966620000084
has a size of 128 x 64,
Figure BDA0003570966620000085
has an image characteristic information size of 64 x 128,
Figure BDA0003570966620000086
has a size of 32 × 32 × 256.
2) The first decoder Dnc _ R includes three convolutional blocks, each defined as Dr1、Dr2、Dr3The image feature information extracted by the encoder Enc _ P is input into the first decoder Dnc _ R to obtain the coarse image degrain information, and the network of the first decoder Dnc _ R is defined as follows:
Figure BDA0003570966620000087
wherein,
Figure BDA0003570966620000088
representing the coarse image degrain information acquired after the ith convolution block of the first decoder Dnc _ R,
Figure BDA0003570966620000089
has an image characteristic information size of 256 × 256 × 32,
Figure BDA00035709666200000810
has a size of 128 x 64,
Figure BDA00035709666200000811
has a size of 64 × 64 × 128.
3) The second decoder Dnc _ E includes three convolution blocks, each defined as De1、De2、De3The image feature information extracted by the encoder Enc _ P is input to the second decoder Dnc _ E to obtain the image edge information, and the network of the second decoder Dnc _ E is defined as follows:
Figure BDA00035709666200000812
wherein,
Figure BDA00035709666200000813
representing the image edge information obtained by the jth convolutional block in the second decoder Dnc _ E,
Figure BDA00035709666200000814
has an image characteristic information size of 256 × 256 × 32,
Figure BDA00035709666200000815
has a size of 128 x 64,
Figure BDA00035709666200000816
has a size of 64 × 64 × 128.
4) Three context aggregation module EGCA1、EGCA2、EGCA3And (5) turning to the step 3.
Step 3, training the multi-scale information network by using the training sample set S to obtain the trained multi-scale information network:
step 3-1, inputting the training sample set S into the encoder Enc _ P, extracting image feature information of the training sample set S, and then correspondingly obtaining rain removal information and image edge information of the coarse-tuned image by using the first decoder Dnc _ R and the second decoder Dnc _ E, which are specifically as follows:
the encoder Enc _ P is used for extracting image characteristic information of the training sample set S
Figure BDA0003570966620000091
The encoder Enc _ P and the first decoder Dnc _ R together construct an upper branch image rain removal network for obtaining coarse image rain removal information
Figure BDA0003570966620000092
The encoder Enc _ P and the second decoder Dnc _ E together construct a lower branch edge information detection network for obtaining image edge information
Figure BDA0003570966620000093
The image characteristic information extracted by the same encoder aims to enable the upper branch and the lower branch to share the weight, and is more beneficial to the rain removal and edge information detection process of the image:
the extraction process of the image characteristic information is specifically developed as follows:
when m is equal to 1, the compound has the following structure,
Figure BDA0003570966620000094
when m is equal to 2, the compound is,
Figure BDA0003570966620000095
when m is 3, the compound is added,
Figure BDA0003570966620000096
when m is 4, the compound is shown in the specification,
Figure BDA0003570966620000097
the process of acquiring the rain removing information of the rough adjusted image by using the upper branch image rain removing network is specifically developed as follows:
when the value of i is 1, the value of i,
Figure BDA0003570966620000098
when the value of i is 2, the ratio of i to i is,
Figure BDA0003570966620000099
when the value of i is 3, the value of i,
Figure BDA00035709666200000910
the process of acquiring the image edge information by using the lower branch edge information detection network specifically expands as follows:
when j is equal to 1, the value of j,
Figure BDA00035709666200000911
when the j is 2, the sum of the j,
Figure BDA00035709666200000912
when j is 3, the number of the adjacent groups is 3,
Figure BDA00035709666200000913
step 3-2, utilizing three context aggregation modules EGCAkPerforming context aggregation processing on the rain removing information and the image edge information of the coarse adjustment image to obtain aggregated information
Figure BDA00035709666200000914
The method comprises the following specific steps:
first, image characteristic information is 256 × 256 × 32, rain information of image is coarsely adjusted, and image is subjected to rain removal
Figure BDA0003570966620000101
And image edge information having an image feature information size of 256 × 256 × 32
Figure BDA0003570966620000102
The number of image channels is reduced by three convolution kernels with the size of 1 multiplied by 1 respectively, and the three convolution image information with the image characteristic information size of 256 multiplied by 16, namely the three convolution image information
Figure BDA0003570966620000103
A second step of
Figure BDA0003570966620000104
The convolution image information is subjected to image characteristic reorganization transformation to obtain first reorganized image information with the image characteristic information size of 256 multiplied by 16
Figure BDA0003570966620000105
And second reconstructed image information having an image feature information size of 16 × 256 × 256
Figure BDA0003570966620000106
Namely, it is
Figure BDA0003570966620000107
Thirdly, the first recombined image information is processed
Figure BDA0003570966620000108
And second reconstructed image information
Figure BDA0003570966620000109
Matrix multiplication is carried out to obtain preliminary image characteristic information Feture1kThe image feature information size is (256 × 256) × (256 × 256), i.e.
Figure BDA00035709666200001010
Figure BDA00035709666200001011
Representing a matrix multiplication.
Fourthly, the preliminary image characteristic information Feture1 is processed1After being processed by the normalization layer, the mixture is mixed with
Figure BDA00035709666200001012
Matrix multiplication is carried out to obtain final image characteristic information Feture21The size of the image feature information is 256 × 256 × 16, i.e.
Figure BDA00035709666200001013
Fifthly, the final characteristic diagram information Feture21After a convolution kernel with the size of 1 multiplied by 1, the information after the polymerization treatment is obtained
Figure BDA00035709666200001014
The image characteristic information size is 256 × 256 × 32, i.e.
Figure BDA00035709666200001015
Similarly, information after polymerization treatment is obtained
Figure BDA00035709666200001016
Step 3-3, utilizing the information after the polymerization treatment
Figure BDA0003570966620000111
Removing rain information from coarse adjustment image
Figure BDA0003570966620000112
Guiding to obtain edge information guided image rain removal information
Figure BDA0003570966620000113
Sending the image rain removing information guided by the edge information into an image output layer to obtain a rain removing image, and further obtaining a trained multi-scale information network, wherein the rain removing information comprises the following specific steps:
defining guiding coarse adjustment image rain removing information as edge information guiding image rain removing information
Figure BDA0003570966620000114
Namely, it is
Figure BDA0003570966620000115
Wherein,
Figure BDA0003570966620000116
representing the rain removal information of the coarse image obtained by the ith convolution block in the first decoder,
Figure BDA0003570966620000117
representing the image edge information obtained by the jth convolutional block in the second decoder,
Figure BDA0003570966620000118
and the information aggregated by the k-th context aggregation module is represented.
Using information after aggregation processing
Figure BDA0003570966620000119
The specific process of guiding the rain removal information of the coarse adjustment image is as follows:
when k is equal to 1, the first step is carried out,
Figure BDA00035709666200001110
when the k is equal to 2, the reaction condition is as follows,
Figure BDA00035709666200001111
when k is 3, the number of the groups is 3,
Figure BDA00035709666200001112
then will be
Figure BDA00035709666200001113
And (4) obtaining a rain removing image after passing through the image output layer, further obtaining a trained multi-scale information network, and turning to the step 4.
And 4, reselecting 1400 images in the Rain removal database of the Rain synthesis image of Rain200L, unifying the sizes of the images into 256 multiplied by 256 through normalization processing to form a test sample set T, and turning to the step 5.
And 5, inputting the rain-containing images in the test sample set T into the trained multi-scale information network to obtain a rain-removing image, so that the image has richer texture information and edge information while the rain-removing information is removed, and the rain-removing result is more realistic.
The method of the invention adopts python programming language and tensoflow framework language to build a network framework on an Nvidia2080Ti GPU host computer to carry out relevant experiments. Firstly, an encoder Enc _ P, a first decoder Dnc _ R and a second decoder Dnc _ E of the multi-scale information network are trained, each convolutional layer uses a ReLU activation function, the learning rate of the network is set to be 2E-4The batch-size of the training encoder Enc _ P, the first decoder Dnc _ R, and the second decoder Dnc _ E is set to 3, and the training iterations 400 times. Then training a context aggregation module, using a ReLU activation function, using a sigmoid activation function in an SE attention mechanism, and setting the learning rate of the network to be 2e-4The batch-size is set to 2 and the training is iterated 400 times. In the network training process, the size of the input image is normalized to 256 × 256, and the whole rain-removing network model is obtained.
In order to better embody the effect of the algorithm proposed by the present invention on image rain removal, a model visualization experiment was designed according to example 1. And the rain removing effect of the image after each context aggregation module is visualized, and the rain removing expression of the image guided by the edge information every time is judged by vision. And experiments of two semi-supervised image rain removing algorithms SIRR and Syn2Real on a rain-containing image sample in a synthetic domain are also carried out, and the results of the experiments are compared with the experimental results of the invention to discover that the experimental results of the invention not only can obtain good image rain removing effect, but also can recover the detail information and edge information of the image.

Claims (6)

1. A context aggregation image rain removing method based on edge information guidance is characterized by comprising the following steps:
step 1, selecting a Rain database with a Rain image synthesized by Rain200L and N images in the Rain database, wherein N is more than 100 and less than 10000, carrying out normalization processing, taking the images with uniform size, namely height multiplied by width multiplied by h multiplied by w as a training sample set S, and turning to step 2;
step 2, constructing a multi-scale information network, wherein the multi-scale information network comprises an encoder Enc _ P, a first decoder Dnc _ R, a second decoder Dnc _ E, an image output layer and three context aggregation modules EGCAkAnd k is 1,2,3, and the process goes to step 3;
step 3, training the multi-scale information network by using the training sample set S to obtain the trained multi-scale information network:
step 3-1, inputting the training sample set S into an encoder Enc _ P, extracting image characteristic information of the training sample set S, correspondingly obtaining rain removing information and image edge information of a rough-adjusted image by respectively utilizing a first decoder Dnc _ R and a second decoder Dnc _ E, and turning to step 3-2;
step 3-2, utilizing three context aggregation modules EGCAkPerforming context aggregation processing on the rain removing information and the image edge information of the coarse adjustment image to obtain aggregated information
Figure FDA0003570966610000011
Turning to the step 3-3;
step 3-3, utilizing the information after the polymerization treatment
Figure FDA0003570966610000012
Guiding the rain removing information of the rough-adjusted image to obtain edge information-guided image rain removing information, sending the edge information-guided image rain removing information to an image output layer to obtain a rain removing image, further obtaining a trained multi-scale information network, and turning to step 4;
step 4, reselecting the Rain200L synthetic image to remove M images in the Rain database, wherein M is more than 100 and less than 10000, unifying the size of the images into hxw through normalization processing to form a test sample set T, and turning to step 5;
and step 5, inputting the rain-containing images in the test sample set T into the trained multi-scale information network to obtain a rain-removing image, so that the image has richer texture information and edge information while the rain-removing information is removed, and the rain-removing result is more realistic.
2. The method of claim 1, wherein in step 2, a multi-scale information network is constructed, the multi-scale information network comprises an encoder Enc _ P, a first decoder Dnc _ R, a second decoder Dnc _ E, an image output layer, and three context aggregation modules EGCAkThe context aggregation module serial number k is 1,2, and 3, which is specifically as follows:
1) the encoder Enc _ P has four convolutional blocks, defined as E1、E2、E3、E4The encoder Enc _ P network is defined as follows:
Figure FDA0003570966610000021
wherein,
Figure FDA0003570966610000022
representing image feature information extracted via an mth convolution block in the encoder, S represents a training sample set input to the encoder,
Figure FDA0003570966610000023
the size of the image characteristic information, i.e. height x width x channel number, is hm×wm×cmWherein h ism＝h/2m-1，wm＝w/2m-1，cm＝32×2m-1；
2) The first decoder Dnc _ R includes three convolutional blocks, each defined as Dr1、Dr2、Dr3The image feature information extracted by the encoder Enc _ P is input into the first decoder Dnc _ R to obtain the coarse image degrain information, and the network of the first decoder Dnc _ R is defined as follows:
Figure FDA0003570966610000024
wherein,
Figure FDA0003570966610000025
representing the coarse image degrain information acquired after the ith convolution block of the first decoder Dnc _ R,
Figure FDA0003570966610000026
the size of the image characteristic information, i.e. height x width x channel number, is hi×wi×ciWherein h isi＝h/2i-1，wi＝w/2i-1，ci＝32×2i-1；
3) The second decoder Dnc _ E includes three convolutional blocks, each defined as De1、De2、De3The image feature information extracted by the encoder Enc _ P is input to the second decoder Dnc _ E to obtain the image edge information, and the network of the second decoder Dnc _ E is defined as follows:
Figure FDA0003570966610000027
wherein,
Figure FDA0003570966610000028
representing the image edge information obtained via the jth convolutional block in the second decoder Dnc _ E,
Figure FDA0003570966610000029
the size of the image characteristic information, i.e. height x width x channel number, is hj×wj×cjWherein h isj＝h/2j-1，wj＝w/2j-1，cj＝32×2j-1；
4) Three context aggregation module EGCAkK is 1,2,3, i.e. EGCA1、EGCA2、EGCA3。
3. The method according to claim 2, wherein h is hm＝hi＝hj，wm＝wi＝wj，cm＝ci＝cj。
4. The method as claimed in claim 2, wherein in step 3-1, the training sample set S is input to the encoder Enc _ P, the image feature information is extracted, and then the first decoder Dnc _ R and the second decoder Dnc _ E are respectively used to obtain the image edge information and the coarse-adjustment image rain removal information, which are as follows:
the encoder Enc _ P is configured to extract image feature information of the training sample set S, the encoder Enc _ P and the first decoder Dnc _ R together construct an upper branch image rain removal network for obtaining coarse image rain removal information, and the encoder Dnc _ P and the second decoder Dnc _ E together construct a lower branch edge information detection network for obtaining image edge information:
the extraction process of the image characteristic information is specifically developed as follows:
when m is equal to 1, the compound is,
Figure FDA0003570966610000031
when m is equal to 2, the compound is,
Figure FDA0003570966610000032
when m is 3, the compound is added,
Figure FDA0003570966610000033
when m is 4, the compound is shown in the specification,
Figure FDA0003570966610000034
wherein E ism(indicates the block extraction by the mth convolution block in the encoderTaking information, wherein m is 1,2,3 and 4;
the process of acquiring the rain removing information of the rough adjusted image by using the upper branch image rain removing network is specifically developed as follows:
when the value of i is 1, the value of i,
Figure FDA0003570966610000035
when the value of i is 2, the ratio of i to i is,
Figure FDA0003570966610000036
when the value of i is 3, the value of i,
Figure FDA0003570966610000037
wherein D isri(xvi) denotes an operation of obtaining a coarse image degrain information via the i-th convolution block of the first decoder Dnc _ R, i ═ 1,2, 3;
the process of acquiring the image edge information by using the lower branch edge information detection network specifically expands as follows:
when j is equal to 1, the value of j,
Figure FDA0003570966610000038
when the j is 2, the sum of the j,
Figure FDA0003570966610000039
when j is 3, the number of the adjacent groups is 3,
Figure FDA00035709666100000310
wherein D isej(×) denotes an operation of acquiring image edge information via the jth convolution block in the second decoder Dnc _ E, where j is 1,2, 3.
5. The method as claimed in claim 4, wherein in step 3-2, three context aggregation modules EGCA are usedkPerforming context aggregation processing on the rain removing information and the image edge information of the rough-adjusted image to obtain aggregated information, which is specifically as follows:
firstly, the size of the image characteristic information is hi×wi×ciCoarse adjustment of image rain removal information
Figure FDA0003570966610000041
And the size of the image characteristic information is hj×wj×cjImage edge information
Figure FDA0003570966610000042
Respectively carrying out three convolution kernels with the size of 1 multiplied by 1 to obtain three image characteristic information with the sizes of hi×wi×ci/2、hi×wi×ciH and/2j×wj×cj[ 2 ] of
Figure FDA0003570966610000043
Convolution of image information, i.e.
Figure FDA0003570966610000044
A second step of
Figure FDA0003570966610000045
The convolution image information is subjected to image characteristic recombination transformation to obtain image characteristic information with the size of hi×wi×ciFirst reconstructed image information of/2
Figure FDA0003570966610000046
And an image characteristic information size of cj/2×hj×wjSecond reconstructed image information of
Figure FDA0003570966610000047
Namely, it is
Figure FDA0003570966610000048
Thirdly, the first recombined image information is processed
Figure FDA0003570966610000049
And second recombined image information
Figure FDA00035709666100000410
Matrix multiplication is carried out to obtain preliminary image characteristic information Feture1kAnd image characteristic information size is (h)i×wi)×(hj×wj) I.e. by
Figure FDA00035709666100000411
Figure FDA00035709666100000412
Represents a matrix multiplication;
fourthly, the preliminary image characteristic information Feture1 is processedkAfter being processed by the normalization layer, the mixture is mixed with
Figure FDA00035709666100000413
Matrix multiplication is carried out to obtain final image characteristic information Feture2kThe size of the image characteristic information is hi×wi×ci2, i.e. that
Figure FDA00035709666100000414
Fifthly, the final characteristic diagram information Feture2kAfter a convolution kernel with the size of 1 multiplied by 1, the information after the polymerization treatment is obtained
Figure FDA0003570966610000051
Image featureSign information of size hi×wi×ciI.e. by
Figure FDA0003570966610000052
6. The method for removing rain based on context aggregation image guided by edge information as claimed in claim 5, wherein the information after aggregation processing is utilized in step 3-3
Figure FDA0003570966610000053
Guiding the rain removal information of the rough-adjusted image, specifically as follows:
defining guiding coarse adjustment image rain removing information as edge information guiding image rain removing information
Figure FDA0003570966610000054
Namely, it is
Figure FDA0003570966610000055
Wherein,
Figure FDA0003570966610000056
representing the rain removal information of the coarse image obtained by the ith convolution block in the first decoder,
Figure FDA0003570966610000057
representing the image edge information obtained by the jth convolutional block in the second decoder,
Figure FDA0003570966610000058
representing the information aggregated by the k context aggregation module;
using information after aggregation processing
Figure FDA0003570966610000059
The process of guiding the rain removal information of the coarse adjustment image is specifically developed as follows:
when k is equal to 1, the first step is carried out,
Figure FDA00035709666100000510
when k is equal to 2, the number of the bits is increased,
Figure FDA00035709666100000511
when k is 3, the number of the groups is 3,
Figure FDA00035709666100000512
then will be
Figure FDA00035709666100000513
And obtaining a rain removing image after passing through the image output layer, and further obtaining a trained multi-scale information network.
CN202210319123.6A 2022-03-29 2022-03-29 Context aggregation image rain removing method based on edge information guidance Active CN114677306B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210319123.6A CN114677306B (en) 2022-03-29 2022-03-29 Context aggregation image rain removing method based on edge information guidance

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210319123.6A CN114677306B (en) 2022-03-29 2022-03-29 Context aggregation image rain removing method based on edge information guidance

Publications (2)

Publication Number Publication Date
CN114677306A true CN114677306A (en) 2022-06-28
CN114677306B CN114677306B (en) 2022-11-15

Family

ID=82076161

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210319123.6A Active CN114677306B (en) 2022-03-29 2022-03-29 Context aggregation image rain removing method based on edge information guidance

Country Status (1)

Country Link
CN (1) CN114677306B (en)

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108765344A (en) * 2018-05-30 2018-11-06 南京信息工程大学 A method of the single image rain line removal based on depth convolutional neural networks
US10482603B1 (en) * 2019-06-25 2019-11-19 Artificial Intelligence, Ltd. Medical image segmentation using an integrated edge guidance module and object segmentation network
CN111553851A (en) * 2020-04-08 2020-08-18 大连理工大学 Video rain removing method based on time domain rain line decomposition and spatial structure guidance
CN112070690A (en) * 2020-08-25 2020-12-11 西安理工大学 Single image rain removing method based on convolutional neural network double-branch attention generation
CN112070687A (en) * 2020-08-20 2020-12-11 武汉大学 Image rain removing method and system based on team recursive feedback mechanism
CN112184573A (en) * 2020-09-15 2021-01-05 西安理工大学 Context aggregation residual single image rain removing method based on convolutional neural network
CN112347859A (en) * 2020-10-15 2021-02-09 北京交通大学 Optical remote sensing image saliency target detection method
US20210150747A1 (en) * 2019-11-14 2021-05-20 Samsung Electronics Co., Ltd. Depth image generation method and device
CN113240613A (en) * 2021-06-07 2021-08-10 北京航空航天大学 Image restoration method based on edge information reconstruction
CN113408398A (en) * 2021-06-16 2021-09-17 西安电子科技大学 Remote sensing image cloud detection method based on channel attention and probability up-sampling
CN113450278A (en) * 2021-06-30 2021-09-28 中国矿业大学 Image rain removing method based on cross-domain collaborative learning
CN113673590A (en) * 2021-08-13 2021-11-19 广东工业大学 Rain removing method, system and medium based on multi-scale hourglass dense connection network
CN113962905A (en) * 2021-12-03 2022-01-21 四川大学 Single image rain removing method based on multi-stage feature complementary network
CN114187275A (en) * 2021-12-13 2022-03-15 贵州大学 Multi-stage and multi-scale attention fusion network and image rain removing method

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108765344A (en) * 2018-05-30 2018-11-06 南京信息工程大学 A method of the single image rain line removal based on depth convolutional neural networks
US10482603B1 (en) * 2019-06-25 2019-11-19 Artificial Intelligence, Ltd. Medical image segmentation using an integrated edge guidance module and object segmentation network
US20210150747A1 (en) * 2019-11-14 2021-05-20 Samsung Electronics Co., Ltd. Depth image generation method and device
CN111553851A (en) * 2020-04-08 2020-08-18 大连理工大学 Video rain removing method based on time domain rain line decomposition and spatial structure guidance
CN112070687A (en) * 2020-08-20 2020-12-11 武汉大学 Image rain removing method and system based on team recursive feedback mechanism
CN112070690A (en) * 2020-08-25 2020-12-11 西安理工大学 Single image rain removing method based on convolutional neural network double-branch attention generation
CN112184573A (en) * 2020-09-15 2021-01-05 西安理工大学 Context aggregation residual single image rain removing method based on convolutional neural network
CN112347859A (en) * 2020-10-15 2021-02-09 北京交通大学 Optical remote sensing image saliency target detection method
CN113240613A (en) * 2021-06-07 2021-08-10 北京航空航天大学 Image restoration method based on edge information reconstruction
CN113408398A (en) * 2021-06-16 2021-09-17 西安电子科技大学 Remote sensing image cloud detection method based on channel attention and probability up-sampling
CN113450278A (en) * 2021-06-30 2021-09-28 中国矿业大学 Image rain removing method based on cross-domain collaborative learning
CN113673590A (en) * 2021-08-13 2021-11-19 广东工业大学 Rain removing method, system and medium based on multi-scale hourglass dense connection network
CN113962905A (en) * 2021-12-03 2022-01-21 四川大学 Single image rain removing method based on multi-stage feature complementary network
CN114187275A (en) * 2021-12-13 2022-03-15 贵州大学 Multi-stage and multi-scale attention fusion network and image rain removing method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
YUXIA LI等: ""Road segmentation of unmanned aerial vehicle remote sensing images using adversarial network with multiscale context aggregation"", 《IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTES SENSING》 *
朱德利等: ""基于注意力生成对抗网络的单幅图像去雨方法"", 《计算机工程与应用》 *

Also Published As

Publication number Publication date
CN114677306B (en) 2022-11-15

Similar Documents

Publication Publication Date Title
CN108510485B (en) Non-reference image quality evaluation method based on convolutional neural network
CN111047541B (en) Image restoration method based on wavelet transformation attention model
CN112508083B (en) Image rain and fog removing method based on unsupervised attention mechanism
CN109903236B (en) Face image restoration method and device based on VAE-GAN and similar block search
CN110458085B (en) Video behavior identification method based on attention-enhanced three-dimensional space-time representation learning
CN112184582B (en) Attention mechanism-based image completion method and device
CN112489164B (en) Image coloring method based on improved depth separable convolutional neural network
CN113298734B (en) Image restoration method and system based on mixed hole convolution
CN112967210B (en) Unmanned aerial vehicle image denoising method based on full convolution twin network
Song et al. A new recurrent plug-and-play prior based on the multiple self-similarity network
CN116168197A (en) Image segmentation method based on Transformer segmentation network and regularization training
CN113962905A (en) Single image rain removing method based on multi-stage feature complementary network
CN108090914B (en) Color image segmentation method based on statistical modeling and pixel classification
CN114677306B (en) Context aggregation image rain removing method based on edge information guidance
CN111814884A (en) Target detection network model upgrading method based on deformable convolution
CN111199288A (en) Novel multi-head attention mechanism
CN116883265A (en) Image deblurring method based on enhanced feature fusion mechanism
CN113034390B (en) Image restoration method and system based on wavelet prior attention
CN114638845A (en) Quantum image segmentation method and device based on double thresholds and storage medium
CN104574320B (en) A kind of image super-resolution restored method based on sparse coding coefficients match
CN114841895A (en) Image shadow removing method based on bidirectional mapping network
CN113781333A (en) Method for processing underwater image by GAN network based on guided filtering
CN110288525B (en) Multi-dictionary super-resolution image reconstruction method
Liu et al. Graph representation learning for spatial image steganalysis
Zhang et al. Scale-progressive multi-patch network for image dehazing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant