CN113935916A - End-to-end underwater image restoration method based on ambient light perception - Google Patents

End-to-end underwater image restoration method based on ambient light perception Download PDF

Info

Publication number
CN113935916A
CN113935916A CN202111195071.8A CN202111195071A CN113935916A CN 113935916 A CN113935916 A CN 113935916A CN 202111195071 A CN202111195071 A CN 202111195071A CN 113935916 A CN113935916 A CN 113935916A
Authority
CN
China
Prior art keywords
ambient light
network
layer
image
underwater
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111195071.8A
Other languages
Chinese (zh)
Other versions
CN113935916B (en
Inventor
王柯俨
黄诗芮
陈静怡
李娇娇
李云松
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202111195071.8A priority Critical patent/CN113935916B/en
Publication of CN113935916A publication Critical patent/CN113935916A/en
Application granted granted Critical
Publication of CN113935916B publication Critical patent/CN113935916B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/30—Assessment of water resources

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses an end-to-end underwater image restoration method based on ambient light perception, which mainly solves the problem that the color cast correction and the clarification processing effect are poor when underwater images are processed in the prior art. The scheme is as follows: respectively constructing an ambient light sensing network and a restoration main network by using a Pythrch frame, and respectively constructing training sets B and C of the two networks; training an ambient light perception network and a restoration main body network by respectively using B and C through a self-adaptive moment estimation algorithm to obtain an image I to be processedcInputting the trained ambient light sensing network and outputting an ambient light value Ac(ii) a A is to becAnd IcInputting the trained restoration main network and outputting a clear image Jc. The invention improves the contrast of underwater images with different degradation degrees, can effectively correct color cast, and has peak signal-to-noise ratio and junctionThe structure similarity, the color difference formula, the no-reference image space quality evaluation and the underwater color image quality evaluation are all superior to the prior art, and can be used for the clarification processing of underwater images.

Description

End-to-end underwater image restoration method based on ambient light perception
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an underwater image restoration method which can be used for processing a single underwater image shot by an imaging system.
Background
Under the influence of water on light absorption and scattering effects under practical conditions, the underwater optical image generally has the problems of low contrast, color distortion and image blurring quality degradation. The degraded images not only affect the subjective feeling of human eyes, but also severely restrict the performance of various intelligent visual information processing systems. Therefore, the reconstruction of clear underwater optical images has very important practical application value. At present, the key problem of the underwater image processing method is how to improve the image definition and correct color cast, and the method is mainly divided into four types of methods, namely traditional image enhancement, traditional image restoration, deep learning image restoration and deep learning image enhancement, wherein:
the traditional image enhancement method selects a proper image enhancement technology to improve the image quality aiming at the concrete performance of the underwater image degradation. Typical methods such as Retinex and histogram equalization based Underwater Image enhancement algorithms proposed by Zhang et al, see Zhang W, Li G, Ying Z, and et al.A New oil Image Enhancing Method video Color Correction and Illumination addition [ C ]// IEEE International Conference on Visual communication and Image processing.2017, DOI 10.1109/VCIP.2017.8305027; an underwater image enhancement algorithm based on a fusion strategy proposed by Ancuti et al, see C.Ancuti, C.O.Ancuti, T.Haber, and P.Bekaert.enhancing underserver images and video by fusion [ C ]// IEEE Conference on Computer Vision and Pattern registration.2012, DOI: 10.1109/CVPR.2012.62661. The method has simple principle and can effectively improve the visual effect of the image, but because the degradation principle of the underwater image is not considered, the relationship between the degradation degree and the depth is ignored, and the enhanced result cannot correctly reflect the real color of the image.
The traditional image restoration method is based on a physical imaging model of an underwater degraded image, extracts image characteristics by using different priors or assumptions, and then designs effective ambient light and transmittance estimation methods by using the characteristics respectively to realize image restoration. Typical methods such as Galdran et al propose Red Channel prior, which is replaced by the difference between 1 and the Red Channel value when calculating the dark Channel, see Galdran A, Alvarez-Gila A. automatic Red-Channel underserver Image retrieval [ J ]. Journal of Visual Communication & Image retrieval, 2015,26(C):132-145. although this method improves the effect of applying the dark Channel prior to the underwater Image, it violates the original statistical significance of the dark Channel and reduces the effectiveness of the prior in the clear Image; li et al estimate transmittance by a method of reducing red channel information loss and estimate ambient light values by virtue of characteristics of high brightness and large red and blue channel differences, see Li C, Guo J, Cong R, et al. The image restoration effect of the method highly depends on the prior reliability, and a large estimation error occurs under the condition of failure of the prior, so that the image restoration effect is poor.
The deep learning image restoration method is based on a physical imaging model and utilizes a neural network to automatically learn the nonlinear mapping relation between an underwater degraded image and imaging model parameters. Hou et al propose to estimate the transmittance of underwater images and ambient light using a residual convolutional neural network URCNN, balance underwater lighting using a scene residual calculation method, see Hou M, Liu R, Fan X and Luo Z. Joint residual learning for underserver Image enhancement [ C ]// IEEE International Conference on Image Processing (ICIP): 2018: 4043-. Wang et al propose a method of estimating transmittance and ambient light respectively using Convolutional Neural networks in Parallel, and then obtaining a clear underwater Image by means of an underwater imaging model, see Wang K, Hu Y, Chen J, Wu X, ZHao X, Li Y. lower water Image retrieval Based on a Parallel Neural Network [ J ]// Remote sensing.2019; 1951, the restored image of the network is more clear and natural, but the color cast cannot be eliminated when the image with high degradation degree is acted on, and the network model cannot avoid error transmission, so that the small model parameter estimation error can be amplified in the restoration process, and the restoration result has larger deviation.
The deep learning image enhancement method is to directly learn the mapping relation between the underwater degraded image and the corresponding clear image end to end without a physical model. Li et al propose a WaterNet network by taking the idea of image fusion as a reference. Firstly, respectively carrying out white balance, histogram equalization and gamma correction on an original Image, inputting the processed Image into a network, learning three confidence maps by utilizing the network, and weighting and fusing the three confidence maps and the Image to obtain an enhanced underwater Image, which is shown in Li C, Guo C, Ren W, and et al.an underserver Image enhancement performance benchmark dataset and beyond [ J ]// IEEE Transactions on Image processing.2020: 4376-; islam et al propose a FUnIE-GAN network to enhance underwater images in real time, see Islam M J, Xia Y and Sattar J.fast underserver image enhancement for improved visual performance [ J ]// IEEE Robotics and Automation letters.2020: 3227-. The method has the advantages that the network design is complex, the learning difficulty is high, the problem of difficulty in fitting is easy to occur, the method depends on the distribution of training data seriously, the generalization capability is insufficient, and the recovery effect on the seriously degraded underwater image is poor.
Disclosure of Invention
The invention aims to provide an underwater image restoration method based on ambient light perception to restore an underwater image end to end by using a convolutional neural network, so that the color distortion and the image blur of the underwater image with different degradation degrees are reduced, and the image restoration effect is improved.
In order to achieve the purpose, the technical scheme of the invention comprises the following steps:
1) constructing an underwater image restoration network ECCN under a Pythrch framework:
1a) establishing an ambient light sensing network comprising eleven convolutional layers, an upper sampling layer and a pooling layer;
1b) establishing a restoration main body network comprising a shallow layer feature extraction module, a multi-scale coding and decoding module and three feature integration reconstruction modules;
1c) cascading an ambient light sensing network and a restoration main network to form an underwater image restoration network;
2) downloading an indoor data set NYU containing depth information, a real single-image defogging data set RESIDE containing depth information and a paired underwater data set EUVP synthesized based on a generation network, and respectively carrying out screening, scaling and cutting on the data sets in sequence to obtain an indoor clear image set J, an indoor depth image set D, an underwater clear image set U and an underwater degraded image set R1;
3) respectively constructing training image sets of an ambient light perception network and a restoration main network:
setting ambient light value A and blue channel transmittance TbSynthesizing an underwater degraded image set R2 by using an indoor clear image set J and an indoor depth image set D, and pairing the underwater degraded image set R2 with an ambient light value A corresponding to the underwater degraded image set to be used as a training image set B of an ambient light sensing network;
matching the indoor clear image set J with the underwater degraded image set R2, adding the underwater clear image set U and the underwater degraded image set R1 into the indoor clear image set J and the underwater degraded image set U to serve as a training image set C of a recovery main network;
4) training the ambient light sensing network by utilizing a training image set B of the ambient light sensing network and adopting a self-adaptive moment estimation algorithm with a minimized Euclidean distance loss value as a target to obtain a trained ambient light sensing network;
5) training the restoration main network by using a training image set C of the restoration main network and a trained ambient light estimation network and adopting an adaptive moment estimation algorithm with a minimum weighted total loss value as a target to obtain the trained restoration main network;
6) an underwater image I needing restoration processingcInputting the light into a trained ambient light sensing network, and outputting an ambient light value A of RGB three channelsc；
7) The ambient light value AcAnd underwater image IcInputting the two into a trained restoration main network, and outputting a restored clear image Jc。
Compared with the prior art, the invention has the following beneficial effects:
firstly, the invention can lead the restoration results of underwater images with different degradation degrees to have more natural and accurate color presentation by sensing the prior information of the environment light related to the color through a simple regression network and assisting the training and restoration of a main network;
secondly, the restoration main network of the invention adopts a multi-scale coding and decoding structure, integrates the characteristic information of three scales from coarse to fine, and can restore the global structure and the local details of the degraded image; meanwhile, because a residual error module is introduced into the coding and decoding structure, the information transmission efficiency of each layer of the network in the training process is improved, the training error of the network is reduced, and the network is more sensitive to detail transformation;
thirdly, the invention designs a weighted loss training network, which comprehensively considers the factors of global structure, local texture and color, so that the detail information and edge texture information of the restored image are better retained, and the color restoration is more accurate.
Simulation results show that the method can better correct the color cast of the image and improve the visual effect on the premise of keeping the contrast of the restored image; compared with other existing algorithms, the processing result has the advantages of clear boundary, more natural color and higher overall image quality, and objective indexes in peak signal-to-noise ratio (PSNR), Structural Similarity (SSIM) and color difference formula CIEDE2000 are superior to those in the prior art.
Drawings
FIG. 1 is a general flow chart of an implementation of the present invention;
FIG. 2 is a structural diagram of an ambient light sensing network constructed in the present invention;
fig. 3 is a structural diagram of a restoration host network constructed in the present invention;
FIG. 4 is a graph comparing the processing effect of the present invention and the existing underwater image processing algorithm on underwater simulation images;
fig. 5 is a comparison graph of the processing effect of the real underwater image by using the underwater image processing algorithm of the present invention and the existing underwater image processing algorithm.
Detailed Description
The following describes the embodiments and effects of the present invention with reference to the drawings.
Referring to fig. 1, the specific implementation steps of the present invention are as follows:
step 1: and constructing an environment light sensing network under a Pythrch framework.
As shown in fig. 2, the ambient light sensing network constructed by the present invention includes eleven convolutional layers, an upsampling layer and a pooling layer, and has the structural relationship: the first buildup layer → the second buildup layer → the third buildup layer → the fourth buildup layer → the upsampling layer → the fifth buildup layer → the sixth buildup layer → the seventh buildup layer → the eighth buildup layer → the pooling layer → the ninth buildup layer → the tenth buildup layer → the eleventh buildup layer;
the parameters of each layer are set as follows:
the convolution kernels of the first convolution layer and the second convolution layer are both 9 x 9,
the convolution kernels of the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer are all 7 x 7,
the convolution kernels of the seventh convolution layer, the eighth convolution layer, the ninth convolution layer and the tenth convolution layer are all 5 x 5,
the convolution kernel size of the eleventh convolution layer is 3 x 3,
the step size of all convolutional layers is 1, all convolutional layers include convolution operation and the ReLU activation function layer,
the up-sampling layer has a scaling factor of 2,
the scaling factor of the pooling layer is 2.
Step 2: and constructing a restoration main network under a Pythrch framework.
As shown in fig. 3, the restoration host network constructed in this example includes a shallow feature extraction module, a multi-scale encoding and decoding module, and three feature integration and reconstruction modules, where:
the shallow layer feature extraction module is formed by cascading a1 st convolution layer → a2 nd convolution layer → a3 rd convolution layer;
the multi-scale coding and decoding module is formed by connecting three branches in parallel, wherein:
the first branch is in turn: the 1 st pooling layer → the 4 th convolution layer → the 1 st encoding unit → the 2 nd encoding unit → the 1 st decoding unit → the 2 nd decoding unit;
the second branch is sequentially as follows: the 2 nd pooling layer → the 5 th convolution layer → the 3 rd encoding unit → the 4 th encoding unit → the 3 rd decoding unit → the 4 th decoding unit; the 3 rd decoding unit is simultaneously connected with the 2 nd encoding unit;
the third branch is sequentially as follows: the 3 rd pooling layer → the 6 th convolution layer → the 5 th encoding unit → the 6 th encoding unit → the 5 th decoding unit → the 6 th decoding unit; the 5 th decoding unit is simultaneously connected with the 4 th encoding unit;
the 1 st, 2 nd and 3 rd pooling layers are connected with the 3 rd convolution layer;
each coding unit is sequentially as follows: convolution layer I → residual module II → residual module III;
each decoding unit is sequentially as follows: the IV residual module → the V residual module → the VI residual module → bilinear interpolation → the II convolutional layer;
the characteristic integration and reconstruction module sequentially comprises: an upsampling layer → a first convolution layer → a second convolution layer → a third convolution layer;
the connection relationship of the modules is as follows:
a2 nd decoding unit, a4 th decoding unit and a 6 th decoding unit in the multi-scale coding and decoding module are respectively connected with the three feature reconstruction modules, and a3 rd convolution layer of the shallow feature extraction module is simultaneously connected with a first convolution layer of the feature reconstruction module;
the output of the main body network is recovered to be the tail end output of the characteristic reconstruction module connected with the third branch;
the parameters of each layer are set as follows:
the convolution kernel size of the 1 st convolution layer is 7 x 7, the convolution kernel size of the 2 nd convolution layer is 5 x 5, and the convolution kernel sizes of the other convolution layers are 3 x 3; the convolution step length of all convolution layers is 1, and all convolution layers comprise convolution operation and a ReLU activation function layer;
the scaling factor of the upsampling layer is 2;
the scaling factor for pooling level 1 is 8,
the scaling factor for the 2 nd pooling layer is 4,
the scaling factor of the 3 rd pooling layer is 2;
the scaling factor of the bilinear interpolation operation is 2.
And step 3: and respectively constructing a training image set of the ambient light perception network and the restoration main body network.
3.1) downloading an indoor data set NYU containing depth information, a real single-image defogging data set RESIDE containing depth information and a paired underwater data set EUVP synthesized based on a generation network from the Internet, respectively carrying out screening, scaling and cutting on the data sets in sequence, and unifying the sizes to be 160 x 160 to obtain an indoor clear image set J, an indoor depth image set D, an underwater clear image set U and an underwater degraded image set R1;
3.2) randomly generating a red channel ambient light value A between 0.1 and 0.6 using random functionrUsing random function at ArRandomly generating a green channel ambient light value A between-1.0gAnd the ambient light value A of the blue channelb；
3.3) randomly generating a blue channel transmittance parameter β between 0.5 and 2.5 using random functionbCalculating the transmittance of blue channel of each image by using the indoor depth map set D
Figure BDA0003300286340000061
Red channel transmission
Figure BDA0003300286340000062
Green channel transmission
Figure BDA0003300286340000063
3.4) calculating an underwater degraded image set R2 (JT + A (1-T)) according to the ambient light value A and the transmissivity T generated by the indoor clear image set J and 3.2);
3.5) the size of the underwater degraded image set R2 is scaled to 49 multiplied by 49, and then the underwater degraded image set R2 is matched with the ambient light value A to be used as a training image set B of the ambient light sensing network;
3.6) matching the indoor clear image set J with the underwater degraded image set R2, and adding the underwater clear image set U and the underwater degraded image set R1 into the matched image set to be used as a training image set C of the restoration main network.
And 4, step 4: and training the neural network.
4a) Training an ambient light perception network:
4a1) taking the Euclidean distance formula as a loss function of the ambient light sensing network:
Figure BDA0003300286340000064
wherein | |2For the operation of two norms on the matrix, c is the color channel of the input image, BcIs the output of the ambient light aware network, B'cCorresponding artificially synthesized ambient light;
4a2) sequentially inputting the underwater degraded images in the training image set B of the ambient light perception network into the ambient light perception network, and outputting an estimated ambient light value A';
4a3) simultaneously substituting the set ambient light value A and the estimated ambient light value A 'into an Euclidean distance formula, and calculating to obtain an Euclidean distance loss value between A and A';
4a4) updating the weight and the offset value of each convolution operation in the ambient light perception network by using a self-adaptive moment estimation algorithm;
4a5) repeating the steps 4a2) -4a4)10000 times until the Euclidean distance loss value is minimum, and obtaining the trained ambient light perception network.
4b) Training a recovery subject network:
3b1) the weighted total loss formula is used as a loss function LossC of the restoration subject network, and is expressed as follows:
LossC＝0.75×Losst+0.25×Lossc
therein, LosstIs the boundary maintenance Loss, LosscIs the color shift correction loss, the calculation formula is:
Figure BDA0003300286340000071
Figure BDA0003300286340000072
wherein | |2To solve the two norm operation on the matrix, |. i is to solve the one norm operation on the matrix, | N is the number of pixels of the input image, Jn、Jn1、Jn2The three characteristics of the restoration main network are respectively integrated and rebuilt the tail end output, J'n，J'n1，J'n2Is a corresponding clear image, J'n1Is J 'reduced by 16 times'n，J'n2Is J 'reduced by 64 times'n；
Figure BDA0003300286340000073
Represents the mean value of the red channel, Δ rn、Δbn、ΔgnRespectively representing the difference between the network output and the clear image in red, green and blue channels;
4b2) sequentially inputting the underwater degraded images in the recovered main network training set C into a trained ambient light estimation network, and outputting an estimated ambient light value A ";
4b3) combining the underwater degraded image in the recovered main network training set C with the estimated ambient light value A ', inputting the combined image into the recovered main network, and outputting an estimated recovered image I';
4b4) simultaneously substituting the clear image I and the estimated restoration image I 'in the restoration main body network training set C into a weighted total loss function LossC, and calculating to obtain a weighted total loss value between the clear image I and the restoration image I';
4b5) updating the weight and the offset value of each convolution operation in the restoration main network by using a self-adaptive moment estimation algorithm;
4b6) repeat 4b3) -4b5) for a total of 20000 times until the weighted total loss value is minimized, resulting in a trained restoration subject network.
And 5: and restoring the image.
5a) An underwater image I needing restoration processingcInputting the data into the ambient light sensing network trained in the step 4a),output ambient light value Ac；
5b) Underwater image IcAnd an ambient light value AcCombining the input to the recovery main network trained in the step 4b) and outputting a clear image J with high qualitycAnd c belongs to { r, g, b }, and the recovery of the underwater degraded image is completed.
The effects of the present invention are further illustrated by the following simulations:
test condition and method
1. Testing pictures: selecting 406 groups of underwater simulation images and 126 underwater real images with different degradation degrees;
2. the test method comprises the following steps: nine methods, existing Anwar's algorithm, Zhang's algorithm, Galdran's algorithm, Li's algorithm, Wang's algorithm, WaterNet, FUnIE-GAN, CWR and the present invention are used.
3. Simulation test content:
simulation test 1: the four underwater simulation images are restored by the nine methods, and the result is shown in fig. 4, wherein:
FIG. 4a is four composite underwater images;
FIG. 4b is the result of processing the underwater simulation image of FIG. 4a using an Anwar's algorithm;
FIG. 4c is the result of recovering the underwater simulated image of FIG. 4a using Zhang's algorithm;
FIG. 4d is a result of restoring the underwater simulated image of FIG. 4a using the algorithm of RCP;
FIG. 4e is the result of using Li method to recover the underwater simulation image of FIG. 4 a;
FIG. 4f is the result of the reconstruction of the underwater simulated image of FIG. 4a using the method of Wang;
FIG. 4g is the result of using WaterNet to reconstruct the underwater simulated image of FIG. 4 a;
FIG. 4h shows the result of reconstructing the underwater simulated image of FIG. 4a using FUnIE-GAN;
FIG. 4i is the result of recovering the underwater simulated image of FIG. 4a using CWR;
FIG. 4j is a result of using the method of the present invention to recover the underwater simulated image of FIG. 4 a;
from fig. 4 the following different recovery effects can be seen:
when the existing image processing methods of Anwar, Zhang, RCP and Li are used for processing the images, the reduction result of color information is poor, and almost all the images keep the original blue/green color cast;
images restored by using the existing Anwar method and the RCP algorithm are still fuzzy sometimes; the color cast can be removed by using the algorithm proposed by the existing Wang et al, but the color transition of a local area is unnatural due to inaccurate transmittance estimation;
the image processed by the existing WaterNet has yellow color cast on the whole, and the color is not real and natural enough;
the processing result of the existing CWR algorithm is unstable, and the color cast removal of part of the image is invalid;
after the FunIEGAN treatment is used, the image definition is good, the color cast can be removed, but the restored image still has the condition of inconsistent coloring;
the image effect recovered by the method is better than that of other algorithms, the recovered image has clear boundary and high saturation, and the method has good recovery effect on the image with rich colors.
Simulation test 2: the above nine methods are used to restore six underwater real images, and the effect is shown in fig. 5, in which:
FIG. 5a is six underwater real images;
FIG. 5b is the result of processing the underwater real image of FIG. 5a using an Anwar's algorithm;
FIG. 5c is the result of using Zhang's algorithm to recover the underwater real image of FIG. 5 a;
FIG. 5d is the result of restoring the underwater real image of FIG. 5a using the RCP algorithm;
FIG. 5e is the result of using Li method to recover the underwater real image of FIG. 5 a;
FIG. 5f is the result of restoring the underwater real image of FIG. 5a using the method of Wang;
FIG. 5g is the result of using WaterNet to recover the underwater real image of FIG. 5 a;
FIG. 5h shows the result of using FUnIE-GAN to recover the underwater real image of FIG. 5 a;
fig. 5i is a result of restoring the underwater real image of fig. 5a by using CWR;
FIG. 5j is a result of using the method of the present invention to recover the underwater real image of FIG. 5 a;
the following recovery effects of the different methods can be seen from fig. 5:
the result processed by the existing Anwar method is improved in overall contrast, but a local area can introduce purple color cast; the images processed by the existing Zhang algorithm and Li algorithm have color oversaturation, and are not real and natural;
after the existing WaterNet processing is used, the image saturation is reduced; the method proposed by the prior RCP, Wang and the like is used for treating the color cast of the image with insufficient intensity and the phenomenon of under-treatment, and a plurality of images still present serious color cast;
the use of the existing FUnIE-GAN eliminates most of the color cast of blue-green, but loses much of the texture detail and major background information;
the color cast is obviously corrected by using the existing CWR method, the visibility of the restored image is improved, however, the CWR shows a low contrast ratio or supersaturation phenomenon in some local areas;
the image effect recovered by the method of the invention is superior to other existing eight methods, and the method has good visual processing effect on images with different degradation degrees.
Simulation test 3: the nine methods are used for processing 406 groups of underwater simulation images, and the structural similarity SSIM index, the peak signal-to-noise ratio PNSR index and the color difference formula CIEDE2000 index are calculated, and the results are shown in Table 1.
TABLE 1
Index (I) Anwar Zhang RCP Li Wang WaterNet FunIEGAN CWR The invention
PSNR 17.578 18.020 20.541 17.526 20.534 22.203 22.102 18.775 23.767
SSIM 0.873 0.883 0.950 0.812 0.936 0.954 0.927 0.866 0.969
CIEDE2000 17.123 14.964 15.992 14.436 16.283 9.004 7.637 12.318 6.075
As can be seen from Table 1, the PSNR, SSIM and color difference index values of the method are superior to those of other algorithms, are consistent with subjective results, effectively correct color cast and improve image definition.
Simulation test 4: the nine methods are used for processing 126 underwater real images, and BRISQE indexes of the no-reference image space quality evaluator and UCIQE indexes of underwater color image quality evaluation are calculated, and the results are shown in Table 2.
TABLE 2
Figure BDA0003300286340000101
As can be seen from Table 2, the BRISQE and UCIQE values of the method are superior to those of other eight methods, and are consistent with subjective results, which shows that the pictures processed by the method are clearer and more natural.
In conclusion, the method disclosed by the invention has better effect on underwater image processing than other eight methods.

Claims (7)

1. An end-to-end underwater image restoration method based on ambient light perception comprises the following steps:
1) constructing an underwater image restoration network ECCN under a Pythrch framework:
1a) establishing an ambient light sensing network comprising eleven convolutional layers, an upper sampling layer and a pooling layer;
1b) establishing a restoration main body network comprising a shallow layer feature extraction module, a multi-scale coding and decoding module and three feature integration reconstruction modules;
1c) cascading an ambient light sensing network and a restoration main network to form an underwater image restoration network;
2) downloading an indoor data set NYU containing depth information, a real single-image defogging data set RESIDE containing depth information and a paired underwater data set EUVP synthesized based on a generation network, and respectively carrying out screening, scaling and cutting on the data sets in sequence to obtain an indoor clear image set J, an indoor depth image set D, an underwater clear image set U and an underwater degraded image set R1;
3) respectively constructing training image sets of an ambient light perception network and a restoration main network:
setting ambient light value A and blue channel transmittance TbSynthesizing an underwater degraded image set R2 by using an indoor clear image set J and an indoor depth image set D, and pairing the underwater degraded image set R2 with an ambient light value A corresponding to the underwater degraded image set to be used as a training image set B of an ambient light sensing network;
matching the indoor clear image set J with the underwater degraded image set R2, adding the underwater clear image set U and the underwater degraded image set R1 into the indoor clear image set J and the underwater degraded image set U to serve as a training image set C of a recovery main network;
4) training the ambient light sensing network by utilizing a training image set B of the ambient light sensing network and adopting a self-adaptive moment estimation algorithm with a minimized Euclidean distance loss value as a target to obtain a trained ambient light sensing network;
5) training the restoration main network by using a training image set C of the restoration main network and a trained ambient light estimation network and adopting an adaptive moment estimation algorithm with a minimum weighted total loss value as a target to obtain the trained restoration main network;
6) an underwater image I needing restoration processingcInputting the light into a trained ambient light sensing network, and outputting an ambient light value A of RGB three channelsc；
7) The ambient light value AcAnd underwater image IcInputting the two into a trained restoration main network, and outputting a restored clear image Jc。
2. The method of claim 1, wherein: 1a) the structural relationship and each layer parameter of the medium-environment light sensing network are as follows:
the first buildup layer → the second buildup layer → the third buildup layer → the fourth buildup layer → the upsampling layer → the fifth buildup layer → the sixth buildup layer → the seventh buildup layer → the eighth buildup layer → the pooling layer → the ninth buildup layer → the tenth buildup layer → the eleventh buildup layer;
the convolution kernels of the first convolution layer and the second convolution layer are both 9 x 9 in size;
the convolution kernels of the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer are all 7 x 7 in size;
the convolution kernels of the seventh convolution layer, the eighth convolution layer, the ninth convolution layer and the tenth convolution layer are all 5 x 5 in size;
the convolution kernel size of the eleventh convolution layer is 3 x 3;
the step length of all the convolution layers is 1; all convolutional layers include convolutional operations and the ReLU activation function layer;
the scaling factor of the upsampling layer is 2;
the scaling factor of the pooling layer is 2.
3. The method of claim 1, wherein: 1b) the structure of each module is as follows:
the shallow layer feature extraction module is formed by cascading a1 st convolution layer → a2 nd convolution layer → a3 rd convolution layer;
the multi-scale coding and decoding module is formed by connecting three branches in parallel, wherein:
the first branch is in turn: the 1 st pooling layer → the 4 th convolution layer → the 1 st encoding unit → the 2 nd encoding unit → the 1 st decoding unit → the 2 nd decoding unit;
the second branch is sequentially as follows: the 2 nd pooling layer → the 5 th convolution layer → the 3 rd encoding unit → the 4 th encoding unit → the 3 rd decoding unit → the 4 th decoding unit; the 3 rd decoding unit is simultaneously connected with the 2 nd encoding unit;
the third branch is sequentially as follows: the 3 rd pooling layer → the 6 th convolution layer → the 5 th encoding unit → the 6 th encoding unit → the 5 th decoding unit → the 6 th decoding unit; the 5 th decoding unit is simultaneously connected with the 4 th encoding unit;
the 1 st, 2 nd and 3 rd pooling layers are connected with the 3 rd convolution layer;
the coding unit is sequentially as follows: convolution layer I → residual module II → residual module III;
the decoding unit is sequentially as follows: IV residual module → V residual module → VI residual module → bilinear interpolation → II convolution layer.
The characteristic integration and reconstruction module sequentially comprises: an upsampling layer → a first convolution layer → a second convolution layer → a third convolution layer;
the 2 nd decoding unit, the 4 th decoding unit and the 6 th decoding unit of the multi-scale coding and decoding module are respectively connected with the three feature reconstruction modules, and the 3 rd convolution layer of the shallow feature extraction module is simultaneously connected with the first convolution layer of the feature reconstruction module;
the output of the restoration master network is the end output of the feature reconstruction module connected to the third branch.
4. The method of claim 3, wherein the parameters of the layers of the resilient host network are set as follows:
the convolution kernel size of the 1 st convolution layer is 7 x 7, the convolution kernel size of the 2 nd convolution layer is 5 x 5, and the convolution kernel sizes of the other convolution layers are 3 x 3;
the convolution step length of all convolution layers is 1, and all convolution layers comprise convolution operation and a ReLU activation function layer;
the scaling factor of the upsampling layer is 2;
the scaling factor of the 1 st pooling layer is 8, the scaling factor of the 2 nd pooling layer is 4, and the scaling factor of the 3 rd pooling layer is 2;
the scaling factor of the bilinear interpolation operation is 2.
5. The method according to claim 1, wherein the 3) synthesizes an underwater degraded image set R2 by using an indoor clear image set J and an indoor depth map set D, which is realized as follows:
3a) randomly generating a red channel ambient light value A between 0.1 and 0.6 using a random functionrUsing random function at ArRandomly generating a green channel ambient light value A between-1.0gAnd the ambient light value A of the blue channelb；
3b) Randomly generating a blue channel transmittance parameter beta between 0.5 and 2.5 using a random functionbCalculating the transmittance of blue channel of each image by using the indoor depth map set D
Figure FDA0003300286330000031
Red channel transmission
Figure FDA0003300286330000032
Green channel transmission
Figure FDA0003300286330000033
3c) And calculating to obtain an underwater degraded image set R2 (JT + A (1-T)) according to the indoor clear image set J and the generated ambient light value A and transmittance T.
6. The method as claimed in claim 1, wherein the training of the ambient light sensing network in 4) is implemented as follows:
4a) the Euclidean distance formula is used as a loss function LossB of the ambient light sensing network, and is expressed as follows:
Figure FDA0003300286330000034
wherein | |2In order to perform a two-norm operation on the matrix,c is the color channel of the input image, BcIs the output of the ambient light aware network, B'cCorresponding artificially synthesized ambient light;
4b) sequentially inputting the underwater degraded images in the training image set B of the ambient light perception network into the ambient light perception network, and outputting an estimated ambient light value A';
4c) simultaneously substituting the set ambient light value A and the estimated ambient light value A 'into an Euclidean distance formula, and calculating to obtain an Euclidean distance loss value between A and A';
4d) and updating the weight and the offset value of each convolution operation in the ambient light perception network by using a self-adaptive moment estimation algorithm, and repeating the steps 4b) -4c) 10000 times until the Euclidean distance loss value is minimum to obtain the trained ambient light perception network.
7. The method of claim 1, wherein the training of the restoration subject network in 5) is implemented as follows:
5a) the weighted total loss formula is used as a loss function LossC of the restoration subject network, and is expressed as follows:
LossC＝0.75×Losst+0.25×Lossc
therein, LosstIs the boundary maintenance Loss, LosscIs the color shift correction loss, the calculation formula is:
Figure FDA0003300286330000041
Figure FDA0003300286330000042
wherein | |2To solve the two norm operation on the matrix, |. i is to solve the one norm operation on the matrix, | N is the number of pixels of the input image, Jn、Jn1、Jn2The three characteristics of the restoration main network are respectively integrated and rebuilt the tail end output, J'n，J'n1，J'n2Is corresponding toClear picture, J'n1Is J 'reduced by 16 times'n，J'n2Is J 'reduced by 64 times'n；
Figure FDA0003300286330000043
Represents the mean value, Δ r, of the red channeln、△bn、△gnRespectively representing the difference between the network output and the clear image in red, green and blue channels;
5b) sequentially inputting the underwater degraded images in the recovered main network training set C into a trained ambient light estimation network, and outputting an estimated ambient light value A ";
5c) combining the underwater degraded image in the recovered main network training set C with the estimated ambient light value A ', inputting the combined image into the recovered main network, and outputting an estimated recovered image I';
5d) simultaneously substituting the clear image I and the estimated restoration image I 'in the restoration main body network training set C into a weighted total loss function LossC, and calculating to obtain a weighted total loss value between the clear image I and the restoration image I';
5e) updating the weight and the offset value of each convolution operation in the restoration main network by using a self-adaptive moment estimation algorithm;
5f) repeating 5c) -5e) for 20000 times until the weighted total loss value is minimum, and obtaining a trained recovery subject network.
CN202111195071.8A 2021-10-12 2021-10-12 End-to-end underwater image restoration method based on ambient light perception Active CN113935916B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111195071.8A CN113935916B (en) 2021-10-12 2021-10-12 End-to-end underwater image restoration method based on ambient light perception

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111195071.8A CN113935916B (en) 2021-10-12 2021-10-12 End-to-end underwater image restoration method based on ambient light perception

Publications (2)

Publication Number Publication Date
CN113935916A true CN113935916A (en) 2022-01-14
CN113935916B CN113935916B (en) 2024-11-22

Family

ID=79279180

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111195071.8A Active CN113935916B (en) 2021-10-12 2021-10-12 End-to-end underwater image restoration method based on ambient light perception

Country Status (1)

Country Link
CN (1) CN113935916B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114387190A (en) * 2022-03-23 2022-04-22 山东省计算中心（国家超级计算济南中心） Adaptive image enhancement method and system based on complex environment
CN115049562A (en) * 2022-07-05 2022-09-13 福州大学 Underwater image restoration method based on distortion guidance

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109584170A (en) * 2018-10-29 2019-04-05 西安电子科技大学 Underwater image restoration method based on convolutional neural networks
CN110223251A (en) * 2019-06-02 2019-09-10 西安电子科技大学 Suitable for manually with the convolutional neural networks underwater image restoration method of lamp
CN113012067A (en) * 2021-03-16 2021-06-22 华南理工大学 Retinex theory and end-to-end depth network-based underwater image restoration method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109584170A (en) * 2018-10-29 2019-04-05 西安电子科技大学 Underwater image restoration method based on convolutional neural networks
CN110223251A (en) * 2019-06-02 2019-09-10 西安电子科技大学 Suitable for manually with the convolutional neural networks underwater image restoration method of lamp
CN113012067A (en) * 2021-03-16 2021-06-22 华南理工大学 Retinex theory and end-to-end depth network-based underwater image restoration method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
YAN HU等: "Underwater Image Restoration Based on Convolutional Neural Network", 《PROCEEDINGS OF MACHINE LEARNING RESEARCH》, 4 November 2018 (2018-11-04) *
丛润民;张禹墨;张晨;李重仪;赵耀;: "深度学习驱动的水下图像增强与复原研究进展", 信号处理, no. 09, 20 November 2020 (2020-11-20) *
陈静怡: "基于深度学习的水下降质图像复原方法研究", 《万方数据》, 14 June 2022 (2022-06-14) *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114387190A (en) * 2022-03-23 2022-04-22 山东省计算中心（国家超级计算济南中心） Adaptive image enhancement method and system based on complex environment
CN114387190B (en) * 2022-03-23 2022-08-16 山东省计算中心（国家超级计算济南中心） Adaptive image enhancement method and system based on complex environment
CN115049562A (en) * 2022-07-05 2022-09-13 福州大学 Underwater image restoration method based on distortion guidance

Also Published As

Publication number Publication date
CN113935916B (en) 2024-11-22

Similar Documents

Publication Publication Date Title
CN109584170B (en) Underwater image restoration method based on convolutional neural network
CN114066747B (en) Low-illumination image enhancement method based on illumination and reflection complementarity
CN110223251B (en) Convolution neural network underwater image restoration method suitable for artificial and natural light sources
CN111161360B (en) Image defogging method of end-to-end network based on Retinex theory
CN115223004A (en) Method for generating confrontation network image enhancement based on improved multi-scale fusion
CN108550130A (en) A kind of multiple dimensioned transmission plot fusion method of image pyramid model
CN110675462A (en) A Colorization Method of Grayscale Image Based on Convolutional Neural Network
CN111553856B (en) Image defogging method based on depth estimation assistance
CN114067018B (en) Infrared image colorization method for generating countermeasure network based on expansion residual error
Wang et al. Underwater color disparities: Cues for enhancing underwater images toward natural color consistencies
CN115393227B (en) Self-adaptive enhancement method and system for low-light full-color video images based on deep learning
CN110738622A (en) Lightweight neural network single image defogging method based on multi-scale convolution
CN113935916B (en) End-to-end underwater image restoration method based on ambient light perception
CN114820408A (en) Infrared and visible light image fusion method based on self-attention and convolutional neural network
CN111598789A (en) Sparse color sensor image reconstruction method based on deep learning
CN111462002B (en) Underwater image enhancement and restoration method based on convolutional neural network
CN117151990A (en) Image defogging method based on self-attention coding and decoding
CN116703750A (en) Image defogging method and system based on edge attention and multi-order differential loss
Shen et al. Pseudo-retinex decomposition-based unsupervised underwater image enhancement and beyond
CN114565539B (en) Image defogging method based on online knowledge distillation
CN116109510A (en) A Face Image Inpainting Method Based on Dual Generation of Structure and Texture
CN116071266A (en) Retinex-based low-light image enhancement method, storage medium and terminal
CN116228550A (en) Image self-enhancement defogging algorithm based on generation of countermeasure network
CN112991201A (en) Image defogging method based on color correction and context aggregation residual error network
CN115082315B (en) Demosaicing method applicable to low-illumination small-pixel CFA sampling and edge computing equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant