CN114612347A - Multi-module cascade underwater image enhancement method - Google Patents

Multi-module cascade underwater image enhancement method Download PDF

Info

Publication number
CN114612347A
CN114612347A CN202210506856.0A CN202210506856A CN114612347A CN 114612347 A CN114612347 A CN 114612347A CN 202210506856 A CN202210506856 A CN 202210506856A CN 114612347 A CN114612347 A CN 114612347A
Authority
CN
China
Prior art keywords
network
image
channel
underwater
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210506856.0A
Other languages
Chinese (zh)
Other versions
CN114612347B (en
Inventor
刘红敏
丁艳
樊彬
曾慧
张利欣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology Beijing USTB
Original Assignee
University of Science and Technology Beijing USTB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology Beijing USTB filed Critical University of Science and Technology Beijing USTB
Priority to CN202210506856.0A priority Critical patent/CN114612347B/en
Publication of CN114612347A publication Critical patent/CN114612347A/en
Application granted granted Critical
Publication of CN114612347B publication Critical patent/CN114612347B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a multi-module cascaded underwater image enhancement method, and belongs to the technical field of computer vision. The method comprises the following steps: cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image; acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets; and acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the enhanced underwater image. By adopting the invention, the degradation problems of different types in underwater imaging can be solved.

Description

Multi-module cascade underwater image enhancement method
Technical Field
The invention relates to the technical field of computer vision, in particular to an underwater image enhancement method based on multi-module cascade.
Background
In recent years, as a major problem in image enhancement research, underwater image enhancement has received increasing attention from researchers. As an important carrier of ocean information, the underwater image plays a vital role in exploring ocean environment, and reasonably developing and utilizing ocean resources. However, due to the complexity of the underwater imaging environment, the obtained underwater image is often accompanied by degradation problems such as blurring, low contrast, color distortion, poor visibility, and the like, which seriously affects the performance of the task based on underwater vision. Therefore, it is urgently required to improve the quality of underwater images.
In the past decades, many methods have been proposed to improve the quality of underwater images, and these methods can be simply classified into non-learning methods and deep learning-based methods. Among the non-learning methods, one is to apply the classical air image enhancement method or its variants (such as histogram equalization, white balance, etc.) directly on the underwater image; the other is a specially designed algorithm aiming at the imaging characteristics of the underwater image or a physical imaging model combined with the underwater image, such as Retinex-based, Fusion-based, GDCP-based and the like. Although these methods improve the quality of underwater images, they are susceptible to degraded image types and have poor generalization capability due to the uncertainty of estimating the physical model parameters and the inaccuracy of a priori knowledge. With the development of deep learning, researchers provide a series of underwater image enhancement methods based on deep learning, such as Water-Net, UIEC2^ Net, Ucolor and the like, which directly model degraded images and clear images, relieve the unsuitability of estimation model parameters and greatly improve the quality of underwater images, but the methods do not consider the attenuation difference between R, G, B channels caused by attenuation related to wavelength, so that color cast exists in the enhanced images, and the methods are still limited by the degradation types of the underwater images, and can not solve the degradation problem existing in the underwater images at the same time. Solving the various degradation problems that coexist in underwater images through a single network remains a significant challenge.
Disclosure of Invention
The embodiment of the invention provides a multi-module cascaded underwater image enhancement method, which can solve the degradation problems of different types in an underwater image. The technical scheme is as follows:
the embodiment of the invention provides a multi-module cascaded underwater image enhancement method, which comprises the following steps:
cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image;
acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets;
and acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the enhanced underwater image.
Further, the step of cascading the existing air image enhancement network with the color correction network to construct a multi-module cascade enhancement network includes:
selecting an existing aerial image enhancement network as a first stage enhancement network E1;
the color correction network is taken as a second-stage enhancement network E2;
and connecting the E1 and the E2 in a residual error mode to obtain the multi-module cascade enhanced network E.
Further, the processing step of the color correction network comprises:
a1, enhancing the output image of the network E1 by the first level
Figure 636057DEST_PATH_IMAGE001
And input image
Figure 415663DEST_PATH_IMAGE002
Connected by a residual structure to obtain an input image of a second-level enhanced network E2
Figure 615526DEST_PATH_IMAGE003
Then extracting the red channel images thereof respectively
Figure 949424DEST_PATH_IMAGE004
Green channel image
Figure 164374DEST_PATH_IMAGE005
And blue channel image
Figure 532907DEST_PATH_IMAGE006
Wherein, in the step (A),
Figure 175110DEST_PATH_IMAGE007
、
Figure 2163DEST_PATH_IMAGE008
respectively representing the height and width of the image,
Figure 755225DEST_PATH_IMAGE009
is a dimension symbol;
a2, performing convolution operation on the three-channel images obtained in the step A1 respectively to obtain red channel feature maps
Figure 712685DEST_PATH_IMAGE010
Green channel profile
Figure 525789DEST_PATH_IMAGE011
And blue channel profile
Figure 834280DEST_PATH_IMAGE012
：
Figure 922190DEST_PATH_IMAGE013
Wherein the content of the first and second substances,
Figure 17315DEST_PATH_IMAGE014
、
Figure 1320DEST_PATH_IMAGE015
and
Figure 797107DEST_PATH_IMAGE016
all show belt
Figure 891971DEST_PATH_IMAGE017
A convolution operation of the layers;
a3, respectively compensating the information of the red channel characteristic diagram and the blue channel characteristic diagram by using the characteristic diagram of the green channel to obtain the compensated red channel characteristic diagram
Figure 89603DEST_PATH_IMAGE018
Green channel profile
Figure 250369DEST_PATH_IMAGE019
And blue channel profile
Figure 533451DEST_PATH_IMAGE020
：
Figure 166427DEST_PATH_IMAGE021
Wherein, the first and the second end of the pipe are connected with each other,
Figure 484145DEST_PATH_IMAGE022
are representative of the compensation parameters that are,
Figure 137849DEST_PATH_IMAGE023
representing splicing operation according to channels;
a4, sending the compensated feature map obtained in the step A3 into a channel-space attention module, and further extracting and refining the features to obtain a red channel feature map
Figure 636789DEST_PATH_IMAGE024
Green channel profile
Figure 604614DEST_PATH_IMAGE025
And blue channel profile
Figure 776838DEST_PATH_IMAGE026
：
Figure 7968DEST_PATH_IMAGE027
Wherein the content of the first and second substances,
Figure 64DEST_PATH_IMAGE028
representing a channel-space attention module;
a5, for the characteristic diagram obtained in the step A4, the characteristic diagram of the green channel is used for compensating the information of the characteristic diagrams of the other two channels to obtain the characteristic diagram after color correction
Figure 246280DEST_PATH_IMAGE029
、
Figure 273011DEST_PATH_IMAGE030
And
Figure 675043DEST_PATH_IMAGE031
：
Figure 420013DEST_PATH_IMAGE032
wherein the content of the first and second substances,
Figure 932903DEST_PATH_IMAGE033
both represent compensation parameters;
a6, correcting the color of the feature map
Figure 79720DEST_PATH_IMAGE034
And
Figure 482731DEST_PATH_IMAGE035
respectively changing into single-channel characteristic diagrams, and splicing according to the channels to obtain color characteristic diagrams
Figure 449418DEST_PATH_IMAGE036
：
Figure 31578DEST_PATH_IMAGE037
Wherein the content of the first and second substances,
Figure 767322DEST_PATH_IMAGE038
、
Figure 42314DEST_PATH_IMAGE039
and
Figure 626792DEST_PATH_IMAGE040
both represent convolution operations;
a7, sending the color feature map into a convolution module to reconstruct a clear underwater image, namely a final enhanced underwater image
Figure 12642DEST_PATH_IMAGE041
：
Figure 602893DEST_PATH_IMAGE042
Wherein the content of the first and second substances,
Figure 783207DEST_PATH_IMAGE043
which represents a convolution operation, is a function of,
Figure 724487DEST_PATH_IMAGE044
representing a volume block.
Further, the processing step of the channel-space attention module comprises:
a41, inputting a feature map
Figure 642590DEST_PATH_IMAGE045
By convolution operations
Figure 352926DEST_PATH_IMAGE046
Obtaining a new feature map
Figure 438563DEST_PATH_IMAGE047
；
Figure 867139DEST_PATH_IMAGE048
Wherein the content of the first and second substances,
Figure 594792DEST_PATH_IMAGE049
meaning that the summation is by element,
Figure 165494DEST_PATH_IMAGE045
in particular, the compensated characteristic diagram obtained in step A3
Figure 218770DEST_PATH_IMAGE050
、
Figure 603483DEST_PATH_IMAGE051
And
Figure 134828DEST_PATH_IMAGE052
；
a42, obtaining the characteristic diagram A41
Figure 288598DEST_PATH_IMAGE047
Respectively sending into a channel attention branch CA _ brach and a space attention branch SA _ brach to obtain a channel feature descriptor
Figure 247195DEST_PATH_IMAGE053
And spatial feature descriptors
Figure 402362DEST_PATH_IMAGE054
Then, the feature map is processed
Figure 206239DEST_PATH_IMAGE047
Respectively with channel feature descriptors
Figure 745674DEST_PATH_IMAGE053
Multiplying the spatial feature descriptors by elements to obtain the output of CA _ burst and SA _ burst
Figure 875173DEST_PATH_IMAGE055
、
Figure 500058DEST_PATH_IMAGE056
(ii) a Wherein the content of the first and second substances,
Figure 379064DEST_PATH_IMAGE057
represents multiplication by element;
a43, splicing the outputs of CA _ break and SA _ break in the step A42 according to channels and performing convolution operation
Figure 773005DEST_PATH_IMAGE058
Obtaining a final output characteristic diagram after processing
Figure 807826DEST_PATH_IMAGE059
：
Figure 920008DEST_PATH_IMAGE060
Wherein the content of the first and second substances,
Figure 65687DEST_PATH_IMAGE059
in particular to
Figure 579714DEST_PATH_IMAGE061
、
Figure 513998DEST_PATH_IMAGE062
And
Figure 113475DEST_PATH_IMAGE063
。
further, the training the multi-module cascade enhancement network includes:
determining a loss function of the multi-module cascade enhancement network E:
Figure 328425DEST_PATH_IMAGE064
wherein, the first and the second end of the pipe are connected with each other,
Figure 431379DEST_PATH_IMAGE065
representing the loss function originally used by the first stage enhancement network E1,
Figure 73582DEST_PATH_IMAGE066
the function of the perceptual loss is represented by,
Figure 635056DEST_PATH_IMAGE067
representing perceptual loss functions
Figure 388117DEST_PATH_IMAGE068
The weight of (c);
determining an initial learning rate of a multi-module cascaded enhanced network E, wherein the initial learning rate of a first stage enhanced network E1
Figure 611157DEST_PATH_IMAGE069
At least one order of magnitude smaller than the initial learning rate set in the original aerial image enhancement network, the initial learning rate of the second stage enhancement network E2
Figure 424261DEST_PATH_IMAGE070
Enhancing the initial learning rate set in the network for the original air image;
and training the multi-module cascade enhancement network E by using the acquired paired underwater image data sets.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:
1) the degradation problem of different underwater scenes is considered, the complicated underwater degradation problem is decomposed into different subproblems, and the degradation problems of different types in underwater imaging are solved by cascading different air image enhancement networks.
2) For the difference of the R, G, B channel attenuation, in this embodiment, the G channel with smaller information attenuation adaptively compensates the R channel and the B channel with more serious information attenuation through the color correction network, so as to correct the color of the underwater image.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of an underwater image enhancement method of multi-module cascade connection according to an embodiment of the present invention;
fig. 2 is a schematic view of a workflow of a multi-module cascade enhanced network according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a color correction network according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an enhanced underwater image according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Different from underwater images, the depth learning-based enhancement algorithm designed for air images is relatively mature (such as an image defogging algorithm and a low-illumination image enhancement algorithm), and on the basis, in order to fully utilize the existing research results, reduce the difficulty of network information processing and solve various coexisting degradation problems in the underwater images, the embodiment of the invention provides an end-to-end multi-module cascaded underwater image enhancement method, and the method provides a multi-module cascaded enhancement network.
As shown in fig. 1 and fig. 2, an embodiment of the present invention provides an underwater image enhancement method with multiple modules cascaded, including:
s101, cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image; the method specifically comprises the following steps:
selecting an existing aerial image enhancement network as a first stage enhancement network E1; wherein the aerial image enhancement network comprises: a defogging network GridDehazeNet, a low-illumination enhancement network MIRNet and the like, and network weight parameters are preloaded;
the color correction network is taken as a second-stage enhancement network E2; wherein, the color correction network is proposed in this embodiment;
and connecting the E1 and the E2 in a residual error mode to obtain the multi-module cascade enhanced network E.
In this embodiment, the multi-module cascade enhancement network E includes two parts, the former is the existing air image enhancement network; the latter is a color correction network designed in consideration of the difference of R, G, B channel attenuation, in which a G channel with relatively small information attenuation is used to adaptively compensate an R channel and a B channel with relatively serious information attenuation, thereby correcting the color of the underwater image.
As shown in fig. 3, the processing steps of the color correction network include:
a1, enhancing the output image of the network E1 at the first level
Figure 732752DEST_PATH_IMAGE071
And input image
Figure 23925DEST_PATH_IMAGE072
Connected by a residual structure to obtain an input image of a second-level enhanced network E2
Figure 915787DEST_PATH_IMAGE003
Then extracting the red channel images thereof respectively
Figure 634213DEST_PATH_IMAGE004
Green channel image
Figure 429999DEST_PATH_IMAGE005
And blue channel image
Figure 524863DEST_PATH_IMAGE006
Wherein, in the step (A),
Figure 722495DEST_PATH_IMAGE073
、
Figure 883261DEST_PATH_IMAGE074
respectively representing the height and width of the image,
Figure 431923DEST_PATH_IMAGE009
is a dimension symbol;
a2, performing convolution operation on the three-channel images obtained in the step A1 respectively to obtain red channel feature maps
Figure 64899DEST_PATH_IMAGE010
Green channel profile
Figure 117037DEST_PATH_IMAGE011
And blue channel profile
Figure 177266DEST_PATH_IMAGE012
：
Figure 947645DEST_PATH_IMAGE075
Wherein the content of the first and second substances,
Figure 644031DEST_PATH_IMAGE014
、
Figure 285097DEST_PATH_IMAGE015
and
Figure 781806DEST_PATH_IMAGE076
all show belt
Figure 39481DEST_PATH_IMAGE017
A convolution operation of the layers;
a3, respectively compensating the information of the red channel characteristic diagram and the blue channel characteristic diagram by using the characteristic diagram of the green channel to obtain the compensated red channel characteristic diagram
Figure 14259DEST_PATH_IMAGE018
Green channel profile
Figure 46849DEST_PATH_IMAGE019
And blue channel profile
Figure 714460DEST_PATH_IMAGE020
：
Figure 459431DEST_PATH_IMAGE077
Wherein the content of the first and second substances,
Figure 237900DEST_PATH_IMAGE078
are representative of the compensation parameters that are,
Figure 119137DEST_PATH_IMAGE023
representing splicing operation according to channels;
a4, sending the compensated feature map obtained in the step A3 into a channel-space attention module, and further extracting and refining the features to obtain a red channel feature map
Figure 240806DEST_PATH_IMAGE024
Green channel profile
Figure 473073DEST_PATH_IMAGE025
And blue channel profile
Figure 55233DEST_PATH_IMAGE026
：
Figure 322135DEST_PATH_IMAGE079
Wherein the content of the first and second substances,
Figure 65969DEST_PATH_IMAGE028
representing a channel-space attention module, the processing steps of which include:
a41, outputCharacteristic diagram
Figure 519953DEST_PATH_IMAGE045
By convolution operations
Figure 442822DEST_PATH_IMAGE046
Obtaining a new feature map
Figure 33072DEST_PATH_IMAGE047
:
Figure 213387DEST_PATH_IMAGE080
Wherein the content of the first and second substances,
Figure 154667DEST_PATH_IMAGE049
meaning that the summation is by element,
Figure 78629DEST_PATH_IMAGE045
in particular, the compensated characteristic diagram obtained in step A3
Figure 517527DEST_PATH_IMAGE081
、
Figure 134322DEST_PATH_IMAGE051
And
Figure 562898DEST_PATH_IMAGE052
；
a42, characterization map obtained from A41
Figure 24972DEST_PATH_IMAGE047
Respectively sending into a channel attention branch CA _ brach and a space attention branch SA _ brach to obtain a channel feature descriptor
Figure 589814DEST_PATH_IMAGE082
And spatial feature descriptors
Figure 117791DEST_PATH_IMAGE054
Then, the feature map is mapped
Figure 33663DEST_PATH_IMAGE047
Respectively with channel feature descriptors
Figure 33849DEST_PATH_IMAGE082
And spatial feature descriptors
Figure 718777DEST_PATH_IMAGE054
Element-by-element multiplication to obtain the outputs of CA _ burst and SA _ burst
Figure 411795DEST_PATH_IMAGE055
、
Figure 549385DEST_PATH_IMAGE056
(ii) a Wherein, the first and the second end of the pipe are connected with each other,
Figure 167577DEST_PATH_IMAGE057
represents multiplication by element;
a43, splicing the outputs of CA _ break and SA _ break in the step A42 according to channels and performing convolution operation
Figure 972591DEST_PATH_IMAGE058
Obtaining a final output characteristic diagram after processing
Figure 367669DEST_PATH_IMAGE059
：
Figure 992554DEST_PATH_IMAGE083
Wherein the content of the first and second substances,
Figure 334543DEST_PATH_IMAGE059
in particular to
Figure 265502DEST_PATH_IMAGE061
、
Figure 831481DEST_PATH_IMAGE084
And
Figure 678083DEST_PATH_IMAGE063
。
a5, for the characteristic diagram obtained in the step A4, the characteristic diagram of the green channel is used to compensate the information of the other two channel characteristic diagrams, and the characteristic diagram after color correction is obtained
Figure 89342DEST_PATH_IMAGE029
、
Figure 72210DEST_PATH_IMAGE030
And
Figure 537653DEST_PATH_IMAGE031
：
Figure 137130DEST_PATH_IMAGE085
wherein the content of the first and second substances,
Figure 86500DEST_PATH_IMAGE033
both represent compensation parameters;
a6, correcting the color of the feature map
Figure 455034DEST_PATH_IMAGE086
And
Figure 831657DEST_PATH_IMAGE035
respectively changing into single-channel characteristic diagrams, and splicing according to the channels to obtain color characteristic diagrams
Figure 652852DEST_PATH_IMAGE087
:
Figure 146193DEST_PATH_IMAGE088
Wherein the content of the first and second substances,
Figure 634812DEST_PATH_IMAGE038
、
Figure 916758DEST_PATH_IMAGE039
and
Figure 490827DEST_PATH_IMAGE089
both represent convolution operations;
a7, sending the color feature map into a convolution module to reconstruct a clear underwater image, namely a final enhanced underwater image
Figure 47580DEST_PATH_IMAGE041
：
Figure 877125DEST_PATH_IMAGE090
Wherein the content of the first and second substances,
Figure 861130DEST_PATH_IMAGE043
which represents a convolution operation, the operation of the convolution,
Figure 391337DEST_PATH_IMAGE044
representing a volume block.
S102, acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets, specifically including the following steps:
b1, acquiring paired underwater image data sets; wherein each pair of underwater images comprises: a degraded underwater image and its corresponding reference image;
in this embodiment, a paired underwater image dataset for training the multi-module cascade enhancement network E is constructed from the existing disclosed underwater dataset.
B2, determining a loss function of the multi-module cascade enhancement network E:
Figure 751781DEST_PATH_IMAGE064
wherein the content of the first and second substances,
Figure 683833DEST_PATH_IMAGE091
representing a first step increaseLoss function originally used by strong network E1;
Figure 838740DEST_PATH_IMAGE067
representing perceptual loss functions
Figure 862103DEST_PATH_IMAGE066
The weight of (a) is 0.04; wherein the perceptual loss function
Figure 26237DEST_PATH_IMAGE066
Expressed as:
Figure 812796DEST_PATH_IMAGE092
wherein the content of the first and second substances,
Figure 138604DEST_PATH_IMAGE093
and
Figure 908983DEST_PATH_IMAGE094
respectively representing the channel number, height and width of the characteristic diagram, the enhanced underwater image and the corresponding reference image,
Figure 74211DEST_PATH_IMAGE095
feature maps corresponding to images in different layers of the VGG-19 are shown, and in the embodiment of the invention, Conv1_2, Conv2_2 and Conv3_3 of the VGG-19 are selected for feature extraction.
In this embodiment, the loss function consists of two parts: part is the loss function originally used by E1
Figure 246435DEST_PATH_IMAGE091
(ii) a Another part is the perceptual loss function
Figure 477565DEST_PATH_IMAGE066
And is used for enabling the image generated by the multi-module cascade enhancement network E and the reference image to be as close as possible in the feature space.
B3, determining the initial learning rate of the multi-module cascade enhanced network E, whereinFirst level enhanced initial learning rate of network E1
Figure 735240DEST_PATH_IMAGE069
At least one order of magnitude smaller than the initial learning rate set in the original aerial image enhancement network, the initial learning rate of the second stage enhancement network E2
Figure 975597DEST_PATH_IMAGE070
Enhancing the initial learning rate set in the network for the original air image;
and B4, training the multi-module cascade enhancement network E by using the acquired paired underwater image data sets.
S103, acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into a trained multi-module cascade enhancement network to obtain the enhanced underwater image, wherein the method specifically comprises the following steps:
in this embodiment, an underwater image to be enhanced is acquired
Figure 2328DEST_PATH_IMAGE096
Inputting the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the final enhanced underwater image
Figure 410219DEST_PATH_IMAGE097
Fig. 4 shows a schematic diagram of the enhanced underwater image. Therefore, the research result of the existing image enhancement is fully utilized, the proposed color correction network is cascaded with different air image enhancement networks, and different underwater image enhancement tasks are realized, such as the defogging of an underwater image, the enhancement of an underwater low-illumination image and the color correction of the underwater image are realized; meanwhile, the method has stronger generalization capability, can be used for various underwater images with different degradation types, has strong universality and obtains more ideal enhancement effect.
The multi-module cascade underwater image enhancement method provided by the embodiment of the invention at least has the following beneficial effects:
1) the degradation problem of different underwater scenes is considered, the complicated underwater degradation problem is decomposed into different subproblems, and the degradation problems of different types in underwater imaging are solved by cascading different air image enhancement networks.
2) For the difference of the R, G, B channel attenuation, in this embodiment, the G channel with smaller information attenuation adaptively compensates the R channel and the B channel with more serious information attenuation through the color correction network, so as to correct the color of the underwater image.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (5)

1. An underwater image enhancement method based on multi-module cascade is characterized by comprising the following steps:
cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image;
acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets;
and acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the enhanced underwater image.
2. The multi-module cascaded underwater image enhancement method according to claim 1, wherein the step of cascading an existing air image enhancement network with a color correction network to construct a multi-module cascaded enhancement network comprises the steps of:
selecting an existing aerial image enhancement network as a first stage enhancement network E1;
the color correction network is taken as a second-stage enhancement network E2;
and connecting the E1 and the E2 in a residual error mode to obtain the multi-module cascade enhanced network E.
3. The underwater image enhancement method of the multi-module cascade as claimed in claim 1 or 2, wherein the processing step of the color correction network comprises:
a1, enhancing the output image of the network E1 at the first level
Figure 698677DEST_PATH_IMAGE001
And input image
Figure 273884DEST_PATH_IMAGE002
Connected by a residual structure to obtain an input image of a second-level enhanced network E2
Figure 155121DEST_PATH_IMAGE003
Then extracting the red channel images thereof respectively
Figure 999492DEST_PATH_IMAGE004
Green channel image
Figure 497339DEST_PATH_IMAGE005
And blue channel images
Figure 79499DEST_PATH_IMAGE006
Wherein, in the process,
Figure 815242DEST_PATH_IMAGE007
、
Figure 90235DEST_PATH_IMAGE008
respectively representing the height and width of the image,
Figure 561797DEST_PATH_IMAGE009
is a dimension symbol;
a2, performing convolution operation on the three-channel images obtained in the step A1 respectively to obtain red channel feature maps
Figure 947648DEST_PATH_IMAGE010
Green channel profile
Figure 537898DEST_PATH_IMAGE011
And blue channel profile
Figure 983791DEST_PATH_IMAGE012
:
Figure 190651DEST_PATH_IMAGE013
Wherein, the first and the second end of the pipe are connected with each other,
Figure 114613DEST_PATH_IMAGE014
、
Figure 830809DEST_PATH_IMAGE015
and
Figure 713183DEST_PATH_IMAGE016
both represent convolution operations of the tape layers;
a3, respectively compensating the information of the red channel characteristic diagram and the blue channel characteristic diagram by using the characteristic diagram of the green channel to obtain the compensated red channel characteristic diagram
Figure 407338DEST_PATH_IMAGE017
Green channel profile
Figure 134992DEST_PATH_IMAGE018
And blue channel profile
Figure 965413DEST_PATH_IMAGE019
：
Figure 747250DEST_PATH_IMAGE020
Wherein the content of the first and second substances,
Figure 928702DEST_PATH_IMAGE021
are representative of the compensation parameter(s),
Figure 194467DEST_PATH_IMAGE022
representing splicing operation according to channels;
a4, sending the compensated feature map obtained in the step A3 into a channel-space attention module, and further extracting and refining the features to obtain a red channel feature map
Figure 879395DEST_PATH_IMAGE023
Green channel profile
Figure 572413DEST_PATH_IMAGE024
And blue channel profile
Figure 247020DEST_PATH_IMAGE025
：
Figure 847635DEST_PATH_IMAGE026
Wherein the content of the first and second substances,
Figure 652649DEST_PATH_IMAGE027
representing a channel-space attention module;
a5, for the characteristic diagram obtained in the step A4, the characteristic diagram of the green channel is used for compensating the information of the characteristic diagrams of the other two channels to obtain the characteristic diagram after color correction
Figure 782148DEST_PATH_IMAGE028
、
Figure 408299DEST_PATH_IMAGE029
And
Figure 15867DEST_PATH_IMAGE030
：
Figure 675387DEST_PATH_IMAGE031
wherein the content of the first and second substances,
Figure 710208DEST_PATH_IMAGE032
both represent compensation parameters;
a6, correcting the color of the feature map
Figure 822390DEST_PATH_IMAGE033
And
Figure 239508DEST_PATH_IMAGE034
respectively changing into single-channel characteristic diagrams, and splicing according to the channels to obtain color characteristic diagrams
Figure 753534DEST_PATH_IMAGE035
：
Figure 224836DEST_PATH_IMAGE036
Wherein, the first and the second end of the pipe are connected with each other,
Figure 824313DEST_PATH_IMAGE037
、
Figure 773684DEST_PATH_IMAGE038
and
Figure 136358DEST_PATH_IMAGE039
both represent convolution operations;
a7, sending the color feature map into a convolution module to reconstruct a clear underwater image, namely a final enhanced underwater image
Figure 778561DEST_PATH_IMAGE040
：
Figure 865334DEST_PATH_IMAGE041
Wherein the content of the first and second substances,
Figure 618395DEST_PATH_IMAGE042
which represents a convolution operation, the operation of the convolution,
Figure 841435DEST_PATH_IMAGE043
representing a volume block.
4. The multi-module cascaded underwater image enhancement method according to claim 3, wherein the processing step of the channel-space attention module comprises:
a41, inputting a feature map
Figure 920118DEST_PATH_IMAGE044
By convolution operations
Figure 500048DEST_PATH_IMAGE045
Obtaining a new feature map
Figure 56800DEST_PATH_IMAGE046
；
Figure 399925DEST_PATH_IMAGE047
Wherein the content of the first and second substances,
Figure 383931DEST_PATH_IMAGE048
meaning that the summation is by element,
Figure 445296DEST_PATH_IMAGE044
in particular compensated, obtained in step A3Characteristic diagram
Figure 823318DEST_PATH_IMAGE049
、
Figure 20950DEST_PATH_IMAGE050
And
Figure 175856DEST_PATH_IMAGE051
；
a42, obtaining the characteristic diagram A41
Figure 458939DEST_PATH_IMAGE046
Respectively sending into a channel attention branch CA _ brach and a space attention branch SA _ brach to obtain a channel feature descriptor
Figure 91914DEST_PATH_IMAGE052
And spatial feature descriptors
Figure 415492DEST_PATH_IMAGE053
Then, the feature map is processed
Figure 741300DEST_PATH_IMAGE046
Respectively with channel feature descriptors
Figure 511678DEST_PATH_IMAGE052
And spatial feature descriptors
Figure 948345DEST_PATH_IMAGE053
Element-by-element multiplication to obtain the outputs of CA _ burst and SA _ burst
Figure 120569DEST_PATH_IMAGE054
、
Figure 611419DEST_PATH_IMAGE055
(ii) a Wherein the content of the first and second substances,
Figure 869094DEST_PATH_IMAGE056
represents multiplication by element;
a43, splicing the outputs of CA _ break and SA _ break in the step A42 according to channels and performing convolution operation
Figure 109451DEST_PATH_IMAGE057
Obtaining a final output characteristic diagram after processing
Figure 136182DEST_PATH_IMAGE058
:
Figure 803792DEST_PATH_IMAGE059
Wherein, the first and the second end of the pipe are connected with each other,
Figure 283184DEST_PATH_IMAGE058
in particular to
Figure 333092DEST_PATH_IMAGE060
、
Figure 479908DEST_PATH_IMAGE061
And
Figure 318420DEST_PATH_IMAGE062
。
5. the method of claim 1, wherein the training the multi-module cascade enhancement network comprises:
determining a loss function of the multi-module cascade enhancement network E:
Figure 550687DEST_PATH_IMAGE063
wherein, the first and the second end of the pipe are connected with each other,
Figure 398426DEST_PATH_IMAGE064
representing the loss function originally used by the first stage enhancement network E1,
Figure 417327DEST_PATH_IMAGE065
a function representing the loss of perception is represented,
Figure 692320DEST_PATH_IMAGE066
representing perceptual loss functions
Figure 146304DEST_PATH_IMAGE067
The weight of (c);
determining an initial learning rate of a multi-module cascaded enhanced network E, wherein the initial learning rate of a first stage enhanced network E1
Figure 266575DEST_PATH_IMAGE068
At least one order of magnitude smaller than the initial learning rate set in the original aerial image enhancement network, the initial learning rate of the second stage enhancement network E2
Figure 387984DEST_PATH_IMAGE069
Enhancing the initial learning rate set in the network for the original air image;
and training the multi-module cascade enhancement network E by using the acquired paired underwater image data sets.
CN202210506856.0A 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method Active CN114612347B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210506856.0A CN114612347B (en) 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210506856.0A CN114612347B (en) 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method

Publications (2)

Publication Number Publication Date
CN114612347A true CN114612347A (en) 2022-06-10
CN114612347B CN114612347B (en) 2022-08-16

Family

ID=81870440

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210506856.0A Active CN114612347B (en) 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method

Country Status (1)

Country Link
CN (1) CN114612347B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115170443A (en) * 2022-09-08 2022-10-11 荣耀终端有限公司 Image processing method, shooting method and electronic equipment
CN116797471A (en) * 2022-12-20 2023-09-22 慧之安信息技术股份有限公司 Underwater target image detection method and system based on deep learning

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2020100175A4 (en) * 2020-02-04 2020-04-09 Hu, Wei MR Retinex-based progressive image enhancement method
CN111415304A (en) * 2020-02-26 2020-07-14 中国农业大学 Underwater vision enhancement method and device based on cascade deep network
CN112508812A (en) * 2020-12-01 2021-03-16 厦门美图之家科技有限公司 Image color cast correction method, model training method, device and equipment
CN113034391A (en) * 2021-03-19 2021-06-25 西安电子科技大学 Multi-mode fusion underwater image enhancement method, system and application
CN113256528A (en) * 2021-06-03 2021-08-13 中国人民解放军国防科技大学 Low-illumination video enhancement method based on multi-scale cascade depth residual error network
CN113920021A (en) * 2021-09-27 2022-01-11 海南大学 Underwater image enhancement method based on two-step residual error network

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2020100175A4 (en) * 2020-02-04 2020-04-09 Hu, Wei MR Retinex-based progressive image enhancement method
CN111415304A (en) * 2020-02-26 2020-07-14 中国农业大学 Underwater vision enhancement method and device based on cascade deep network
CN112508812A (en) * 2020-12-01 2021-03-16 厦门美图之家科技有限公司 Image color cast correction method, model training method, device and equipment
CN113034391A (en) * 2021-03-19 2021-06-25 西安电子科技大学 Multi-mode fusion underwater image enhancement method, system and application
CN113256528A (en) * 2021-06-03 2021-08-13 中国人民解放军国防科技大学 Low-illumination video enhancement method based on multi-scale cascade depth residual error network
CN113920021A (en) * 2021-09-27 2022-01-11 海南大学 Underwater image enhancement method based on two-step residual error network

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115170443A (en) * 2022-09-08 2022-10-11 荣耀终端有限公司 Image processing method, shooting method and electronic equipment
CN115170443B (en) * 2022-09-08 2023-01-13 荣耀终端有限公司 Image processing method, shooting method and electronic equipment
CN116797471A (en) * 2022-12-20 2023-09-22 慧之安信息技术股份有限公司 Underwater target image detection method and system based on deep learning

Also Published As

Publication number Publication date
CN114612347B (en) 2022-08-16

Similar Documents

Publication Publication Date Title
CN114612347B (en) Multi-module cascade underwater image enhancement method
CN112233038B (en) True image denoising method based on multi-scale fusion and edge enhancement
CN111127336B (en) Image signal processing method based on self-adaptive selection module
CN111127331B (en) Image denoising method based on pixel-level global noise estimation coding and decoding network
CN110189260B (en) Image noise reduction method based on multi-scale parallel gated neural network
CN101466046A (en) Method and apparatus for removing color noise of image signal
CN112419151A (en) Image degradation processing method, device, storage medium and electronic equipment
CN111986275B (en) Inverse halftoning method for multi-mode halftone image
CN116416561A (en) Video image processing method and device
CN113284070A (en) Non-uniform fog image defogging algorithm based on attention transfer mechanism
EP3451294B1 (en) Luminance-normalised colour spaces
US11823352B2 (en) Processing video frames via convolutional neural network using previous frame statistics
CN117011194A (en) Low-light image enhancement method based on multi-scale dual-channel attention network
CN111476744A (en) Underwater image enhancement method based on classification and atmospheric imaging model
CN111369477A (en) Method for pre-analysis and tool self-adaptation of video recovery task
CN103516959B (en) Image processing method and equipment
CN116433525A (en) Underwater image defogging method based on edge detection function variation model
CN107424134A (en) Image processing method, device, computer-readable recording medium and computer equipment
CN114862711B (en) Low-illumination image enhancement and denoising method based on dual complementary prior constraints
CN115937029A (en) Underwater image enhancement method
TWI736112B (en) Pixel value calibrationmethod and pixel value calibration device
CN115841523A (en) Double-branch HDR video reconstruction algorithm based on Raw domain
CN115311149A (en) Image denoising method, model, computer-readable storage medium and terminal device
CN113762016A (en) Key frame selection method and device
CN117994161B (en) RAW format weak light image enhancement method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant