CN114612347B - Multi-module cascade underwater image enhancement method - Google Patents

Multi-module cascade underwater image enhancement method Download PDF

Info

Publication number
CN114612347B
CN114612347B CN202210506856.0A CN202210506856A CN114612347B CN 114612347 B CN114612347 B CN 114612347B CN 202210506856 A CN202210506856 A CN 202210506856A CN 114612347 B CN114612347 B CN 114612347B
Authority
CN
China
Prior art keywords
network
image
channel
underwater
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210506856.0A
Other languages
Chinese (zh)
Other versions
CN114612347A (en
Inventor
刘红敏
丁艳
樊彬
曾慧
张利欣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology Beijing USTB
Original Assignee
University of Science and Technology Beijing USTB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology Beijing USTB filed Critical University of Science and Technology Beijing USTB
Priority to CN202210506856.0A priority Critical patent/CN114612347B/en
Publication of CN114612347A publication Critical patent/CN114612347A/en
Application granted granted Critical
Publication of CN114612347B publication Critical patent/CN114612347B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a multi-module cascaded underwater image enhancement method, and belongs to the technical field of computer vision. The method comprises the following steps: cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image; acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets; and acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the enhanced underwater image. By adopting the method and the device, the degradation problems of different types in underwater imaging can be solved.

Description

Multi-module cascade underwater image enhancement method
Technical Field
The invention relates to the technical field of computer vision, in particular to an underwater image enhancement method based on multi-module cascade.
Background
In recent years, as a major problem in image enhancement research, underwater image enhancement has received increasing attention from researchers. As an important carrier of ocean information, the underwater image plays a vital role in exploring ocean environment, and reasonably developing and utilizing ocean resources. However, due to the complexity of the underwater imaging environment, the obtained underwater image is often accompanied by degradation problems such as blurring, low contrast, color distortion, poor visibility, and the like, which seriously affects the performance of the task based on underwater vision. Therefore, it is urgently required to improve the quality of underwater images.
In the past decades, many methods have been proposed to improve the quality of underwater images, and these methods can be simply divided into non-learning methods and deep learning based methods. Among the non-learning methods, one is to apply the classical air image enhancement method or its variants (such as histogram equalization, white balance, etc.) directly on the underwater image; the other is a specially designed algorithm aiming at the imaging characteristics of the underwater image or a physical imaging model combined with the underwater image, such as Retinex-based, Fusion-based, GDCP-based and the like. Although these methods improve the quality of underwater images, they are susceptible to degraded image types and have poor generalization capability due to the uncertainty of estimating the physical model parameters and the inaccuracy of a priori knowledge. With the development of deep learning, researchers provide a series of underwater image enhancement methods based on deep learning, such as Water-Net, UIEC2^ Net, Ucolor and the like, which directly model degraded images and clear images, relieve the unsuitability of estimation model parameters and greatly improve the quality of underwater images, but the methods do not consider the attenuation difference between R, G, B channels caused by attenuation related to wavelength, so that color cast exists in the enhanced images, and the methods are still limited by the degradation types of the underwater images, and can not solve the degradation problem existing in the underwater images at the same time. Solving the various degradation problems that coexist in underwater images through a single network remains a significant challenge.
Disclosure of Invention
The embodiment of the invention provides a multi-module cascaded underwater image enhancement method, which can solve the degradation problems of different types in an underwater image. The technical scheme is as follows:
the embodiment of the invention provides a multi-module cascaded underwater image enhancement method, which comprises the following steps:
cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image;
acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets;
and acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the enhanced underwater image.
Further, the step of cascading the existing air image enhancement network with the color correction network to construct a multi-module cascade enhancement network includes:
selecting an existing aerial image enhancement network as a first stage enhancement network E1;
the color correction network is taken as a second-stage enhancement network E2;
and connecting the E1 and the E2 in a residual error mode to obtain the multi-module cascade enhanced network E.
Further, the processing step of the color correction network comprises:
a1, enhancing the output image of the network E1 by the first level
Figure 636057DEST_PATH_IMAGE001
And input image
Figure 415663DEST_PATH_IMAGE002
Obtaining input images of a second-level enhanced network E2 through residual error structural connection
Figure 615526DEST_PATH_IMAGE003
Then extracting the red channel images thereof respectively
Figure 949424DEST_PATH_IMAGE004
Green channel image
Figure 164374DEST_PATH_IMAGE005
And blue channel image
Figure 532907DEST_PATH_IMAGE006
Wherein, in the step (A),
Figure 175110DEST_PATH_IMAGE007
、
Figure 2163DEST_PATH_IMAGE008
respectively representing the height and width of the image,
Figure 755225DEST_PATH_IMAGE009
is a dimension symbol;
a2, for the one obtained in step A1Carrying out convolution operation on the images of the three channels respectively to obtain a red channel characteristic diagram
Figure 712685DEST_PATH_IMAGE010
Green channel profile
Figure 525789DEST_PATH_IMAGE011
And blue channel profile
Figure 834280DEST_PATH_IMAGE012
：
Figure 922190DEST_PATH_IMAGE013
Wherein the content of the first and second substances,
Figure 17315DEST_PATH_IMAGE014
、
Figure 1320DEST_PATH_IMAGE015
and
Figure 797107DEST_PATH_IMAGE016
all show belt
Figure 891971DEST_PATH_IMAGE017
Convolution operations of the layers;
a3, respectively compensating the information of the red channel characteristic diagram and the blue channel characteristic diagram by using the characteristic diagram of the green channel to obtain the compensated red channel characteristic diagram
Figure 89603DEST_PATH_IMAGE018
Green channel profile
Figure 250369DEST_PATH_IMAGE019
And blue channel profile
Figure 533451DEST_PATH_IMAGE020
：
Figure 166427DEST_PATH_IMAGE021
Wherein the content of the first and second substances,
Figure 484145DEST_PATH_IMAGE022
are representative of the compensation parameters that are,
Figure 137849DEST_PATH_IMAGE023
representing splicing operation according to channels;
a4, sending the compensated feature map obtained in the step A3 into a channel-space attention module, and further extracting and refining the features to obtain a red channel feature map
Figure 636789DEST_PATH_IMAGE024
Green channel profile
Figure 604614DEST_PATH_IMAGE025
And blue channel profile
Figure 776838DEST_PATH_IMAGE026
：
Figure 7968DEST_PATH_IMAGE027
Wherein the content of the first and second substances,
Figure 64DEST_PATH_IMAGE028
representing a channel-space attention module;
a5, for the characteristic diagram obtained in the step A4, the characteristic diagram of the green channel is used for compensating the information of the characteristic diagrams of the other two channels to obtain the characteristic diagram after color correction
Figure 246280DEST_PATH_IMAGE029
、
Figure 273011DEST_PATH_IMAGE030
And
Figure 675043DEST_PATH_IMAGE031
：
Figure 420013DEST_PATH_IMAGE032
wherein the content of the first and second substances,
Figure 932903DEST_PATH_IMAGE033
both represent compensation parameters;
a6, correcting the color of the feature map
Figure 79720DEST_PATH_IMAGE034
And
Figure 482731DEST_PATH_IMAGE035
respectively changing into single-channel characteristic diagrams, and splicing according to the channels to obtain color characteristic diagrams
Figure 449418DEST_PATH_IMAGE036
：
Figure 31578DEST_PATH_IMAGE037
Wherein the content of the first and second substances,
Figure 767322DEST_PATH_IMAGE038
、
Figure 42314DEST_PATH_IMAGE039
and
Figure 626792DEST_PATH_IMAGE040
both represent convolution operations;
a7, sending the color feature map into a convolution module to reconstruct a clear underwater image, namely a final enhanced underwater image
Figure 12642DEST_PATH_IMAGE041
：
Figure 602893DEST_PATH_IMAGE042
Wherein the content of the first and second substances,
Figure 783207DEST_PATH_IMAGE043
which represents a convolution operation, the operation of the convolution,
Figure 724487DEST_PATH_IMAGE044
representing a volume block.
Further, the processing step of the channel-space attention module comprises:
a41, inputting a feature map
Figure 642590DEST_PATH_IMAGE045
By convolution operations
Figure 352926DEST_PATH_IMAGE046
Obtaining a new characteristic diagram
Figure 438563DEST_PATH_IMAGE047
；
Figure 867139DEST_PATH_IMAGE048
Wherein the content of the first and second substances,
Figure 594792DEST_PATH_IMAGE049
meaning that the summation is by element,
Figure 165494DEST_PATH_IMAGE045
in particular, the compensated characteristic diagram obtained in step A3
Figure 218770DEST_PATH_IMAGE050
、
Figure 603483DEST_PATH_IMAGE051
And
Figure 134828DEST_PATH_IMAGE052
；
a42, obtaining the characteristic diagram A41
Figure 288598DEST_PATH_IMAGE047
Respectively sending into a channel attention branch CA _ brach and a space attention branch SA _ brach to obtain a channel feature descriptor
Figure 247195DEST_PATH_IMAGE053
And spatial feature descriptors
Figure 402362DEST_PATH_IMAGE054
Then, the feature map is processed
Figure 206239DEST_PATH_IMAGE047
Respectively with channel feature descriptors
Figure 745674DEST_PATH_IMAGE053
Multiplying the spatial feature descriptors by elements to obtain the output of CA _ burst and SA _ burst
Figure 875173DEST_PATH_IMAGE055
、
Figure 500058DEST_PATH_IMAGE056
(ii) a Wherein the content of the first and second substances,
Figure 379064DEST_PATH_IMAGE057
represents multiplication by element;
a43, splicing the outputs of CA _ break and SA _ break in the step A42 according to channels and performing convolution operation
Figure 773005DEST_PATH_IMAGE058
Obtaining a final output characteristic diagram after processing
Figure 807826DEST_PATH_IMAGE059
：
Figure 920008DEST_PATH_IMAGE060
Wherein the content of the first and second substances,
Figure 65687DEST_PATH_IMAGE059
in particular to
Figure 579714DEST_PATH_IMAGE061
、
Figure 513998DEST_PATH_IMAGE062
And
Figure 113475DEST_PATH_IMAGE063
。
further, the training the multi-module cascade enhancement network includes:
determining a loss function of the multi-module cascade enhancement network E:
Figure 328425DEST_PATH_IMAGE064
wherein, the first and the second end of the pipe are connected with each other,
Figure 431379DEST_PATH_IMAGE065
representing the loss function originally used by the first stage enhancement network E1,
Figure 73582DEST_PATH_IMAGE066
the function of the perceptual loss is represented by,
Figure 635056DEST_PATH_IMAGE067
representing perceptual loss functions
Figure 388117DEST_PATH_IMAGE068
The weight of (c);
determining an initial learning rate of a multi-module cascaded enhanced network E, wherein the initial learning rate of a first stage enhanced network E1
Figure 611157DEST_PATH_IMAGE069
At least one amount less than the initial learning rate set in the original aerial image enhancement networkLevel two level enhancing initial learning rate of network E2
Figure 424261DEST_PATH_IMAGE070
Enhancing the initial learning rate set in the network for the original air image;
and training the multi-module cascade enhancement network E by using the acquired paired underwater image data sets.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:
1) the degradation problem of different underwater scenes is considered, the complicated underwater degradation problem is decomposed into different subproblems, and the degradation problems of different types in underwater imaging are solved by cascading different air image enhancement networks.
2) For the difference of the R, G, B channel attenuation, in this embodiment, the G channel with smaller information attenuation adaptively compensates the R channel and the B channel with more serious information attenuation through the color correction network, so as to correct the color of the underwater image.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of an underwater image enhancement method of multi-module cascade connection according to an embodiment of the present invention;
fig. 2 is a schematic view of a workflow of a multi-module cascade enhanced network according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a color correction network according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an enhanced underwater image according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Different from underwater images, the deep learning-based enhancement algorithm designed for air images is relatively mature (such as an image defogging algorithm and a low-illumination image enhancement algorithm), and on the basis, in order to fully utilize the existing research results, reduce the difficulty of network information processing and solve various coexisting degradation problems in the underwater images, the embodiment of the invention provides an end-to-end multi-module cascaded underwater image enhancement method, and the method provides a multi-module cascaded enhancement network.
As shown in fig. 1 and fig. 2, an embodiment of the present invention provides an underwater image enhancement method with multiple modules cascaded, including:
s101, cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image; the method specifically comprises the following steps:
selecting an existing aerial image enhancement network as a first stage enhancement network E1; wherein the aerial image enhancement network comprises: a defogging network GridDehazeNet, a low-illumination enhancement network MIRNet and the like, and network weight parameters are preloaded;
the color correction network is taken as a second-stage enhancement network E2; wherein, the color correction network is proposed in this embodiment;
and connecting the E1 and the E2 in a residual error mode to obtain the multi-module cascade enhanced network E.
In this embodiment, the multi-module cascade enhancement network E includes two parts, the former is the existing air image enhancement network; the latter is a color correction network designed in consideration of the difference of R, G, B channel attenuation, in which a G channel with relatively small information attenuation is used to adaptively compensate an R channel and a B channel with relatively serious information attenuation, thereby correcting the color of the underwater image.
As shown in fig. 3, the processing steps of the color correction network include:
a1, enhancing the output image of the network E1 by the first level
Figure 732752DEST_PATH_IMAGE071
And input image
Figure 23925DEST_PATH_IMAGE072
Connected by a residual structure to obtain an input image of a second-level enhanced network E2
Figure 915787DEST_PATH_IMAGE003
Then extracting the red channel images thereof respectively
Figure 634213DEST_PATH_IMAGE004
Green channel image
Figure 429999DEST_PATH_IMAGE005
And blue channel image
Figure 524863DEST_PATH_IMAGE006
Wherein, in the step (A),
Figure 722495DEST_PATH_IMAGE073
、
Figure 883261DEST_PATH_IMAGE074
respectively representing the height and width of the image,
Figure 431923DEST_PATH_IMAGE009
is a dimension symbol;
a2, performing convolution operation on the three-channel images obtained in the step A1 respectively to obtain red channel feature maps
Figure 64899DEST_PATH_IMAGE010
Green channel profile
Figure 117037DEST_PATH_IMAGE011
And blue channel profile
Figure 177266DEST_PATH_IMAGE012
：
Figure 947645DEST_PATH_IMAGE075
Wherein the content of the first and second substances,
Figure 644031DEST_PATH_IMAGE014
、
Figure 285097DEST_PATH_IMAGE015
and
Figure 781806DEST_PATH_IMAGE076
all show belt
Figure 39481DEST_PATH_IMAGE017
A convolution operation of the layers;
a3, respectively compensating the information of the red channel characteristic diagram and the blue channel characteristic diagram by using the characteristic diagram of the green channel to obtain the compensated red channel characteristic diagram
Figure 14259DEST_PATH_IMAGE018
Green channel profile
Figure 46849DEST_PATH_IMAGE019
And blue channel profile
Figure 714460DEST_PATH_IMAGE020
：
Figure 459431DEST_PATH_IMAGE077
Wherein the content of the first and second substances,
Figure 237900DEST_PATH_IMAGE078
are representative of the compensation parameters that are,
Figure 119137DEST_PATH_IMAGE023
representing splicing operation according to channels;
a4, sending the compensated feature map obtained in the step A3 into a channel-space attention module, and further extracting and refining the features to obtain a red channel feature map
Figure 240806DEST_PATH_IMAGE024
Green channel profile
Figure 473073DEST_PATH_IMAGE025
And blue channel profile
Figure 55233DEST_PATH_IMAGE026
：
Figure 322135DEST_PATH_IMAGE079
Wherein the content of the first and second substances,
Figure 65969DEST_PATH_IMAGE028
representing a channel-space attention module, the processing steps of which include:
a41, inputting a feature map
Figure 519953DEST_PATH_IMAGE045
By convolution operations
Figure 442822DEST_PATH_IMAGE046
Obtaining a new characteristic diagram
Figure 33072DEST_PATH_IMAGE047
:
Figure 213387DEST_PATH_IMAGE080
Wherein the content of the first and second substances,
Figure 154667DEST_PATH_IMAGE049
meaning that the summation is by element,
Figure 78629DEST_PATH_IMAGE045
in particular, the compensated characteristic diagram obtained in step A3
Figure 517527DEST_PATH_IMAGE081
、
Figure 134322DEST_PATH_IMAGE051
And
Figure 562898DEST_PATH_IMAGE052
；
a42, obtaining the characteristic diagram A41
Figure 24972DEST_PATH_IMAGE047
Respectively sending into a channel attention branch CA _ brach and a space attention branch SA _ brach to obtain a channel feature descriptor
Figure 589814DEST_PATH_IMAGE082
And spatial feature descriptors
Figure 117791DEST_PATH_IMAGE054
Then, the feature map is processed
Figure 33663DEST_PATH_IMAGE047
Respectively with channel feature descriptors
Figure 33849DEST_PATH_IMAGE082
And spatial feature descriptors
Figure 718777DEST_PATH_IMAGE054
Element-by-element multiplication to obtain the outputs of CA _ burst and SA _ burst
Figure 411795DEST_PATH_IMAGE055
、
Figure 549385DEST_PATH_IMAGE056
(ii) a Wherein the content of the first and second substances,
Figure 167577DEST_PATH_IMAGE057
represents multiplication by element;
a43, splicing the outputs of CA _ break and SA _ break in the step A42 according to channels and performing convolution operation
Figure 972591DEST_PATH_IMAGE058
Obtaining a final output characteristic diagram after processing
Figure 367669DEST_PATH_IMAGE059
：
Figure 992554DEST_PATH_IMAGE083
Wherein the content of the first and second substances,
Figure 334543DEST_PATH_IMAGE059
in particular to
Figure 265502DEST_PATH_IMAGE061
、
Figure 831481DEST_PATH_IMAGE084
And
Figure 678083DEST_PATH_IMAGE063
。
a5, for the characteristic diagram obtained in the step A4, the characteristic diagram of the green channel is used for compensating the information of the characteristic diagrams of the other two channels to obtain the characteristic diagram after color correction
Figure 89342DEST_PATH_IMAGE029
、
Figure 72210DEST_PATH_IMAGE030
And
Figure 537653DEST_PATH_IMAGE031
：
Figure 137130DEST_PATH_IMAGE085
wherein the content of the first and second substances,
Figure 86500DEST_PATH_IMAGE033
both represent compensation parameters;
a6, correcting the color of the feature map
Figure 455034DEST_PATH_IMAGE086
And
Figure 831657DEST_PATH_IMAGE035
respectively changing into single-channel characteristic diagrams, and splicing according to the channels to obtain color characteristic diagrams
Figure 652852DEST_PATH_IMAGE087
:
Figure 146193DEST_PATH_IMAGE088
Wherein the content of the first and second substances,
Figure 634812DEST_PATH_IMAGE038
、
Figure 916758DEST_PATH_IMAGE039
and
Figure 490827DEST_PATH_IMAGE089
both represent convolution operations;
a7, sending the color feature map into a convolution module to reconstruct a clear underwater image, namely a final enhanced underwater image
Figure 47580DEST_PATH_IMAGE041
：
Figure 877125DEST_PATH_IMAGE090
Wherein the content of the first and second substances,
Figure 861130DEST_PATH_IMAGE043
which represents a convolution operation, the operation of the convolution,
Figure 391337DEST_PATH_IMAGE044
representing a volume block.
S102, acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets, specifically including the following steps:
b1, acquiring paired underwater image data sets; wherein each pair of underwater images comprises: a degraded underwater image and its corresponding reference image;
in this embodiment, a paired underwater image dataset for training the multi-module cascade enhancement network E is constructed from the existing disclosed underwater dataset.
B2, determining a loss function of the multi-module cascade enhancement network E:
Figure 751781DEST_PATH_IMAGE064
wherein the content of the first and second substances,
Figure 683833DEST_PATH_IMAGE091
a loss function representing the original use of the first level enhancement network E1;
Figure 838740DEST_PATH_IMAGE067
representing perceptual loss functions
Figure 862103DEST_PATH_IMAGE066
The weight of (a) is 0.04; wherein the perceptual loss function
Figure 26237DEST_PATH_IMAGE066
Expressed as:
Figure 812796DEST_PATH_IMAGE092
wherein the content of the first and second substances,
Figure 138604DEST_PATH_IMAGE093
and
Figure 908983DEST_PATH_IMAGE094
respectively representing the channel number, height and width of the characteristic diagram, the enhanced underwater image and the corresponding reference image,
Figure 74211DEST_PATH_IMAGE095
the characteristic diagrams of the images in different layers of the VGG-19 are shown, and in the embodiment of the invention, Conv1_2, Conv2_2 and Conv3_3 of the VGG-19 are selected for characteristic extraction.
In this embodiment, the loss function consists of two parts: some are the loss functions originally used by E1
Figure 246435DEST_PATH_IMAGE091
(ii) a Another part is the perceptual loss function
Figure 477565DEST_PATH_IMAGE066
And is used for enabling the image generated by the multi-module cascade enhancement network E and the reference image to be as close as possible in the feature space.
B3, determining the initial learning rate of the multi-module cascade enhanced network E, wherein the initial learning rate of the first-stage enhanced network E1
Figure 735240DEST_PATH_IMAGE069
At least one order of magnitude smaller than the initial learning rate set in the original aerial image enhancement network, the initial learning rate of the second stage enhancement network E2
Figure 975597DEST_PATH_IMAGE070
Enhancing the initial learning rate set in the network for the original air image;
and B4, training the multi-module cascade enhancement network E by using the acquired paired underwater image data sets.
S103, acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into a trained multi-module cascade enhancement network to obtain the enhanced underwater image, wherein the method specifically comprises the following steps:
in this embodiment, an underwater image to be enhanced is acquired
Figure 2328DEST_PATH_IMAGE096
Inputting the underwater image to be enhanced into the trained multi-module cascade enhancement network to obtain the final enhanced underwater image
Figure 410219DEST_PATH_IMAGE097
Fig. 4 shows a schematic diagram of the enhanced underwater image. Therefore, the research result of the existing image enhancement is fully utilized, the proposed color correction network is cascaded with different air image enhancement networks, and different underwater image enhancement tasks are realized, such as the defogging of an underwater image, the enhancement of an underwater low-illumination image and the color correction of the underwater image are realized; meanwhile, the method has stronger generalization capability, can be used for various underwater images with different degradation types, has strong universality and obtains more ideal enhancement effect.
The multi-module cascade underwater image enhancement method provided by the embodiment of the invention at least has the following beneficial effects:
1) the degradation problem of different underwater scenes is considered, the complicated underwater degradation problem is decomposed into different subproblems, and the degradation problems of different types in underwater imaging are solved by cascading different air image enhancement networks.
2) For the difference of the R, G, B channel attenuation, in this embodiment, the G channel with smaller information attenuation adaptively compensates the R channel and the B channel with more serious information attenuation through the color correction network, so as to correct the color of the underwater image.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (3)

1. An underwater image enhancement method based on multi-module cascade is characterized by comprising the following steps:
cascading an existing air image enhancement network and a color correction network to construct a multi-module cascade enhancement network, wherein the air image enhancement network is used for solving the degradation problem similar to an air image in an underwater image, and the color correction network is used for correcting color cast in the underwater image;
wherein, the cascade connection of the existing air image enhancement network and the color correction network to construct a multi-module cascade enhancement network comprises:
selecting an existing aerial image enhancement network as a first stage enhancement network E1;
taking the color correction network as a second-stage enhancement network E2;
connecting the E1 and the E2 in a residual error mode to obtain a multi-module cascade enhanced network E;
acquiring paired underwater image data sets, and training the multi-module cascade enhancement network by using the acquired paired underwater image data sets;
acquiring an underwater image to be enhanced, and sending the underwater image to be enhanced into a trained multi-module cascade enhancement network to obtain an enhanced underwater image;
wherein, the processing steps of the color correction network comprise:
a1, enhancing the output image of the network E1 by the first level
Figure DEST_PATH_IMAGE001
And input image
Figure 630577DEST_PATH_IMAGE002
Connected by a residual structure to obtain an input image of a second-level enhanced network E2
Figure DEST_PATH_IMAGE003
Then extracting the red channel images thereof respectively
Figure 916065DEST_PATH_IMAGE004
Green channel image
Figure DEST_PATH_IMAGE005
And blue channel image
Figure 59602DEST_PATH_IMAGE006
Wherein, in the step (A),
Figure DEST_PATH_IMAGE007
respectively representing the height and width of the image,
Figure 114145DEST_PATH_IMAGE008
is a dimension symbol;
a2, performing convolution operation on the three-channel images obtained in the step A1 respectively to obtain red channel feature maps
Figure DEST_PATH_IMAGE009
Green channel profile
Figure 708769DEST_PATH_IMAGE010
And blue channel profile
Figure DEST_PATH_IMAGE011
：
Figure 481553DEST_PATH_IMAGE012
Wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE013
all show belt
Figure 163201DEST_PATH_IMAGE014
A convolution operation of the layers;
a3, respectively compensating the information of the red channel characteristic diagram and the blue channel characteristic diagram by using the characteristic diagram of the green channel to obtain the compensated red channel characteristic diagram
Figure DEST_PATH_IMAGE015
And blue channel profile
Figure 337831DEST_PATH_IMAGE016
And carrying out convolution and splicing operation on the characteristic diagram of the green channel to obtain the characteristic diagram of the green channel
Figure DEST_PATH_IMAGE017
：
Figure 79522DEST_PATH_IMAGE018
Wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE019
are representative of the compensation parameters that are,
Figure 339602DEST_PATH_IMAGE020
representing splicing operation according to channels;
a4, sending the compensated feature map obtained in the step A3 into a channel-space attention module, and further extracting and refining the features to obtain a red channel feature map
Figure DEST_PATH_IMAGE021
Green channel profile
Figure 824941DEST_PATH_IMAGE022
And blue channel profile
Figure DEST_PATH_IMAGE023
：
Figure 854077DEST_PATH_IMAGE024
Wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE025
representing a channel-space attention module;
a5, for the characteristic diagram obtained in the step A4, the characteristic diagram of the green channel is used for compensating the information of the characteristic diagrams of the other two channels to obtain the characteristic diagram after color correction
Figure 796363DEST_PATH_IMAGE026
And
Figure DEST_PATH_IMAGE027
：
Figure 153526DEST_PATH_IMAGE028
wherein, the first and the second end of the pipe are connected with each other,
Figure DEST_PATH_IMAGE029
all represent compensation parameters, a feature map of the color corrected green channel
Figure 567190DEST_PATH_IMAGE030
；
A6, correcting the color of the feature map
Figure DEST_PATH_IMAGE031
Respectively changing into single-channel characteristic diagrams, and splicing according to the channels to obtain color characteristic diagrams
Figure 326198DEST_PATH_IMAGE032
：
Figure DEST_PATH_IMAGE033
Wherein the content of the first and second substances,
Figure 799905DEST_PATH_IMAGE034
both represent convolution operations;
a7, color feature map
Figure 503419DEST_PATH_IMAGE036
Sending the data to a convolution module to reconstruct a clear underwater image, namely a final enhanced underwater image
Figure DEST_PATH_IMAGE037
：
Figure 596140DEST_PATH_IMAGE038
Wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE039
which represents a convolution operation, the operation of the convolution,
Figure 973769DEST_PATH_IMAGE040
representing a volume block.
2. The multi-module cascaded underwater image enhancement method according to claim 1, wherein the processing step of the channel-space attention module comprises:
a41, inputting a feature map
Figure DEST_PATH_IMAGE041
By convolution operations
Figure 618377DEST_PATH_IMAGE039
Obtaining a new characteristic diagram
Figure 809187DEST_PATH_IMAGE042
；
Figure DEST_PATH_IMAGE043
Wherein the content of the first and second substances,
Figure 440020DEST_PATH_IMAGE044
meaning that the summation is by element,
Figure 767096DEST_PATH_IMAGE041
in particular, the compensated characteristic diagram obtained in step A3
Figure DEST_PATH_IMAGE045
、
Figure 457971DEST_PATH_IMAGE046
；
A42, obtaining the characteristic diagram A41
Figure 136077DEST_PATH_IMAGE042
Respectively sending into a channel attention branch CA _ brach and a space attention branch SA _ brach to obtain a channel feature descriptor
Figure DEST_PATH_IMAGE047
And spatial feature descriptors
Figure 695235DEST_PATH_IMAGE048
Then, the feature map is mapped
Figure 283342DEST_PATH_IMAGE042
Respectively with channel feature descriptors
Figure 738594DEST_PATH_IMAGE047
And spatial feature descriptors
Figure 903996DEST_PATH_IMAGE048
Element-by-element multiplication to obtain the outputs of CA _ burst and SA _ burst
Figure DEST_PATH_IMAGE049
(ii) a Wherein the content of the first and second substances,
Figure 375166DEST_PATH_IMAGE050
represents multiplication by element;
a43, splicing the outputs of CA _ break and SA _ break in the step A42 according to channels and performing convolution operation
Figure 942414DEST_PATH_IMAGE039
Obtaining a final output characteristic diagram after processing
Figure DEST_PATH_IMAGE051
：
Figure 709513DEST_PATH_IMAGE052
Wherein the content of the first and second substances,
Figure 362211DEST_PATH_IMAGE051
in particular to
Figure DEST_PATH_IMAGE053
。
3. The method of claim 1, wherein the training the multi-module cascade enhancement network comprises:
determining loss function of multi-module cascade enhanced network E
Figure 263171DEST_PATH_IMAGE054
：
Figure DEST_PATH_IMAGE055
Wherein the content of the first and second substances,
Figure 560291DEST_PATH_IMAGE056
representing the loss of original usage of the first stage enhancement network E1The function of the function is that of the function,
Figure DEST_PATH_IMAGE057
the function of the perceptual loss is represented by,
Figure 888504DEST_PATH_IMAGE058
representing perceptual loss functions
Figure 638286DEST_PATH_IMAGE057
The weight of (c);
determining an initial learning rate of a multi-module cascaded enhanced network E, wherein the initial learning rate of a first stage enhanced network E1
Figure DEST_PATH_IMAGE059
At least one order of magnitude smaller than the initial learning rate set in the original aerial image enhancement network, the initial learning rate of the second stage enhancement network E2
Figure 608516DEST_PATH_IMAGE060
Enhancing the initial learning rate set in the network for the original air image;
and training the multi-module cascade enhancement network E by using the acquired paired underwater image data sets.
CN202210506856.0A 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method Active CN114612347B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210506856.0A CN114612347B (en) 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210506856.0A CN114612347B (en) 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method

Publications (2)

Publication Number Publication Date
CN114612347A CN114612347A (en) 2022-06-10
CN114612347B true CN114612347B (en) 2022-08-16

Family

ID=81870440

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210506856.0A Active CN114612347B (en) 2022-05-11 2022-05-11 Multi-module cascade underwater image enhancement method

Country Status (1)

Country Link
CN (1) CN114612347B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115170443B (en) * 2022-09-08 2023-01-13 荣耀终端有限公司 Image processing method, shooting method and electronic equipment
CN116797471A (en) * 2022-12-20 2023-09-22 慧之安信息技术股份有限公司 Underwater target image detection method and system based on deep learning

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2020100175A4 (en) * 2020-02-04 2020-04-09 Hu, Wei MR Retinex-based progressive image enhancement method
CN111415304A (en) * 2020-02-26 2020-07-14 中国农业大学 Underwater vision enhancement method and device based on cascade deep network
CN112508812A (en) * 2020-12-01 2021-03-16 厦门美图之家科技有限公司 Image color cast correction method, model training method, device and equipment
CN113034391B (en) * 2021-03-19 2023-08-08 西安电子科技大学 Multi-mode fusion underwater image enhancement method, system and application
CN113256528B (en) * 2021-06-03 2022-05-27 中国人民解放军国防科技大学 Low-illumination video enhancement method based on multi-scale cascade depth residual error network
CN113920021A (en) * 2021-09-27 2022-01-11 海南大学 Underwater image enhancement method based on two-step residual error network

Also Published As

Publication number Publication date
CN114612347A (en) 2022-06-10

Similar Documents

Publication Publication Date Title
CN114612347B (en) Multi-module cascade underwater image enhancement method
CN112233038B (en) True image denoising method based on multi-scale fusion and edge enhancement
CN111898701B (en) Model training, frame image generation and frame insertion methods, devices, equipment and media
WO2022105638A1 (en) Image degradation processing method and apparatus, and storage medium and electronic device
WO2018188260A1 (en) Image display control method and device, and display screen control system
CN111127336A (en) Image signal processing method based on self-adaptive selection module
CN110189260B (en) Image noise reduction method based on multi-scale parallel gated neural network
CN101466046A (en) Method and apparatus for removing color noise of image signal
CN111127331A (en) Image denoising method based on pixel-level global noise estimation coding and decoding network
CN116416561A (en) Video image processing method and device
EP3451294B1 (en) Luminance-normalised colour spaces
CN112508812A (en) Image color cast correction method, model training method, device and equipment
US11823352B2 (en) Processing video frames via convolutional neural network using previous frame statistics
CN110717864B (en) Image enhancement method, device, terminal equipment and computer readable medium
CN113781318A (en) Image color mapping method and device, terminal equipment and storage medium
CN110047038B (en) Single-image super-resolution reconstruction method based on hierarchical progressive network
CN103516959B (en) Image processing method and equipment
US20210044303A1 (en) Neural network acceleration device and method
CN114862711B (en) Low-illumination image enhancement and denoising method based on dual complementary prior constraints
CN116433525A (en) Underwater image defogging method based on edge detection function variation model
TWI736112B (en) Pixel value calibrationmethod and pixel value calibration device
CN115841523A (en) Double-branch HDR video reconstruction algorithm based on Raw domain
CN111754412A (en) Method and device for constructing data pairs and terminal equipment
CN104284207B (en) Information transmission method based on video image
CN107392871A (en) Image defogging method, device, mobile terminal and computer-readable recording medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant