CN116152120B - Low-light image enhancement method and device integrating high-low frequency characteristic information - Google Patents
Low-light image enhancement method and device integrating high-low frequency characteristic information Download PDFInfo
- Publication number
- CN116152120B CN116152120B CN202310426401.2A CN202310426401A CN116152120B CN 116152120 B CN116152120 B CN 116152120B CN 202310426401 A CN202310426401 A CN 202310426401A CN 116152120 B CN116152120 B CN 116152120B
- Authority
- CN
- China
- Prior art keywords
- low
- light image
- channel
- branch
- component
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000005286 illumination Methods 0.000 claims abstract description 69
- 238000000605 extraction Methods 0.000 claims abstract description 16
- 238000000354 decomposition reaction Methods 0.000 claims description 33
- 238000010586 diagram Methods 0.000 claims description 23
- 238000011176 pooling Methods 0.000 claims description 12
- 230000004913 activation Effects 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 7
- 230000009977 dual effect Effects 0.000 claims description 6
- 239000004973 liquid crystal related substance Substances 0.000 claims description 6
- 238000012935 Averaging Methods 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 230000003213 activating effect Effects 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 5
- 230000002708 enhancing effect Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000002310 reflectometry Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 235000008733 Citrus aurantifolia Nutrition 0.000 description 1
- 235000011941 Tilia x europaea Nutrition 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 239000004571 lime Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G06T5/90—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/54—Extraction of image or video features relating to texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a low-light image enhancement method and device integrating high-low frequency characteristic information, and belongs to the field of computer vision and image processing. The method comprises the following steps: s1, collecting a normal light-low light image pair in an RGB format; s2, decomposing the collected low-light image into illumination components I low And a reflection component R low The method comprises the steps of carrying out a first treatment on the surface of the S3, decomposing the obtained illumination component and reflection component into a three-level Laplacian pyramid; s4, sequentially inputting three-level Laplacian pyramid images of the illumination component and the reflection component into three branch networks to obtain the enhanced illumination componentAnd a reflection componentThe method comprises the steps of carrying out a first treatment on the surface of the S5, pairAndperforming channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement; the reasoning steps include the steps S2-S5 described above. The invention utilizes the Laplace multi-scale feature extraction block LRMSDA under the double-channel attention to realize high-quality low-light image enhancement capable of effectively suppressing noise and enhancing texture details.
Description
Technical Field
The invention relates to a low-light image enhancement method and device integrating high-low frequency characteristic information, belonging to the field of computer vision and image processing.
Background
The low-light image is shot by the imaging equipment under dim illumination condition, and the insufficient illumination causes the problems of poor visibility, low contrast, unexpected noise and the like of the image, so that the performance of a plurality of computer vision systems designed for normal light images is further impaired. Therefore, as a low-level visual task, it is particularly important to convert a low-light image into a normally exposed high-quality image.
The conventional method is mainly divided into two types when processing a dim light image, the first type is based on a Histogram Equalization (HE) technique, and the second type is based on the Retinex theory, and the image modeling is decomposed into two components, namely an illumination component and a reflection component. While conventional approaches may yield good results in some cases, they are still limited by reflection and illumination decomposition models, i.e., it is difficult to design constraint models that can be applied to image decomposition of various scenes.
In recent years, due to the rich possibilities of generating models based on Deep Neural Networks (DNNs), the problem of low-light image enhancement has been re-expressed as an image-to-image conversion problem, i.e. the enhancement results can be restored without any physical assumptions. While DNN-based models greatly improve the visual quality of the recovered results, they continue to suffer from deficiencies in the processing of high frequency texture features.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a low-light image enhancement method and device for fusing high-low frequency characteristic information, wherein the low-frequency illumination characteristic and the high-frequency texture characteristic are separated by multi-scale deep stacking Laplace, a Laplace multi-scale characteristic extraction block LRMSDA (a Laplace multi-scale characteristic extraction block Laplacian Residual Multi-Scale feature extraction block based on Dual channel Attention based on the dual-channel attention) based on the dual-channel attention is used on UNet, the perception and fusion of the low-frequency illumination characteristic and the high-frequency texture information by a network are further enhanced, and noise can be effectively suppressed and texture details can be enhanced in low-light image enhancement.
The technical scheme adopted by the invention is as follows:
the low-light image enhancement method integrating the high-low frequency characteristic information is characterized by comprising the following steps of:
s1: collecting a normal light-low light image pair in RGB format, and recording the image information as、/>；
S2: for the collected low-light imageDecomposition into illumination component I low And a reflection component R low ；
S3: the illumination component I and the reflection component R obtained by decomposition are further decomposed into three-level Laplacian pyramids which are respectively marked as、/>；
S4: three-level Laplacian pyramid for respectively dividing illumination component I and reflection component R、Is input into three U-Net branch networks N1, N2 and N3 with LRMSDA modules to obtain enhanced illumination component ∈>And reflection component->；
S5: for the enhanced illumination componentAnd reflection component->And performing channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement.
In one embodiment, the S2 is using classical Retinex-Net decomposition. The classical Retinex-Net decomposition network is a two-branch fully convolutional neural network ending in a Sigmoid layer, which are used to estimate the illumination component I and the reflection component R of the normal illumination map and the low-light image, respectively.
In one embodiment, the illumination component I obtained by the decomposition in the step S3 low And a reflection component R low The formula for further decomposition into a three-level laplacian pyramid is:、the method comprises the steps of carrying out a first treatment on the surface of the Wherein I is k、 R k Downsampling of the illumination component I or the reflection component R, respectively, representing the original input is adjusted to +.>A version of the scale; />Representing double up-sampling of the image.
In one embodiment, the three U-Net branching networks N1, N2, N3 in S4 include an encoder, an LRMSDA module, and a decoder. The encoder and the decoder are of classical U-Net structure.
In one embodiment, the LRMSDA module includes a laplace multiscale feature extraction unit, a dual attention unit.
In one embodiment, the laplacian multi-scale feature extraction unit first further decomposes the potential features into 3 laplacian residual sequences, using a different number of convolution layers for each sequence to extract multi-scale context information.
In one embodiment, the dual attention unit includes a channel attention branch and a spatial attention branch. The channel attention branch first maps the feature mapEncoding global context by global averaging pooling to obtain feature descriptor +.>The method comprises the steps of carrying out a first treatment on the surface of the Feature descriptor->Further generating an activation operator +.>The method comprises the steps of carrying out a first treatment on the surface of the Finally use the activator->Recalibrating the input profile +.>Get the output of channel attention branch +.>The method comprises the steps of carrying out a first treatment on the surface of the The spatial attention branch first of all subjects the feature map +.>Independently applying global average pooling and maximum pooling operations along the channel dimension and concatenating the outputs to form a feature map +.>For characteristic diagram->Convolution and Sigmoid activation are performed to obtain a spatial attention map +.>Subsequently using the spatial attention map +.>Recalibrating the input profile +.>Obtaining output of spatial attention branchesThe method comprises the steps of carrying out a first treatment on the surface of the Finally the output of the channel attention branch +.>Output from spatial attention branch->After channel splicing, the channel is subjected to convolution layer and then is combined with the original input characteristic diagram +.>The addition operation is performed as the final output of the dual-attention unit.
In one embodiment, the U-Net network loss function with LRMSDA module in S4 is designed as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing reconstruction loss->、/>Respectively through step S4 and corresponding normal light image/>Decomposition into illumination component I normal And a reflection component R normal The resulting output of the enhanced combinations in the k-th Laplacian pyramid obtained later, the loss calculated at each pyramid level is calculated by T k Normalization, T k Representation->Total number of pixels in>Indicating a loss of l 2; />Indicating the Laplace penalty in order to preserve the local sharpness of the enhanced result,/for>、/>Representing the Laplace residual image and the normal light image restored by step S4, respectively +.>Decomposition into illumination component I normal And a reflection component R normal Residual images in the kth Laplacian pyramid layer;
representing color loss in order to ensure that the color vector consisting of R, G, B channels of predicted enhancement results is consistent with the direction of the corresponding normal light image, where H k And W is k The height and width of the enhancement result of the kth Laplacian pyramid, i, respectively, is +.>Is>Representing the vector inner product; w (w) d 、w l 、w c Representing the weight value of each loss item.
In one embodiment, the enhanced three-level residual image is in S4(k=1, 2, 3) performing cross-pyramid layer stepwise combination to obtain an enhanced image pyramid, denoted +.>(k=1, 2, 3); the process is represented as follows:wherein->To finally enhance the image.
In one embodiment, the reasoning step includes steps S2-S5 described above.
The invention also provides a low-light image enhancement device fusing the high-low frequency characteristic information, which comprises the following modules:
the image acquisition module is used for acquiring a normal light-low light image pair;
the image decomposition module is used for decomposing the low-light image acquired by the image acquisition module into an illumination component and a reflection component;
the image decomposition module is used for decomposing the illumination component and the reflection component into three-level Laplacian pyramids;
the low light enhancement network module is used for inputting the three-level Laplacian pyramid images of the illumination component and the reflection component and gradually combining the three-level Laplacian pyramid images across pyramid layers to obtain the enhanced illumination componentAnd reflection component->；
The module is reconfigured by a reconfiguration module,for combining the enhanced illumination componentAnd reflection component->And performing channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement.
The low-light image enhancement device integrating the high-low frequency characteristic information can implement the low-light image enhancement method integrating the high-low frequency characteristic information.
The invention has the following beneficial effects:
(1) The invention fully considers the property of the low-light image, and combines the Retinex theory of the traditional method with the deep neural network to recover the normally exposed high-quality image.
(2) The Laplacian pyramid is utilized in the image space and the feature space to respectively adjust the low-frequency global illumination and recover the high-frequency local detail. The application of the laplacian pyramid to the feature space makes optimization based on rich links between higher order residuals in a multi-scale structure easier, so that convergence is stabilized with relatively little loss in the network training phase.
(3) Aiming at the task of low-light image enhancement, an improvement is provided on the traditional U-Net structure, an additional Laplace multi-scale feature extraction block LRMSDA based on double-channel attention is added at the encoder stage and the decoder stage, and the context information in images with different resolutions is enriched.
Drawings
FIG. 1 is a schematic diagram of a low-light image enhancement device incorporating high-low frequency characteristic information according to the present invention;
FIG. 2 is a schematic flow chart of a low-light image enhancement method integrating high-low frequency characteristic information;
FIG. 3 is a schematic diagram of the overall structure of a low-light image enhancement network incorporating high-low frequency characteristic information according to the present invention;
FIG. 4 is a schematic diagram of a U-Net network architecture with LRMSDA modules of the present invention;
fig. 5 is a schematic structural diagram of the LRMSDA module of the present invention.
Detailed Description
The following describes the scheme of the invention in detail with reference to the accompanying drawings.
As shown in fig. 1, the present embodiment provides a low-light image enhancement device that merges high-low frequency characteristic information, including the following modules:
the image acquisition module is used for acquiring normal light-low light image pairs in the same scene, the obtained image format is PNG, and the arrangement mode is RGB;
the image decomposition module is used for decomposing the acquired low-light image into an illumination component and a reflection component;
image decomposition module based on Laplace residual error for decomposing illumination component and reflection component into three-level Laplace pyramid、/>。
A low light enhancement network module for imaging three-level Laplacian pyramid、/>Respectively performing low light enhancement recovery and gradually combining to obtain enhanced illumination component +.>And reflection component->；
A reconstruction module for reconstructing the enhanced illumination componentAnd reflection component->And performing channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement.
As shown in fig. 2, the present embodiment provides a low-light image enhancement method for fusing high-low frequency characteristic information, which includes the following steps:
s1: collecting a normal light-low light image pair in RGB format, wherein the normal light image has the characteristics of high detail, high contrast, high definition and the like, and the image information is respectively recorded as、/>。
S2: for the collected low-light imageDecomposition into illumination component I using classical Retinex-Net low And a reflection component R low The method comprises the steps of carrying out a first treatment on the surface of the Wherein the Retinex-Net network is a two-branch full convolutional neural network ending with a Sigmoid layer, and the two branches are used for estimating illumination component I and reflection component R of the normal illumination image and the low-light image respectively, which are marked as (">、) And (/ ->、/>). It consists of 4 Conv-ReLUs stacked with 1 Conv-Sigmoid, with input size ofIs output with a size of +.>Is of the illumination component of (1)And size is +.>Is included in the reflection component of the (c). The Retinex-Net network loss function is:
wherein the method comprises the steps of
For reconstruction loss, N represents the number of pixels,representing the reflection component and the illumination component of the normal illumination image at pixel i point,/and>representing a reflection component and an illumination component of the low-light image at a pixel i point, wherein the loss function represents a pixel-by-pixel square error between the reconstructed image and the real image after decomposition; />For reflectivity loss, the loss function constrains reflectivity consistency according to Retinex theory; />An initializing a priori condition for reflectivity, wherein +.>Representing taking normal illumination map->Maximum value between RGB three color channels;for color loss, the color vector used to ensure the RGB channel composition of the predicted reflection component R has the same direction as the corresponding real mapWhere k=1, 2,3 stands for RGB three channel, +.>Respectively representing the weight values of the loss functions.
S3: the illumination component I and the reflection component R obtained by decomposition are further decomposed into three-level Laplacian pyramids which are respectively marked as、/>. For the illumination component I obtained by decomposition low And a reflection component R low The formula for further decomposition into a three-level laplacian pyramid is: />、/>. Wherein I is k、 R k Respectively represent the original input illumination component I low And a reflection component R low Image downsampling is adjusted to +.>Version of the scale. />Representing double up-sampling of the image component using a bilinear sampling mode through the torch. Nn. Functional. Interface API.
S4: three-level Laplacian pyramid for respectively dividing illumination component I and reflection component R、The enhanced illumination component is sequentially input into three U-Net branch networks N1, N2 and N3>And reflection component->. For the branch network N1, first of all +.>And->Input into the enhanced U-Net network with LRMSDA module>And->Subsequently->And->Performing bilinear upsampling to obtain +.>And->And then toAnd->Performing Laplace operation to obtain ∈>And->And is in charge of>、/>Channel splicing is performed to transfer the coarser level information to the next branch for finer feature reconstruction, and finally for +.>And (3) withPerforming bilinear upsampling and Laplacian operation again to obtain +.>And->. For the branch network N2, input +.>And->、/>And->The characteristic diagram after channel splicing is enhanced +.>And->Subsequently->And->Performing bilinear upsampling to obtain +.>And->. For branch N3, input +.>、、/>And +.>、/>、/>The characteristic diagram after channel splicing is enhanced +.>And->. For post-enhancement->、/>、/>、/>、/>、/>Pyramid gradual combination is carried out to obtain enhanced illumination component +.>And reflection componentQuantity->。
Specifically, the LRMSDA module comprises a Laplacian multi-scale feature extraction unit and a dual-attention unit.
The laplacian multi-scale feature extraction unit first further decomposes the potential features into 3 laplacian residual sequences, using a different number of convolution layers for each sequence to extract multi-scale context information.
In one embodiment, the dual attention unit includes a channel attention branch and a spatial attention branch. The channel attention branch first maps the feature mapEncoding global context by global averaging pooling to obtain feature descriptor +.>The method comprises the steps of carrying out a first treatment on the surface of the Feature descriptor->Further generating an activation operator +.>The method comprises the steps of carrying out a first treatment on the surface of the Finally use the activator->Recalibrating the input profile +.>Get the output of channel attention branch +.>The method comprises the steps of carrying out a first treatment on the surface of the The spatial attention branch first of all subjects the feature map +.>Independently applying global average pooling and maximum pooling operations along the channel dimension and concatenating the outputs to formFeature map->For characteristic diagram->Convolution and Sigmoid activation are performed to obtain a spatial attention map +.>Subsequently using the spatial attention map +.>Recalibrating the input profile +.>Obtaining output of spatial attention branchesThe method comprises the steps of carrying out a first treatment on the surface of the Finally the output of the channel attention branch +.>Output from spatial attention branch->After channel splicing, the channel is subjected to convolution layer and then is combined with the original input characteristic diagram +.>The addition operation is performed as the final output of the dual-attention unit.
The U-Net network loss function with LRMSDA module in S4 is designed as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing reconstruction loss->、/>Respectively, through step S4 and the corresponding normal light image +.>Decomposition into illumination component I normal And a reflection component R normal The resulting output of the enhanced combinations in the k-th Laplacian pyramid obtained later, the loss calculated at each pyramid level is calculated by T k Normalization, T k Representation->Total number of pixels in>Indicating a loss of l 2; />Indicating the Laplace penalty in order to preserve the local sharpness of the enhanced result,/for>、/>Representing the Laplace residual image and the normal light image restored by step S4, respectively +.>Decomposition into illumination component I normal And a reflection component R normal Residual images in the kth Laplacian pyramid layer;
representing color loss in order to ensure that the color vector consisting of R, G, B channels of predicted enhancement results is consistent with the direction of the corresponding normal light image, where H k And W is k The height and width of the enhancement result of the kth Laplacian pyramid, i, respectively, is +.>Is>Representing the vector inner product; w (w) d 、w l 、w c Representing the weight value of each loss item.
The enhanced three-level residual image in the S4(k=1, 2, 3) performing cross-pyramid layer stepwise combination to obtain an enhanced image pyramid, denoted +.>(k=1, 2, 3); the process is represented as follows: />Wherein->To finally enhance the image.
S5: for the enhanced illumination componentAnd reflection component->And performing channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement.
As shown in fig. 4, the present embodiment provides a U-Net network structure with LRMSDA modules, which is composed of an encoder, LRMSDA modules, and a decoder. The system specifically comprises 4 downsampling layers of an encoder, 4 upsampling layers of a decoder and 7 LRMSDA modules. Classical U-Net model first uses an encoder with 4 downsampling layers to sizeIs encoded to a size of +.>Is a feature map of (1); then a decoder with 4 upsampling layers will be dimensioned +.>Is decoded to a size +.>(c=3) or (c=1) the illumination component. The U-Net network structure with the LRMSDA module inputs the characteristic diagram into the LRMSDA module before each downsampling and downsampling, and the input and output characteristic diagram of the LRMSDA module are consistent in size.
As shown in fig. 5, the embodiment provides an LRMSDA module, which includes a laplace multiscale feature extraction unit and a dual-attention unit. The Laplace multi-scale feature extraction unit decomposes the input feature map into three-level Laplace pyramids, and the decomposition formula is consistent with the step S3. And (3) carrying out convolution operation of which the convolution kernels are 3*3 on the three-level Laplacian pyramid feature images from a large scale to a small scale to obtain three output feature images, and carrying out double up-sampling and addition operation to reconstruct the three output feature images into one feature image. The feature map is sequentially input to a channel attention branch and a space attention branch after passing through two convolution layers and a PReLU activation function. The channel attention branch firstly carries Out parallel global average pooling and global maximum pooling operation on an input feature diagram, then carries Out channel splicing on the output of the input feature diagram and the output of the input feature diagram, and finally carries Out multiplication operation on the input feature diagram and the feature diagram of the original input channel attention branch sequentially through a convolution layer and a Sigmoid activation function to obtain the output Out1 of the channel attention branch. The spatial attention branch performs global average pooling operation on the input feature diagram, and multiplies the output obtained by the convolution layer, the PReLU activation function, the convolution layer and the Sigmoid activation function by the feature diagram of the original input spatial attention branch to obtain the output Out2 of the spatial attention branch. And finally, carrying Out channel splicing on Out1 and Our2, then carrying Out addition operation on the channel spliced Out1 and Our and the characteristic diagram before Laplacian decomposition to obtain the output of the final LRMSDA module.
Quantitative experiment detection: in the invention, a parallel experiment is carried out on a LOL data set on a classical model which is commonly used for enhancing low-light images to verify the image processing effect of the invention. Peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM), which are commonly used in the field of image enhancement, are adopted as evaluation indexes. The PSNR is used for comparing the required signal intensity with the intensity of background noise, and the larger the value is, the smaller the image noise is, and the higher the image quality is; the SSIM reflects the similarity between two images, the higher the SSIM value, indicating that the two images are more similar. Wherein the comparison involved a classical LIME, retinex-Net, enlightenGAN, MBLLEN model, the results of which are shown in Table 1.
TABLE 1
In summary, the method provided in this embodiment combines the conventional Retinex theory with the deep neural network, separates the low-frequency illumination feature and the high-frequency texture feature by using the multi-scale deep stacked laplace, and further enhances the perception and fusion of the network to the low-frequency illumination feature and the high-frequency texture information by using the laplace multi-scale feature extraction block LRMSDA based on the dual-channel attention on UNet, thereby realizing high-quality low-light image enhancement capable of effectively suppressing noise and enhancing texture details.
The above description is only a specific embodiment of the present invention, and is not intended to limit the present invention in any way. It should be noted that the normal light image capturing device used does not limit the present invention, the image resolution does not limit the present invention, and the image content does not limit the present invention. The scope of the present invention is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the technical scope of the present invention, and it is intended to cover the scope of the present invention.
Claims (10)
1. The low-light image enhancement method integrating the high-low frequency characteristic information is characterized by comprising the following steps of:
s1: collecting normal light in RGB formatLow-light image pairs, the image information is respectively recorded as、/>；
S2: for the collected low-light imageDecomposition into illumination component I low And a reflection component R low ；
S3: for the illumination component I obtained by decomposition low And a reflection component R low Further decomposed into three-level Laplacian pyramids respectively marked as、/>；
S4: respectively divide the illumination component I low And a reflection component R low Three-level Laplacian pyramid of (2)、/>Is input into three U-Net branch networks N1, N2, N3 with LRMSDA modules, for branch network N1, first +.>And->Input into the enhanced U-Net network with LRMSDA module>And->Subsequently->And->Performing bilinear upsampling to obtain +.>And->Further to->And->Performing Laplace operation to obtain ∈>And (3) withAnd is in charge of>、/>Channel splicing is performed to transfer the coarser level information to the next branch for finer feature reconstruction, and finally for +.>And->Performing bilinear upsampling and Laplacian operation again to obtain +.>And (3) withThe method comprises the steps of carrying out a first treatment on the surface of the For the branch network N2, input +.>And->、/>And->The characteristic diagram after channel splicing is enhanced +.>And->Subsequently->And->Performing bilinear upsampling to obtain +.>And->The method comprises the steps of carrying out a first treatment on the surface of the For branch N3, input、/>、/>And +.>、/>、/>The characteristic diagram after channel splicing is enhanced +.>And->The method comprises the steps of carrying out a first treatment on the surface of the For post-enhancement->、/>、/>、/>、/>、/>Pyramid gradual combination is carried out to obtain enhanced illumination component +.>And reflection component->The method comprises the steps of carrying out a first treatment on the surface of the Three U-Net branchesThe branch networks N1, N2 and N3 comprise encoders, LRMSDA modules and decoders; the LRMSDA module is a Laplace multi-scale feature extraction block based on double-channel attention, namely Laplacian Residual Multi-Scale feature extraction block based on Dual channel Attention; the LRMSDA module comprises a Laplace multi-scale feature extraction unit and a double-attention unit; the dual attention unit includes a channel attention branch and a spatial attention branch;
s5: for the enhanced illumination componentAnd reflection component->And performing channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement.
2. The low light image enhancement method according to claim 1, wherein said S2 is using classical Retinex-Net decomposition; the classical Retinex-Net decomposition network is a two-branch fully convolutional neural network ending in a Sigmoid layer, which are used to estimate the illumination component I and the reflection component R of the normal illumination map and the low-light image, respectively.
3. The low light image enhancement method according to claim 1, wherein the illumination component I obtained by the decomposition in S3 low And a reflection component R low The formula for further decomposition into a three-level laplacian pyramid is:、/>the method comprises the steps of carrying out a first treatment on the surface of the Wherein I is k、 R k Downsampling of the illumination component I or the reflection component R, respectively, representing the original input is adjusted to +.>A version of the scale; />Representing double up-sampling of the image.
4. The low light image enhancement method according to claim 1, wherein the laplacian multi-scale feature extraction unit first further decomposes the latent features into 3 laplacian residual sequences, using a different number of convolution layers for each sequence to extract multi-scale context information.
5. The low light image enhancement method according to claim 1, wherein the channel attention branch first maps featuresEncoding global context by global averaging pooling to obtain feature descriptor +.>The method comprises the steps of carrying out a first treatment on the surface of the Feature descriptor->Further generating an activation operator +.>The method comprises the steps of carrying out a first treatment on the surface of the Finally use the activator->Recalibrating the input profile +.>Get the output of channel attention branch +.>。
6. The low light image enhancement method according to claim 1, wherein the spatial attention branch first maps featuresIndependently applying global average pooling and maximum pooling operations along the channel dimension and concatenating the outputs to form a feature map +.>For characteristic diagram->Convolving and Sigmoid activating to obtain spatial attention mapSubsequently using the spatial attention map +.>Recalibrating the input profile +.>Obtain the output of the spatial attention branch +.>The method comprises the steps of carrying out a first treatment on the surface of the Finally the output of the channel attention branch +.>Output from spatial attention branch->After channel splicing, the channel is subjected to convolution layer and then is combined with the original input characteristic diagram +.>The addition operation is performed as the final output of the dual-attention unit.
7. The low light image enhancement method according to claim 1, wherein the U-Net network loss function with LRMSDA module in S4 is designed as follows:
；
wherein, the liquid crystal display device comprises a liquid crystal display device,representing reconstruction loss->Representing the enhanced combination result output in the kth laplacian pyramid obtained by step S4,/>Representing normal light image +.>Decomposition into illumination component I normal And a reflection component R normal The resulting output of the enhanced combinations in the k-th Laplacian pyramid obtained later, the loss calculated at each pyramid level is calculated by T k Normalization, T k Representation->Total number of pixels in>Indicating a loss of l 2; />Indicating the Laplace penalty in order to preserve the local sharpness of the enhanced result,/for>、/>Representing the Laplace residual image and the normal light image restored by step S4, respectively +.>Decomposition into illumination component I normal And a reflection component R normal Residual images in the kth Laplacian pyramid layer;
representing color loss in order to ensure that the color vector consisting of R, G, B channels of predicted enhancement results is consistent with the direction of the corresponding normal light image, where H k And W is k The height and width of the enhancement result of the kth Laplacian pyramid, i, respectively, is +.>Is>Representing the vector inner product; w (w) d 、w l 、w c Representing the weight value of each loss item.
8. The low light image enhancement method according to claim 7, wherein the enhanced three-level residual image is subjected to the step S4(k=1, 2, 3) performing cross-pyramid layer stepwise combination to obtain an enhanced image pyramid, denoted +.>(k=1, 2, 3); the process is represented as follows: />Wherein->For final enhancement of the image->Representing double up-sampling of the image component using a bilinear sampling mode through the torch. Nn. Functional. Interface API.
9. The low-light image enhancement device integrating the high-low frequency characteristic information is characterized by comprising the following modules:
the image acquisition module is used for acquiring a normal light-low light image pair;
the image decomposition module is used for decomposing the low-light image acquired by the image acquisition module into an illumination component and a reflection component;
the image decomposition module is used for decomposing the illumination component and the reflection component into three-level Laplacian pyramids;
the low light enhancement network module is used for inputting the three-level Laplacian pyramid images of the illumination component and the reflection component and gradually combining the three-level Laplacian pyramid images across pyramid layers to obtain the enhanced illumination componentAnd reflection component->Wherein the illumination components I are respectively low And a reflection component R low Third-level Laplacian pyramid->、/>Is input into three U-Net branch networks N1, N2, N3 with LRMSDA modules, for branch network N1, first +.>And->Input into the enhanced U-Net network with LRMSDA module>And->Subsequently->And->Performing bilinear upsampling to obtain +.>And (3) withFurther to->And->Performing Laplace operation to obtain ∈>And->And is in charge of>、/>Channel splicing is performed to transfer the coarser level information to the next branch for finer feature reconstruction, and finally for +.>And->Performing bilinear upsampling and Laplacian operation again to obtain +.>And->The method comprises the steps of carrying out a first treatment on the surface of the For the branch network N2, input +.>And->、/>And->The characteristic diagram after channel splicing is enhanced +.>And->Subsequently->And (3) withPerforming bilinear upsampling to obtain +.>And->The method comprises the steps of carrying out a first treatment on the surface of the For branch N3, input +.>、/>、/>And +.>、/>、/>The characteristic diagram after channel splicing is enhanced +.>And->The method comprises the steps of carrying out a first treatment on the surface of the For post-enhancement->、/>、/>、、/>、/>Pyramid gradual combination is carried out to obtain enhanced illumination component +.>And reflection component->；
A reconstruction module for reconstructing the enhanced illumination componentAnd reflection component->And performing channel-by-channel pixel-by-pixel multiplication operation to obtain a normal light image after low light enhancement.
10. The low light image enhancement device according to claim 9, wherein the low light image enhancement device is capable of implementing the low light image enhancement method according to any one of claims 2-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310426401.2A CN116152120B (en) | 2023-04-20 | 2023-04-20 | Low-light image enhancement method and device integrating high-low frequency characteristic information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310426401.2A CN116152120B (en) | 2023-04-20 | 2023-04-20 | Low-light image enhancement method and device integrating high-low frequency characteristic information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116152120A CN116152120A (en) | 2023-05-23 |
| CN116152120B true CN116152120B (en) | 2023-07-21 |
Family
ID=86374025
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310426401.2A Active CN116152120B (en) | 2023-04-20 | 2023-04-20 | Low-light image enhancement method and device integrating high-low frequency characteristic information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116152120B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117011194B (en) * | 2023-10-07 | 2024-01-30 | 暨南大学 | Low-light image enhancement method based on multi-scale dual-channel attention network |
| CN117372307B (en) * | 2023-12-01 | 2024-02-23 | 南京航空航天大学 | Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method |
| CN117496160B (en) * | 2023-12-29 | 2024-03-19 | 中国民用航空飞行学院 | Indoor scene-oriented semantic segmentation method for low-illumination image shot by unmanned aerial vehicle |
| CN117893413A (en) * | 2024-03-15 | 2024-04-16 | 博创联动科技股份有限公司 | Vehicle-mounted terminal man-machine interaction method based on image enhancement |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113256510A (en) * | 2021-04-21 | 2021-08-13 | 浙江工业大学 | CNN-based low-illumination image enhancement method with color restoration and edge sharpening effects |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111047515B (en) * | 2019-12-29 | 2024-01-09 | 兰州理工大学 | Attention mechanism-based cavity convolutional neural network image super-resolution reconstruction method |
| CN112927160B (en) * | 2021-03-12 | 2022-11-18 | 郑州轻工业大学 | Single low-light image enhancement method based on depth Retinex |
| CN114266707A (en) * | 2021-11-24 | 2022-04-01 | 重庆理工大学 | Low-light image enhancement method combining attention mechanism and Retinex model |
| CN115063318A (en) * | 2022-06-29 | 2022-09-16 | 深圳市安软慧视科技有限公司 | Adaptive frequency-resolved low-illumination image enhancement method and related equipment |
| CN115393227B (en) * | 2022-09-23 | 2023-06-06 | 南京大学 | Low-light full-color video image self-adaptive enhancement method and system based on deep learning |
| CN115841438A (en) * | 2022-10-24 | 2023-03-24 | 中国科学院长春光学精密机械与物理研究所 | Infrared image and visible light image fusion method based on improved GAN network |
-
2023
- 2023-04-20 CN CN202310426401.2A patent/CN116152120B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113256510A (en) * | 2021-04-21 | 2021-08-13 | 浙江工业大学 | CNN-based low-illumination image enhancement method with color restoration and edge sharpening effects |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116152120A (en) | 2023-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116152120B (en) | Low-light image enhancement method and device integrating high-low frequency characteristic information | |
| Yang et al. | Scale-free single image deraining via visibility-enhanced recurrent wavelet learning | |
| Engin et al. | Cycle-dehaze: Enhanced cyclegan for single image dehazing | |
| Yang et al. | Deep edge guided recurrent residual learning for image super-resolution | |
| Guo et al. | Dense scene information estimation network for dehazing | |
| CN111028150B (en) | Rapid space-time residual attention video super-resolution reconstruction method | |
| CN111915530B (en) | End-to-end-based haze concentration self-adaptive neural network image defogging method | |
| Huang et al. | Deep hyperspectral image fusion network with iterative spatio-spectral regularization | |
| CN111161360B (en) | Image defogging method of end-to-end network based on Retinex theory | |
| CN110232653A (en) | The quick light-duty intensive residual error network of super-resolution rebuilding | |
| CN108269244B (en) | Image defogging system based on deep learning and prior constraint | |
| CN111179167A (en) | Image super-resolution method based on multi-stage attention enhancement network | |
| CN113139898B (en) | Light field image super-resolution reconstruction method based on frequency domain analysis and deep learning | |
| CN111784582A (en) | DEC-SE-based low-illumination image super-resolution reconstruction method | |
| CN112001843B (en) | Infrared image super-resolution reconstruction method based on deep learning | |
| Shen et al. | Convolutional neural pyramid for image processing | |
| Li et al. | Underwater image high definition display using the multilayer perceptron and color feature-based SRCNN | |
| CN113284061B (en) | Underwater image enhancement method based on gradient network | |
| CN111340744A (en) | Attention double-flow deep network-based low-quality image down-sampling method and system | |
| CN112767283A (en) | Non-uniform image defogging method based on multi-image block division | |
| CN114187275A (en) | Multi-stage and multi-scale attention fusion network and image rain removing method | |
| CN114820408A (en) | Infrared and visible light image fusion method based on self-attention and convolutional neural network | |
| CN115526779A (en) | Infrared image super-resolution reconstruction method based on dynamic attention mechanism | |
| Zheng et al. | Depth image super-resolution using multi-dictionary sparse representation | |
| CN117274059A (en) | Low-resolution image reconstruction method and system based on image coding-decoding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |