CN114638749A - Low-illumination image enhancement model, method, electronic device and storage medium - Google Patents

Low-illumination image enhancement model, method, electronic device and storage medium Download PDF

Info

Publication number
CN114638749A
CN114638749A CN202210135560.2A CN202210135560A CN114638749A CN 114638749 A CN114638749 A CN 114638749A CN 202210135560 A CN202210135560 A CN 202210135560A CN 114638749 A CN114638749 A CN 114638749A
Authority
CN
China
Prior art keywords
illumination
layer
initialization
module
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210135560.2A
Other languages
Chinese (zh)
Inventor
王旭
翁键
邬文慧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to CN202210135560.2A priority Critical patent/CN114638749A/en
Publication of CN114638749A publication Critical patent/CN114638749A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Processing (AREA)

Abstract

The invention is suitable for the technical field of image processing, and provides a low-illumination image enhancement model, a method, an electronic device and a storage medium, wherein the low-illumination image enhancement model comprises an initialization module, an optimization module, an illumination adjustment module and an image reconstruction module which are sequentially connected, the initialization module is used for carrying out initialization decomposition on an input image to obtain an initialization illumination layer and an initialization reflection layer, the optimization module is used for carrying out alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm to obtain an optimization illumination layer and an optimization reflection layer, the illumination adjustment module is used for carrying out illumination adjustment on the optimization illumination layer to obtain a target illumination layer, the image reconstruction module is used for carrying out image reconstruction according to the target illumination layer and the optimization reflection layer to obtain a target illumination image, so that the flexibility and the interpretability of the low-illumination image enhancement model are ensured, the robustness of the low-illumination image enhancement model is improved.

Description

Low-illumination image enhancement model, method, electronic device and storage medium
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a low-illumination image enhancement model, a low-illumination image enhancement method, electronic equipment and a storage medium.
Background
Visual perception is one of the most important ways for human beings to know the world, and with the rapid development of society, the demand of people for information is larger and larger, and in real application, unpredictable situations often occur to cause image quality degradation. In these degradation situations, low-light images are always one of the hot issues of computer society interest, especially in the fields of urban traffic, surveillance video, medical assistance, and the like. The low-illumination image is usually due to insufficient illumination intensity or too short exposure time, so that the whole pixel intensity of the acquired image is relatively small, the contrast is relatively low, a large amount of details are invisible, and the visual experience and the algorithm processing accuracy are seriously influenced.
In the existing low-illumination image enhancement technology, although most of deep learning algorithms are combined with the traditional Retinex theory and a deep learning method, a network is adopted for direct solution in the process of solving intermediate variables, and complex prior hypothesis and an iteration flow in traditional image processing are omitted. Although the deep learning method has the advantage of fast reasoning speed, the development of the deep learning technology is limited to a certain extent due to the lack of interpretability. In order to accelerate the convergence rate of the traditional iterative optimization algorithm, as early as 2010, a famous learner, Lecun et al, designs an iterative solution frame integrated with deep learning, and the frame has the idea that a nonlinear operator and a linear matrix of each iteration in the traditional iterative algorithm can be respectively replaced by an activation function and a neural network of a deep learning neighborhood, and the whole optimization process is more adaptive to the real distribution of data in a data-driven mode without depending on a specific prior assumption, so that the method has stronger robustness and obtains better effect under the condition of less iteration times. After that, such an algorithm (hereinafter, an unfolding algorithm is used instead) which expands an iterative process is gradually applied to the neighborhood of image super-resolution reconstruction, image denoising, pulse compression and the like. Recently, Liu et al applied the unfolding algorithm to low-illumination enhancement for illumination estimation and denoising processing. This work, while providing a new solution for unsupervised low-light enhancement, strips the relationship of light and noise resulting in the risk of excessive smoothing and noise retention of the image after enhancement, and furthermore, the image enhancement effect is not ideal since the model is easily overexposed in most cases, ignoring the reflective layer.
Disclosure of Invention
The invention aims to provide a low-illumination image enhancement model, a low-illumination image enhancement method, electronic equipment and a storage medium, and aims to solve the problem that the image enhancement effect in the prior art is not ideal.
In one aspect, the present invention provides a low-light image enhancement model, which includes an initialization module, an optimization module, a light adjustment module, and an image reconstruction module, which are connected in sequence,
the initialization module is used for performing initialization decomposition on an input image to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image;
the optimization module is used for performing a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm to obtain an optimization illumination layer and an optimization reflection layer;
the illumination adjusting module is used for adjusting illumination of the optimized illumination layer to obtain a target illumination layer;
and the image reconstruction module is used for reconstructing an image according to the target illumination layer and the optimized reflection layer to obtain a target illumination image.
Preferably, the initialization module is a fully connected neural network.
Preferably, the fully-connected neural network is a full convolutional neural network comprising 4 convolutional layers.
Preferably, the loss function adopted by the initialization module during training includes a fidelity term and a prior term, the fidelity term is used for measuring the closeness of an initialization image composed of an initialization illumination layer and an initialization reflection layer of a training sample and the training sample, and the prior term is used for measuring the closeness of the initialization illumination layer of the training sample and the R, G, B three-channel maximum value of the training sample.
Preferably, the loss function adopted by the initialization module during training is as follows:
Figure BDA0003504364810000021
wherein L isinitRepresenting the loss of the initialization module, I representing the training sample, R0An initialisation reflecting layer, L, representing the training sample0Represents the initialization illumination layer of the training sample, μ is a constant, and R, G, B represents the red, green, and blue channels, respectively.
Preferably, the optimization module includes a variable computation sub-network, a reflection layer repair network, and an illumination layer repair network, when performing current-time alternate iterative optimization, the variable computation sub-network is configured to compute a first intermediate variable and a second intermediate variable after current-time iterative optimization, the reflection layer repair network is configured to obtain an optimized reflection layer after current-time iterative optimization based on the first intermediate variable and the second intermediate variable after current-time iterative optimization, and the illumination layer repair network is configured to obtain an optimized illumination layer after current-time iterative optimization based on the second intermediate variable after current-time iterative optimization.
Preferably, the variable quantity operator network is configured to calculate the first intermediate variable and the second intermediate variable after the current iterative optimization by using a least square method.
Preferably, the reflection layer repair network is configured to perform convolution operation on the first intermediate variable and the second intermediate variable after the current iterative optimization to obtain a first intermediate feature map and a second intermediate feature map, perform cascade operation on the first feature map and the second intermediate feature map to obtain a spliced feature map, perform channel attention calculation on the spliced feature map by using a channel attention mechanism to obtain a re-weighted feature map, obtain noise distribution of the re-weighted feature map, and obtain the optimized reflection layer after the current iterative optimization based on the noise distribution and the first intermediate feature map.
Preferably, the illumination adjustment module comprises an adjustment factor expansion submodule, a splicing submodule and a brightness adjustment network which are connected in sequence, wherein,
the adjustment factor expansion submodule is used for expanding a preset adjustment scale factor into a matrix with the same size as the optimized illumination layer;
the splicing submodule is used for splicing the matrix and the optimized illumination layer to obtain a splicing result;
and the brightness adjusting network is used for adjusting the brightness of the optimized illumination layer based on the splicing result to obtain the target illumination layer.
Preferably, the network structure of the brightness adjustment network is the same as that of the initialization module, and the size of the convolution kernel of the convolution layer of the brightness adjustment network is larger than that of the convolution kernel of the convolution layer of the initialization module.
Preferably, the loss function used in the brightness adjustment network training includes one or more combinations of a gradient level fidelity term, a color level fidelity term, and a structure level fidelity term, where the gradient level fidelity term is used to measure a horizontal or vertical gradient distance between an optimized illumination layer and a target illumination layer of a training sample, the color level fidelity term is used to measure a reconstruction loss between a target illumination image and a reference image of the training sample, and the structure level fidelity term is used to measure a distance between the target illumination image and the reference image of the training sample.
Preferably, the loss function adopted in the brightness adjustment network training is as follows:
Figure BDA0003504364810000041
wherein L isadjustRepresenting the loss of the brightness adjustment network,
Figure BDA0003504364810000042
in the horizontal or vertical direction representing an optimized illumination layer of the training sampleThe gradient of the gradient is changed,
Figure BDA0003504364810000043
representing the gradient of the target illumination layer of the training sample in the horizontal or vertical direction, IrefRepresenting the reference image, R representing an optimized reflective layer of the training sample,
Figure BDA0003504364810000044
representing a target illumination layer of the training sample, SSIM representing an image quality loss function.
In another aspect, the present invention provides a low-light image enhancement method based on the above low-light image enhancement model, including the following steps:
performing initialization decomposition on an input image through the initialization module to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image;
performing a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm through the optimization module to obtain an optimization illumination layer and an optimization reflection layer;
performing illumination adjustment on the optimized illumination layer through the illumination adjustment module to obtain a target illumination layer;
and reconstructing an image according to the target illumination layer and the optimized reflection layer through the image reconstruction module to obtain a target illumination image.
In another aspect, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method when executing the computer program.
In another aspect, the present invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method as described above.
The low-illumination image enhancement model comprises an initialization module, an optimization module, an illumination adjustment module and an image reconstruction module which are sequentially connected, wherein the initialization module is used for carrying out initialization decomposition on an input image to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image, the optimization module is used for carrying out a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an undermining algorithm to obtain an optimization illumination layer and an optimization reflection layer, the illumination adjustment module is used for carrying out illumination adjustment on the optimization illumination layer to obtain a target illumination layer, and the image reconstruction module is used for carrying out image reconstruction according to the target illumination layer and the optimization reflection layer to obtain a target illumination image.
Drawings
Fig. 1A is a schematic structural diagram of a low-light image enhancement model according to an embodiment of the present invention;
fig. 1B is a schematic diagram illustrating an operation principle of a reflective layer repair network according to an embodiment of the present invention;
fig. 1C is a schematic diagram illustrating an operating principle of a low-illumination image enhancement model according to an embodiment of the present invention;
FIG. 2A is a visualization result of an SICE data set providing enhancement effects of 10 different low-light image enhancement models according to a second embodiment of the present invention;
fig. 2B is a visualization result of enhancement effects of 10 different low-light image enhancement models provided by the second embodiment of the present invention on an LOL data set;
FIG. 3 is a flowchart of an implementation of a low-light image enhancement method according to a third embodiment of the present invention; and
fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The following detailed description of specific implementations of the present invention is provided in conjunction with specific embodiments:
the first embodiment is as follows:
fig. 1A illustrates a structure of a low-illumination image enhancement model according to an embodiment of the present invention, where the low-illumination image enhancement model includes an initialization module 11, an optimization module 12, an illumination adjustment module 13, and an image reconstruction module 14, which are connected in sequence; the initialization module is used for performing initialization decomposition on an input image to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image; the optimization module is used for performing a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm to obtain an optimization illumination layer and an optimization reflection layer; the illumination adjusting module is used for performing illumination adjustment on the optimized illumination layer to obtain a target illumination layer; the image reconstruction module is used for reconstructing an image according to the target illumination layer and the optimized reflection layer to obtain a target illumination image. The input image is a low-light image to be subjected to image enhancement.
For the initialization module, considering that variable initialization plays an important role for iterative optimization algorithms (e.g., ADMM), the initialization module may perform initialization decomposition on the input image by using the commonly used all-zero initialization decomposition and random initialization. In order to provide the correct direction for the subsequent optimization by the initialized decomposition, therefore, the initialized illumination layer and the initialized reflection layer should contain important information of the input image, assuming that the reflection layer is an RGB three-channel image and three channels share the same illumination layer, the three-channel maximum value of the input image can be directly assigned to the illumination layer as the initialized illumination layer, and according to Retinex theory, the color of the object is composed of the reflectivity of the surface and the illumination intensity falling on the surface, so that the following imaging expression is provided:
I＝R·L (1)
where I denotes an input image, R denotes a reflective layer of the input image, and L denotes an illumination layer of the input image, and thus, the initialized reflective layer can be obtained based on the Retinex theory described above.
However, the above-mentioned initialization decomposition method will enlarge the difference of three channel values in the reflective layer, and further destroy the statistical properties of R, G, and B. Therefore, the initialization module preferably adopts a fully-connected network to realize the initialization decomposition of the illumination layer and the reflection layer, so that the initialization illumination layer and the initialization reflection layer contain more information, and meanwhile, the statistical properties of R, G and B are prevented from being damaged. Further preferably, the fully-connected neural network is a full convolution neural network including 4 convolution layers, so as to improve the operation efficiency of the low-illumination image enhancement model.
Specifically, the input of the initialization module is a low-illumination image, the output is an initialization illumination layer and an initialization reflection layer corresponding to the input image, and the abstract representation is as follows:
Figure BDA0003504364810000071
wherein R is0Denotes an initialization reflective layer, L0It is indicated that the illumination layer is initialized,
Figure BDA0003504364810000072
indicating an initialization module, I indicating an input image, thetaDIndicating the parameters of the initialization module.
For the purpose of preserving effective information of the input image by the initialization decomposition, preferably, the loss function adopted by the initialization module in training includes a fidelity term and a prior term, so that the initialization illumination layer and the initialization reflection layer after the initialization decomposition preserve effective information of the input image. The fidelity term is used for measuring the closeness degree of an initialization image formed by an initialization illumination layer and an initialization reflection layer of a training sample and the training sample so as to ensure that the initialization decomposition meets the Retinex theory, and the priori term is used for measuring the closeness degree of the initialization illumination layer of the training sample and the R, G, B three-channel maximum value of the training sample so as to enable the initialization illumination layer to learn richer structural information. The training sample is an input image during low-illumination image enhancement model training.
Preferably, the fidelity term in the loss function adopted during the training of the initialization module is calculated by using a norm L1, and the prior term is calculated by using a norm L2, so as to ensure the training effect of the initialization module, and enable the illumination layer and the reflection layer after the initialization decomposition to retain the effective information of the input image as much as possible. The loss function formula adopted during the training of the initialization module is as follows:
Figure BDA0003504364810000073
wherein L isinitDenotes the loss of the initialization module, I denotes the training samples (input images during model training), R0Initialized reflective layer, L, representing training samples0Represents the initial illumination layer of the training sample, μ is a constant, and R, G, B represents the red, green, and blue channels, respectively.
For the optimization module, according to the Retinex theory, the decomposition problem of the imaging expression (1) is a ill-conditioned problem, and therefore prior model constraints R and L need to be established when solving the problem, which are expressed as:
Figure BDA0003504364810000074
where Φ (R) and Ψ (L) represent a priori assumptions for the reflective layer R and the illumination layer L, respectively. To solve the problem, two new variables are introduced, namely a first intermediate variable P and a second intermediate variable Q, and equation (4) can then be rewritten as:
Figure BDA0003504364810000081
so far, formula (5) can be reasonably divided into fidelity terms and prior terms for independent solution, and the following expression is obtained:
Figure BDA0003504364810000082
Figure BDA0003504364810000083
Figure BDA0003504364810000084
Figure BDA0003504364810000085
wherein, PkAnd QkRepresents the result of the fidelity term solution, R, at the kth iterationkAnd LkThe solution of the prior term at the kth iteration is shown. When k is 1, PkAnd QkIndicating an initialization reflective layer and an initialization reflective layer illumination layer.
It is considered that the initialized reflective layer tends to be saturated with noise, severely affecting the presentation of important details. The initialization lighting layer not only completely retains the structural information, but also leaves redundant texture details to a great extent. Therefore, the purpose of the optimization module in this embodiment is to repair the initialized reflective layer, so that effective details can be completely retained while removing noise. The satisfactory illumination layer should be structurally complete and smooth in texture detail, and in order to achieve the above purpose, the fidelity terms and the prior terms in the above equations (6) - (9) can be alternately and iteratively optimized by using an unfolding algorithm based on the framework of the deep neural network. Based on the comprehensive consideration of time and performance, the number of times of alternate iterative optimization performed by the optimization module is preferably 3, so as to determine the number of times of alternate iterative optimization according to the actual experimental effect.
Preferably, the optimization module comprises a variable calculation sub-network, a reflection layer repair network and an illumination layer repair network, when current-time alternate iteration optimization is performed, the variable calculation sub-network is used for calculating a first intermediate variable and a second intermediate variable after current-time iteration optimization, the reflection layer repair network is used for obtaining an optimized reflection layer after current-time iteration optimization based on the first intermediate variable and the second intermediate variable after current-time iteration optimization, and the illumination layer repair network is used for obtaining an optimized illumination layer after current-time iteration optimization based on the second intermediate variable after current-time iteration optimization to replace the traditional optimized illumination layer based on current-time iteration optimizationManual prior solution of the Retinex optimization algorithm enables the optimization module to learn more robust prior information from the data. The reflection layer repairing network and the illumination layer repairing network are constructed on the basis of a neural network. Preferably, the variable quantity operator network is used for calculating the first intermediate variable and the second intermediate variable after the current iteration optimization by adopting a least square method so as to realize the calculation of the intermediate variables. In specific implementation, when the kth iterative optimization is performed, Q obtained according to the previous (k-1) iterative optimization can be firstly obtainedk-1、Rk-1Solving the above equation (6) to obtain a first intermediate variable P of the current iteration optimizationkThen fixing the first intermediate variable P optimized in the current iterationkOptimized illumination layer L optimized with previous iterationk-1Solving the above formula (7) to obtain a second intermediate variable Q of the current iteration optimizationkThe above equations (6) - (7) are understood as a classical least squares problem, so that the following closed-loop solution can be obtained by derivation:
Figure BDA0003504364810000091
Figure BDA0003504364810000092
wherein k is more than or equal to 1, k is a positive integer, and lambda is a constant.
In the formation of PkAnd QkThereafter, the prior term R is learnedkAnd Lk。
According to experiments, the noise on the reflecting layer and the brightness distribution on the illuminating layer are highly correlated, and the noise is less when the brightness is higher, and vice versa. Preferably, the reflection layer repairing network is configured to perform convolution operation on the calculated feature map of the first intermediate variable and the second intermediate variable to obtain a first intermediate feature map and a second intermediate feature map, perform fusion processing on the first intermediate feature map and the second intermediate feature map to obtain a fusion feature map, and then obtain an optimized reflection layer after the current iteration based on the noise distribution of the fusion feature map and the first intermediate feature map, so that the reflection layer is repaired by combining information of the illumination layer, denoising of the reflection layer is achieved, and a learning effect of the optimized reflection layer is improved.
Further preferably, the reflection layer repair network is configured to perform convolution operation on the first intermediate variable and the second intermediate variable after the current iteration optimization to obtain a first intermediate feature map and a second intermediate feature map, perform cascade operation on the first feature map and the second intermediate feature map to obtain a spliced feature map, perform channel attention calculation on the spliced feature map by using a channel attention mechanism to obtain a re-weighted feature map, obtain noise distribution of the re-weighted feature map, and obtain the optimized reflection layer after the current iteration optimization based on the noise distribution and the first intermediate feature map.
Optimizing the reflective layer R at the time of the kth alternate iterative optimization in equation (8)kThe abstract representation is:
Figure BDA0003504364810000101
wherein, thetaRRepresenting a reflective layer repair network
Figure BDA0003504364810000102
Parameter of (A), PkFirst intermediate variable, Q, representing the kth alternate iterative optimizationkA second intermediate variable representing the kth alternate iteration optimization.
Optimizing the illumination layer L during the kth alternate iterative optimization in equation (9)kThe abstract representation is: :
Figure BDA0003504364810000103
wherein, thetaLRepresenting lighting layer repair networks
Figure BDA0003504364810000104
The parameter (c) of (c).
Fig. 1B is a schematic diagram of the working principle of the reflective layer repair network in this embodiment. In fig. 1B, the reflective layer repair network includes a channel self-attention module and a noise extraction module, Conv represents convolution, circle C represents cascade operation (coordination), Average position represents Average pooling, FC represents full connection, circle x represents pixel multiplication (Element-wise multiplication), and circle-represents denoising, and the reflective layer repair network operates according to the principle that a first intermediate variable P is first processedkAnd a second intermediate variable QkPerforming convolution operation to obtain a first intermediate feature map M1 and a second intermediate feature map M2, then performing cascade operation on the first intermediate feature map M1 and the second intermediate feature map M2 through the cascade operation to obtain a spliced feature map M3, performing channel attention calculation on a spliced feature map M3 through a channel self-attention module to obtain a re-weighted feature map M4, obtaining a noise distribution M5 of the re-weighted feature map M4 through a noise extraction module, and denoising the first intermediate feature map M1 based on the noise distribution M5 to obtain an optimized reflection layer M6 after current iteration optimization.
For the above illumination adjustment module, in the Retinex theory, the reflective layer is an inherent property of an object and does not change with the illumination condition. Based on the theory, the low-illumination imaging is caused by the lower intensity of the illumination layer, so that after the optimized illumination layer and the optimized reflection layer are obtained, the optimized reflection layer can be fixed and unchanged, and the optimized illumination layer is adjusted. The illumination adjustment module may adjust illumination of the illumination layer by using a gamma correction (gamma correction) technique, which can achieve different brightness enhancement effects by adjusting parameters, but it is difficult to determine an adjustment scale factor by adjusting the scale. Preferably, the illumination adjustment module is configured to perform illumination adjustment on the optimized illumination layer according to a preset adjustment scale factor, where the adjustment scale factor is specified by a user, so as to generate the target illumination layer according to a brightness adjustment scale specified by the user. In specific implementation, the input of the illumination adjustment module is an optimized illumination layer and a user-specified adjustment scale factor, and the output is a high-light illumination layer under a target adjustment scale, namely a target illumination layer.
Preferably, the illumination adjustment module includes an adjustment factor expansion submodule, a splicing submodule and a brightness adjustment network which are connected in sequence, wherein the adjustment factor expansion submodule is used for expanding a preset adjustment scale factor into a matrix with the same size as the optimized illumination layer, the splicing submodule is used for splicing the matrix and the optimized illumination layer to obtain a splicing result, and the brightness adjustment network is used for adjusting the brightness of the optimized illumination layer based on the splicing result to obtain a target illumination layer so as to adjust the illumination intensity. In a specific implementation, the scaling factor is expanded into a matrix with the same size as the illumination layer, and then the matrix is spliced with the optimized illumination layer to serve as an input of the electrical network after the brightness is adjusted, which can be specifically expressed as:
Figure BDA0003504364810000111
where ω denotes the scale factor of the adjustment, θARepresenting a brightness adjustment network
Figure BDA0003504364810000112
L denotes the optimized illumination layer.
Preferably, the brightness adjustment network has the same network structure as the initialization module, that is, the brightness adjustment network may be a full convolution neural network including 4 convolution layers, and the convolution kernel of the convolution layer of the brightness adjustment network has a larger size than that of the convolution kernel of the convolution layer of the initialization module, so as to maintain consistency and limit smoothness of the illumination layer.
In order to train the brightness adjustment network better, considering that the high-brightness illumination layer output by the brightness adjustment network should be consistent with the input low-brightness illumination layer in structure, preferably, the loss function adopted in the training of the brightness adjustment network includes a gradient level fidelity term to ensure the training effect of the brightness adjustment network. And the gradient layer fidelity item is used for measuring the horizontal or vertical gradient distance between the optimized illumination layer and the target illumination layer of the training sample.
In order to reconstruct an image of normal illumination based on the target illumination layer, it is preferable that the loss function used in the training of the brightness adjustment network includes a color-level fidelity term to further improve the training effect of the brightness adjustment network. The color-level fidelity item is used for measuring the reconstruction loss of a target illumination image and a reference image of a training sample, so that the reconstructed image is a normal illumination image, namely the reference image.
In order to make the reconstructed image consistent with the reference image in structure, brightness and contrast, the loss function adopted in the training of the brightness adjustment network preferably includes one or more combinations of color-level fidelity terms and structure-level fidelity terms, so as to further improve the training effect of the brightness adjustment network. The structure-level fidelity item is used for measuring the distance between a target illumination image and a reference image of a training sample, so that the reconstructed image is consistent with the reference image in structure, brightness and contrast.
Preferably, the loss function adopted in the lighting adjustment module training includes a gradient level fidelity term, a color level fidelity term and a structure level fidelity term, so as to further improve the training effect of the brightness adjustment network through the constraints of three aspects.
Further preferably, the gradient level fidelity term is calculated by using a norm of L1, the color level fidelity term is calculated by using a norm of L2, the structural level fidelity term is calculated by using SSIM (image quality) loss, and the loss function adopted during the training of the brightness adjustment network is represented as:
Figure BDA0003504364810000121
wherein L isadjustIndicating the loss of the brightness adjustment network,
Figure BDA0003504364810000122
represents the gradient in the horizontal or vertical direction of the optimized illumination layer of the training sample,
Figure BDA0003504364810000123
representing the gradient of the target illumination layer of the training sample in the horizontal or vertical direction, IrefRepresenting a reference image, R represents an optimized reflective layer of a training sample,
Figure BDA0003504364810000124
representing the target illumination layer of the training sample, SSIM representing the image quality loss function.
After the target illumination layer is obtained, the image reconstruction module multiplies the target illumination layer and the optimized reflection layer to reconstruct an image to obtain a target illumination image.
Fig. 1C is a schematic diagram of an operating principle of the low-illumination image enhancement model according to the embodiment of the present invention. In FIG. 1C, an input image I passes through an initialization module
Figure BDA0003504364810000125
Obtaining an initialized illumination layer L0 and an initialized reflection layer R0 after the initialized decomposition, outputting an optimized illumination layer and an optimized reflection layer after T times of alternate iterative optimization training of an optimization module, inputting the optimized illumination layer into an illumination adjusting module, splicing an adjusting scale factor omega with the optimized illumination layer, and inputting a splicing characteristic diagram into a brightness adjusting network
Figure BDA0003504364810000126
Adjusting brightness to obtain a target illumination layer
Figure BDA0003504364810000127
Then, the layer is illuminated based on the target
Figure BDA0003504364810000128
And optimizing the reflecting layer to reconstruct the image, and inputting the enhanced image, namely the target illumination image.
In the embodiment of the invention, the low-illumination image enhancement model comprises an initialization module, an optimization module, an illumination adjustment module and an image reconstruction module which are sequentially connected, wherein the initialization module is used for carrying out initialization decomposition on an input image to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image, the optimization module is used for carrying out a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm to obtain an optimization illumination layer and an optimization reflection layer, the illumination adjustment module is used for carrying out illumination adjustment on the optimization illumination layer to obtain a target illumination layer, the image reconstruction module is used for carrying out image reconstruction according to the target illumination layer and the optimization reflection layer to obtain a target illumination image, so that the flexibility and the interpretability of the low-illumination image enhancement model are ensured, and the robustness of the low-illumination image enhancement model is improved, and the low-illumination image enhancement model can restrain noise and simultaneously retain detail information.
In the embodiment of the present invention, each unit/module of the low-light image enhancement module may be implemented by corresponding hardware or software units, and each unit/module may be an independent software or hardware unit/module, or may be integrated into one software or hardware unit/module, which is not limited herein.
Example two:
this example further illustrates the low-light enhancement model described in the first experimental example with reference to the experimental example:
the experimental example subjectively and objectively evaluates the unfolding-based low-light image enhancement model described in example one on two disclosed low-light image enhancement test sets. The two representative datasets are the LOL dataset and the SICE dataset, respectively. The experimental examples used general reference indices for evaluating Image quality, namely Mean Absolute Error (MAE), Structural Similarity (SSIM), Peak Signal to Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS). A good model would have high PSNR and SSIM index scores, but low MAE and LPIPS scores. This example compares the low-light image enhancement model proposed in the first example with some existing reference models, such as LIME, NPE, SRIE, RRM, LR3M, Retinex-Net, KinD, Zero-DCE, RUAS.
The results of the comparison of the model performances are given in table one and table two, and it is clear that the low-light image enhancement model (uretiex-Net) proposed in example one achieves very good performance on the LOL and SICE datasets compared to other reference models.
Table one embodiment one describes the experimental evaluation of low-light image enhancement model and reference model on the LOL database test set
Figure BDA0003504364810000131
Figure BDA0003504364810000141
Table two examples experiments of the low-light image enhancement model and the reference model described in the first embodiment on the SICE database test set
Figure BDA0003504364810000142
Table one shows the experimental results of different low-light image enhancement models on the test set of the LOL database. It can be seen that the low-illumination image enhancement model (uretinix-Net) proposed in the first embodiment achieves good performance. Compared with the traditional Retinex optimization model based on manual prior, the model of the experimental example shows excellent effects on all indexes, and the optimization module described in the first embodiment can learn a more robust prior rule from data. Compared with other deep learning-based methods, the experimental example only slightly differs from KinD (0.0832vs 0.0804) in the index of MAE, and the difference is very small. However, in other indexes (PSNR, SSIM and LPIPS), the low-illumination image enhancement model (uretiniex-Net) proposed in the embodiment is significantly superior to other models, and further illustrates the advantages of the enhancement mode based on the iterative alternation optimization proposed in the present application.
In addition, in order to verify the generalization ability of the low-light image enhancement model proposed in the first embodiment, the performance of the model trained on the LOL data set was evaluated on the SICE data set, and the comparison results are given in table one. As is apparent from table one, the MAE, PSNR, and SSIM index scores of the low-light image enhancement model (uretinix-Net) proposed in the first embodiment are significantly better than those of other reference models, and the low-light image enhancement model proposed in the first embodiment exhibits better noise suppression capability and image structure information retention capability for the same data set. This is enough to show that the low-light image enhancement model proposed in the first embodiment has a relatively strong generalization capability, and can achieve a relatively good result even in scenes that do not appear in the training set.
The experimental example optionally visualizes some test results of the low-light image enhancement models on the LOL and SICE data sets in fig. 2A and 2B, including the low-light image enhancement model (Ours) proposed in the first embodiment, and the reference map (Ground-route) is displayed in the last column of the last row. Fig. 2A and 2B compare the low-light image enhancement model provided in this embodiment with LIME, NPE, SRIE, RRM, LR3M, Retinex-Net, KinD, Zero-DCE, and RUAS models. As shown in fig. 2B, the low-light enhancement model (Ours) proposed in the first embodiment performs well in some challenging cases. For example, the regions highlighted in FIG. 2B, it can be seen from FIG. 2B that the brightness is very low in the source image (Input), which necessarily introduces a lot of noise (e.g., LIME, NPE, SRIE, Retinex-Net and Zero-DCE) if only the contrast is increased, which severely disturbs important texture details. While algorithms that take noise into account (such as RRM, LR3M, KinD and RUAS), while the noise is significantly reduced, excessive smoothing results in loss of important details. In contrast, the low-illumination image enhancement model provided by the application can both sufficiently remove noise and retain important texture details. It can be seen from fig. 2A that the low-light image enhancement model (Ours) proposed in the first embodiment performs well in terms of color fidelity, noise suppression or exposure.
Example three:
third embodiment of the present invention is implemented based on the low-illumination image enhancement model described in the first embodiment, and fig. 3 shows an implementation flow of the low-illumination image enhancement method provided in the third embodiment of the present invention, and for convenience of description, only the parts related to the third embodiment of the present invention are shown, and details are as follows:
in step S301, the initialization module performs initialization decomposition on the input image to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image.
In an embodiment of the present invention, the input image is a low-light image to be subjected to image enhancement. Considering that variable initialization plays an important role for iterative optimization algorithms (e.g., ADMM), the input image can be initially decomposed using the commonly used all-zero initialization decomposition and random initialization. In order to provide the correct direction for the subsequent optimization by the initialized decomposition, therefore, the initialized illumination layer and the initialized reflection layer should contain important information of the input image, assuming that the reflection layer is an RGB three-channel image and three channels share the same illumination layer, the three-channel maximum value of the input image can be directly assigned to the illumination layer as the initialized illumination layer, and according to Retinex theory, the color of the object is composed of the reflectivity of the surface and the illumination intensity falling on the surface, so that the following imaging expression is provided:
I＝R·L (1)
where I denotes an input image, R denotes a reflective layer of the input image, and L denotes an illumination layer of the input image, and thus, the initialized reflective layer can be obtained based on the Retinex theory described above.
However, the above-mentioned initialization decomposition method will enlarge the difference of three channel values in the reflective layer, and further destroy the statistical properties of R, G, and B. Thus, the initialization module preferably implements the initialization decomposition of the illumination layer and the reflective layer using a fully connected network such that the initialization illumination layer and the initialization reflective layer contain more information while avoiding corrupting R, G, B the statistical properties. Further preferably, the fully-connected neural network is a full convolution neural network including 4 convolution layers, so as to improve the operation efficiency of the low-illumination image enhancement model.
Specifically, the input of the initialization module is a low-illumination image, the output is an initialization illumination layer and an initialization reflection layer corresponding to the input image, and the abstract representation is as follows:
Figure BDA0003504364810000161
wherein R is0Denotes an initialization reflective layer, L0It is indicated that the illumination layer is initialized,
Figure BDA0003504364810000162
indicating an initialization module, I indicating an input image, thetaDIndicating the parameters of the initialization module.
For the purpose of preserving effective information of the input image by the initialization decomposition, preferably, the loss function adopted by the initialization module in training includes a fidelity term and a prior term, so that the initialization illumination layer and the initialization reflection layer after the initialization decomposition preserve effective information of the input image. The fidelity term is used for measuring the closeness degree of an initialization image formed by an initialization illumination layer and an initialization reflection layer of a training sample and the training sample so as to ensure that the initialization decomposition meets the Retinex theory, and the priori term is used for measuring the closeness degree of the initialization illumination layer of the training sample and the R, G, B three-channel maximum value of the training sample so as to enable the initialization illumination layer to learn richer structural information. The training sample is an input image during low-illumination image enhancement model training.
Preferably, fidelity terms in the loss function adopted during training of the initialization module are calculated by using a norm L1, and prior terms are calculated by using a norm L2, so that the training effect of the initialization module is ensured, and effective information of the input image is kept as much as possible by an illumination layer and a reflection layer after initialization decomposition. The loss function formula adopted during the training of the initialization module is as follows:
Figure BDA0003504364810000171
wherein L isinitDenotes the loss of the initialization module, I denotes the training samples (input images during model training), R0Initialized reflective layer, L, representing training samples0Represents the initial illumination layer of the training sample, μ is a constant, and R, G, B represents the red, green, and blue channels, respectively.
In step S302, the initializing illumination layer and the initializing reflection layer are subjected to a plurality of times of alternating iterative optimization by an optimization module using an undermining algorithm to obtain an optimized illumination layer and an optimized reflection layer.
In the embodiment of the present invention, according to the Retinex theory, the decomposition problem of the imaging expression (1) is a pathological problem, and therefore prior model constraints R and L need to be established when solving the problem, which are expressed as:
Figure BDA0003504364810000172
where Φ (R) and Ψ (L) represent a priori assumptions for the reflective layer R and the illumination layer L, respectively. To solve the problem, two new variables are introduced, namely a first intermediate variable P and a second intermediate variable Q, and equation (4) can then be rewritten as:
Figure BDA0003504364810000173
so far, formula (3) can be reasonably divided into fidelity terms and prior terms for independent solution, and the following expression is obtained:
Figure BDA0003504364810000174
Figure BDA0003504364810000175
Figure BDA0003504364810000176
Figure BDA0003504364810000177
wherein, PkAnd QkRepresenting the result of the solution of the fidelity term, R, at the kth iterationkAnd LkThe solution of the prior term at the kth iteration is shown. When k is 1, PkAnd QkIndicating an initialization reflective layer and an initialization reflective layer illumination layer.
It is considered that the initialized reflective layer tends to be saturated with noise, severely affecting the presentation of important details. The initialization illumination layer not only completely retains structural information, but also leaves redundant texture details to a great extent. Therefore, the purpose of the optimization module in this embodiment is to repair the initialized reflective layer, so that effective details can be completely retained while removing noise. The satisfactory illumination layer should be structurally complete and smooth in texture detail, and in order to achieve the above purpose, the fidelity terms and the prior terms in the above equations (6) - (9) can be alternately and iteratively optimized by using an unfolding algorithm based on the framework of the deep neural network. Based on the comprehensive consideration of time and performance, the number of times of alternate iterative optimization performed by the optimization module is preferably 3, so as to determine the number of times of alternate iterative optimization according to the actual experimental effect.
Preferably, the optimization module comprises a variable calculation sub-network, a reflection layer repair network and an illumination layer repair network, when current alternate iterative optimization is performed, the variable calculation sub-network is used for calculating a first intermediate variable and a second intermediate variable after current iterative optimization, the reflection layer repair network is used for obtaining an optimized reflection layer after current iterative optimization based on the first intermediate variable and the second intermediate variable after current iterative optimization, and the illumination layer repair network is used for obtaining an optimized illumination layer after current iterative optimization based on the second intermediate variable after current iterative optimization to replace the traditional manual prior solution based on the Retinex optimization algorithm, so that the optimization module learns more robust prior information from data. The reflection layer repairing network and the illumination layer repairing network are constructed on the basis of a neural network. Preferably, the variable quantity operator network is used for calculating the first intermediate variable and the second intermediate variable after the current iteration optimization by adopting a least square method so as to realize the calculation of the intermediate variables. In specific implementation, when the kth iterative optimization is performed, Q obtained according to the previous (k-1) iterative optimization can be firstly obtainedk-1、Rk-1Solving the above equation (6) to obtain a first intermediate variable P of the current iteration optimizationkThen fixing the first intermediate variable P optimized in the current iterationkAnd the previous oneOptimized illumination layer L for sub-iterative optimizationk-1Solving the above formula (7) to obtain a second intermediate variable Q of the current iteration optimizationkThe above equations (6) - (7) are understood as a classical least squares problem, so that the following closed-loop solution can be obtained by derivation:
Figure BDA0003504364810000181
Figure BDA0003504364810000191
wherein k is more than or equal to 1, k is a positive integer, and lambda is a constant.
In the formation of PkAnd QkThereafter, the prior term R is learnedkAnd Lk。
According to experiments, the noise on the reflecting layer and the brightness distribution on the illuminating layer are highly correlated, and the noise is less when the brightness is higher, and vice versa. Preferably, the reflection layer repairing network is configured to perform convolution operation on the calculated feature map of the first intermediate variable and the second intermediate variable to obtain a first intermediate feature map and a second intermediate feature map, perform fusion processing on the first intermediate feature map and the second intermediate feature map to obtain a fusion feature map, and then obtain an optimized reflection layer after the current iteration based on the noise distribution of the fusion feature map and the first intermediate feature map, so that the reflection layer is repaired by combining information of the illumination layer, denoising of the reflection layer is achieved, and a learning effect of the optimized reflection layer is improved.
Further preferably, the reflection layer repair network is configured to perform convolution operation on the first intermediate variable and the second intermediate variable after the current iteration optimization to obtain a first intermediate feature map and a second intermediate feature map, perform cascade operation on the first feature map and the second intermediate feature map to obtain a spliced feature map, perform channel attention calculation on the spliced feature map by using a channel attention mechanism to obtain a re-weighted feature map, obtain noise distribution of the re-weighted feature map, and obtain the optimized reflection layer after the current iteration optimization based on the noise distribution and the first intermediate feature map.
Optimizing the reflective layer R at the time of the kth alternate iterative optimization in equation (8)kThe abstract representation is:
Figure BDA0003504364810000192
wherein, thetaRRepresenting a reflective layer repair network
Figure BDA0003504364810000193
Parameter of (A), PkFirst intermediate variable, Q, representing the kth alternate iterative optimizationkA second intermediate variable representing the kth alternating iterative optimization.
Optimizing the illumination layer L during the kth alternate iterative optimization in equation (9)kThe abstract representation is: :
Figure BDA0003504364810000194
wherein, thetaLRepresenting lighting layer repair networks
Figure BDA0003504364810000195
The parameter (c) of (c).
In step S303, the illumination adjustment module performs illumination adjustment on the optimized illumination layer to obtain a target illumination layer.
In the embodiments of the present invention, in the Retinex theory, the reflective layer is an inherent property of an object and does not change with the illumination condition. Based on the theory, the low-illumination imaging is caused by the lower intensity of the illumination layer, so that after the optimized illumination layer and the optimized reflection layer are obtained, the optimized reflection layer can be fixed and not changed, and the optimized illumination layer is adjusted.
The illumination adjustment module may adjust illumination of the illumination layer by using a gamma correction (gamma correction) technique, which can achieve different brightness enhancement effects by adjusting parameters, but it is difficult to determine an adjustment scale factor by adjusting the scale. Preferably, the illumination adjustment module is configured to perform illumination adjustment on the optimized illumination layer according to a preset adjustment scale factor, where the adjustment scale factor is specified by a user, so as to generate the target illumination layer according to a brightness adjustment scale specified by the user. In specific implementation, the input of the illumination adjustment module is an optimized illumination layer and a user-specified adjustment scale factor, and the output is a high-light illumination layer under a target adjustment scale, namely a target illumination layer.
Preferably, the illumination adjustment module includes an adjustment factor expansion submodule, a splicing submodule and a brightness adjustment network which are connected in sequence, wherein the adjustment factor expansion submodule is used for expanding a preset adjustment scale factor into a matrix with the same size as the optimized illumination layer, the splicing submodule is used for splicing the matrix and the optimized illumination layer to obtain a splicing result, and the brightness adjustment network is used for adjusting the brightness of the optimized illumination layer based on the splicing result to obtain a target illumination layer so as to adjust the illumination intensity. In a specific implementation, the scaling factor is expanded into a matrix with the same size as the illumination layer, and then the matrix is spliced with the optimized illumination layer to serve as an input of the electrical network after the brightness is adjusted, which can be specifically expressed as:
Figure BDA0003504364810000201
where ω denotes the scale factor of the adjustment, θARepresenting a brightness adjustment network
Figure BDA0003504364810000202
L denotes the optimized illumination layer.
Preferably, the brightness adjustment network has the same network structure as the initialization module, that is, the brightness adjustment network may be a full convolution neural network including 4 convolution layers, and the convolution kernel of the convolution layer of the brightness adjustment network has a larger size than that of the convolution layer of the initialization module, so as to maintain uniformity and limit the smoothness of the illumination layer.
In order to train the brightness adjustment network better, considering that the high-brightness illumination layer output by the brightness adjustment network should be consistent with the input low-brightness illumination layer in structure, preferably, the loss function adopted in the training of the brightness adjustment network includes a gradient level fidelity term to ensure the training effect of the brightness adjustment network. And the gradient layer fidelity item is used for measuring the horizontal or vertical gradient distance between the optimized illumination layer and the target illumination layer of the training sample.
In order to reconstruct an image of normal illumination based on the target illumination layer, it is preferable that the loss function used in the training of the brightness adjustment network includes a color-level fidelity term to further improve the training effect of the brightness adjustment network. The color level fidelity item is used for measuring the reconstruction loss of a target illumination image and a reference image of a training sample, so that the reconstructed image is a normal illumination image, namely the reference image.
In order to make the reconstructed image consistent with the reference image in structure, brightness and contrast, the loss function adopted in the training of the brightness adjustment network preferably includes one or more combinations of color-level fidelity terms and structure-level fidelity terms, so as to further improve the training effect of the brightness adjustment network. The structure-level fidelity item is used for measuring the distance between a target illumination image of a training sample and a reference image, so that the reconstructed image is consistent with the reference image in structure, brightness and contrast.
Preferably, the loss function adopted in the lighting adjustment module training includes a gradient level fidelity term, a color level fidelity term and a structure level fidelity term, so as to further improve the training effect of the brightness adjustment network through the constraints of three aspects.
Further preferably, the gradient level fidelity term is calculated by using a norm of L1, the color level fidelity term is calculated by using a norm of L2, the structural level fidelity term is calculated by using SSIM (image quality) loss, and the loss function adopted during the training of the brightness adjustment network is represented as:
Figure BDA0003504364810000211
wherein L isadjustIndicating the loss of the brightness adjustment network,
Figure BDA0003504364810000212
represents the gradient in the horizontal or vertical direction of the optimized illumination layer of the training sample,
Figure BDA0003504364810000213
representing the gradient of the target illumination layer of the training sample in the horizontal or vertical direction, IrefRepresenting a reference image, R represents an optimized reflective layer of a training sample,
Figure BDA0003504364810000214
representing the target illumination layer of the training sample, SSIM representing the image quality loss function.
In step S304, an image reconstruction module reconstructs an image according to the target illumination layer and the optimized reflection layer to obtain a target illumination image.
In the embodiment of the invention, the image reconstruction module multiplies the target illumination layer and the optimized reflection layer to reconstruct the image to obtain the target illumination image.
In the embodiment of the invention, an input image is initialized and decomposed to obtain an initialized illumination layer and an initialized reflection layer corresponding to the input image, the initialized illumination layer and the initialized reflection layer are subjected to a plurality of times of alternate iterative optimization by adopting an unfolding algorithm to obtain an optimized illumination layer and an optimized reflection layer, the optimized illumination layer is subjected to illumination adjustment to obtain a target illumination layer, and image reconstruction is carried out according to the target illumination layer and the optimized reflection layer to obtain a target illumination image, so that the flexibility and the interpretability of the low-illumination image enhancement model are ensured, the robustness of the low-illumination image enhancement model is improved, and the low-illumination image enhancement model retains detail information while suppressing noise.
Example four:
fig. 4 shows a structure of an electronic device according to a fourth embodiment of the present invention, and only a part related to the fourth embodiment of the present invention is shown for convenience of description.
The electronic device 4 of an embodiment of the invention comprises a processor 40, a memory 41 and a computer program 42 stored in the memory 41 and executable on the processor 40. The processor 40, when executing the computer program 42, implements the steps in the above-described method embodiments, for example, the steps S301 to S304 shown in fig. 3. Alternatively, the processor 40, when executing the computer program 42, implements the functions of the modules in the low-light image enhancement model embodiment described above, e.g., the functions of the modules 11 to 14 shown in fig. 1A.
Example five:
in an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps in the above-described method embodiment, for example, steps S301 to S304 shown in fig. 3. Alternatively, the computer program, when executed by a processor, implements the functionality of the modules in the above-described low-light image enhancement model embodiment, e.g., the functionality of modules 11 to 14 shown in fig. 1A.
The computer readable storage medium of the embodiments of the present invention may include any entity or device capable of carrying computer program code, a recording medium, such as a ROM/RAM, a magnetic disk, an optical disk, a flash memory, or the like.
The above description is intended to be illustrative of the preferred embodiment of the present invention and should not be taken as limiting the invention, but rather, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Claims (10)

1. A low-illumination image enhancement model is characterized by comprising an initialization module, an optimization module, an illumination adjustment module and an image reconstruction module which are sequentially connected, wherein,
the initialization module is used for performing initialization decomposition on an input image to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image;
the optimization module is used for performing a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm to obtain an optimization illumination layer and an optimization reflection layer;
the illumination adjusting module is used for adjusting illumination of the optimized illumination layer to obtain a target illumination layer;
and the image reconstruction module is used for reconstructing an image according to the target illumination layer and the optimized reflection layer to obtain a target illumination image.
2. The low-light image enhancement model of claim 1, wherein the initialization module is a fully-connected neural network, the fully-connected neural network being a fully-convolutional neural network comprising 4 convolutional layers.
3. The low-light image enhancement model of claim 1, wherein the loss function used by the initialization module in training comprises a fidelity term and an a priori term, wherein the fidelity term is used for measuring the proximity of an initialization image consisting of an initialization illumination layer and an initialization reflection layer of a training sample to the training sample, and the a priori term is used for measuring the proximity of the initialization illumination layer of the training sample to the R, G, B three-channel maximum of the training sample;
the loss function adopted during the training of the initialization module is as follows:
Figure FDA0003504364800000011
wherein L isinitRepresenting the loss of the initialization module, I representing the training sample, R0An initialisation reflecting layer, L, representing the training sample0Represents the initialization illumination layer of the training sample, μ is a constant, and R, G, B represents the red, green, and blue channels, respectively.
4. The low-light image enhancement model of claim 1, wherein the optimization module comprises a variable computation sub-network, a reflection layer repair network and an illumination layer repair network, when performing current alternate iterative optimization, the variable computation sub-network is configured to compute a first intermediate variable and a second intermediate variable after current iterative optimization, the reflection layer repair network is configured to obtain an optimized reflection layer after current iterative optimization based on the first intermediate variable and the second intermediate variable after current iterative optimization, and the illumination layer repair network is configured to obtain an optimized illumination layer after current iterative optimization based on the second intermediate variable after current iterative optimization.
5. The low-light image enhancement model of claim 4, wherein the variable computation sub-network is configured to compute the first intermediate variable and the second intermediate variable after the current iterative optimization using a least squares method;
the reflection layer repairing network is used for performing convolution operation on the first intermediate variable and the second intermediate variable after the current iteration optimization to obtain a first intermediate characteristic diagram and a second intermediate characteristic diagram, performing cascade operation on the first characteristic diagram and the second intermediate characteristic diagram to obtain a spliced characteristic diagram, performing channel attention calculation on the spliced characteristic diagram by adopting a channel attention mechanism to obtain a re-weighted characteristic diagram, obtaining noise distribution of the re-weighted characteristic diagram, and obtaining an optimized reflection layer after the current iteration optimization based on the noise distribution and the first intermediate characteristic diagram.
6. The low-illumination image enhancement model of claim 1, wherein the illumination adjustment module comprises an adjustment factor expansion sub-module, a stitching sub-module, and a brightness adjustment network connected in sequence, wherein,
the adjustment factor expansion submodule is used for expanding a preset adjustment scale factor into a matrix with the same size as the optimized illumination layer;
the splicing submodule is used for splicing the matrix and the optimized illumination layer to obtain a splicing result;
and the brightness adjusting network is used for adjusting the brightness of the optimized illumination layer based on the splicing result to obtain the target illumination layer, wherein the brightness adjusting network has the same network structure as the initialization module, and the size of the convolution kernel of the convolution layer of the brightness adjusting network is larger than that of the convolution kernel of the convolution layer of the initialization module.
7. The low-light image enhancement model of claim 6, wherein the loss function used in the training of the luma adjustment network comprises one or more combinations of a gradient-level fidelity term, a color-level fidelity term, and a structure-level fidelity term, wherein the gradient-level fidelity term is used to measure a horizontal or vertical gradient distance between an optimized illumination layer and a target illumination layer of a training sample, the color-level fidelity term is used to measure a reconstruction loss of a target illumination image and a reference image of the training sample, and the structure-level fidelity term is used to measure a distance between the target illumination image and the reference image of the training sample;
the loss function adopted during the brightness adjustment network training is as follows:
Figure FDA0003504364800000031
wherein L isadjustRepresents a loss of the brightness adjustment network,
Figure FDA0003504364800000032
representing a gradient in a horizontal or vertical direction of an optimized illumination layer of the training sample,
Figure FDA0003504364800000033
representing the gradient of the target illumination layer of the training sample in the horizontal or vertical direction, IrefRepresenting the reference image, R representing an optimized reflective layer of the training sample,
Figure FDA0003504364800000034
representing the trainingThe target illumination layer of the training sample, SSIM, represents the image quality loss function.
8. A low-light image enhancement method based on the low-light image enhancement model of any one of claims 1 to 7, characterized in that the method comprises the following steps:
performing initialization decomposition on an input image through the initialization module to obtain an initialization illumination layer and an initialization reflection layer corresponding to the input image;
performing a plurality of times of alternate iterative optimization on the initialization illumination layer and the initialization reflection layer by adopting an unfolding algorithm through the optimization module to obtain an optimization illumination layer and an optimization reflection layer;
performing illumination adjustment on the optimized illumination layer through the illumination adjustment module to obtain a target illumination layer;
and reconstructing an image according to the target illumination layer and the optimized reflection layer through the image reconstruction module to obtain a target illumination image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the functionality of the low-light image enhancement model of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the functions of the low-light image enhancement model according to any one of claims 1 to 7.
CN202210135560.2A 2022-02-14 2022-02-14 Low-illumination image enhancement model, method, electronic device and storage medium Pending CN114638749A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210135560.2A CN114638749A (en) 2022-02-14 2022-02-14 Low-illumination image enhancement model, method, electronic device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210135560.2A CN114638749A (en) 2022-02-14 2022-02-14 Low-illumination image enhancement model, method, electronic device and storage medium

Publications (1)

Publication Number Publication Date
CN114638749A true CN114638749A (en) 2022-06-17

Family

ID=81945688

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210135560.2A Pending CN114638749A (en) 2022-02-14 2022-02-14 Low-illumination image enhancement model, method, electronic device and storage medium

Country Status (1)

Country Link
CN (1) CN114638749A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116310276A (en) * 2023-05-24 2023-06-23 泉州装备制造研究所 Target detection method, target detection device, electronic equipment and storage medium
CN116797490A (en) * 2023-07-12 2023-09-22 青岛理工大学 Lightweight turbid water body image enhancement method

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116310276A (en) * 2023-05-24 2023-06-23 泉州装备制造研究所 Target detection method, target detection device, electronic equipment and storage medium
CN116310276B (en) * 2023-05-24 2023-08-08 泉州装备制造研究所 Target detection method, target detection device, electronic equipment and storage medium
CN116797490A (en) * 2023-07-12 2023-09-22 青岛理工大学 Lightweight turbid water body image enhancement method
CN116797490B (en) * 2023-07-12 2024-02-09 青岛理工大学 Lightweight turbid water body image enhancement method

Similar Documents

Publication Publication Date Title
Li et al. Underwater scene prior inspired deep underwater image and video enhancement
US11055828B2 (en) Video inpainting with deep internal learning
Li et al. Low-light image and video enhancement using deep learning: A survey
Zhang et al. Image restoration: From sparse and low-rank priors to deep priors [lecture notes]
Khashabi et al. Joint demosaicing and denoising via learned nonparametric random fields
Mairal et al. Sparse representation for color image restoration
CN114638749A (en) Low-illumination image enhancement model, method, electronic device and storage medium
JP2003018398A (en) Method for generating a super-resolution image from pixel image
CN113822830B (en) Multi-exposure image fusion method based on depth perception enhancement
CN113096029A (en) High dynamic range image generation method based on multi-branch codec neural network
Lyu et al. An efficient learning-based method for underwater image enhancement
CN111178499A (en) Medical image super-resolution method based on generation countermeasure network improvement
CN115950837A (en) Snapshot type spectral imaging method, system and medium based on plug-and-play prior
CN115829876A (en) Real degraded image blind restoration method based on cross attention mechanism
CN116468605A (en) Video super-resolution reconstruction method based on time-space layered mask attention fusion
Cui et al. TPET: two-stage perceptual enhancement transformer network for low-light image enhancement
Mai et al. Deep unrolled low-rank tensor completion for high dynamic range imaging
CN109741258A (en) Image super-resolution method based on reconstruction
Luo et al. A fast denoising fusion network using internal and external priors
Siddiqui et al. Hierarchical color correction for camera cell phone images
Ding et al. Learning-based underwater image enhancement: An efficient two-stream approach
Tian et al. A modeling method for face image deblurring
Wang et al. Image editing with varying intensities of processing
Zhao et al. A fast alternating minimization algorithm for coded aperture snapshot spectral imaging based on sparsity and deep image priors
Barua et al. ArtHDR-Net: Perceptually Realistic and Accurate HDR Content Creation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination