CN112927162A - Low-illumination image oriented enhancement method and system - Google Patents
Low-illumination image oriented enhancement method and system Download PDFInfo
- Publication number
- CN112927162A CN112927162A CN202110287338.XA CN202110287338A CN112927162A CN 112927162 A CN112927162 A CN 112927162A CN 202110287338 A CN202110287338 A CN 202110287338A CN 112927162 A CN112927162 A CN 112927162A
- Authority
- CN
- China
- Prior art keywords
- network
- image
- feature
- enhancement
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000005286 illumination Methods 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 32
- 230000009467 reduction Effects 0.000 claims abstract description 46
- 230000000007 visual effect Effects 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 31
- 238000000605 extraction Methods 0.000 claims abstract description 26
- 230000007246 mechanism Effects 0.000 claims abstract description 12
- 238000011176 pooling Methods 0.000 claims description 22
- 230000004927 fusion Effects 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000002708 enhancing effect Effects 0.000 claims description 8
- 239000003623 enhancer Substances 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a low-illumination image-oriented enhancement method, which relates to the technical field of image enhancement and comprises the following steps: inputting the acquired low-illumination image into a visual attention network to perform brightness information extraction processing, and further generating a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high luminance region and a low luminance region; inputting the channel attention image into a noise reduction network to perform noise reduction processing, and further generating a noise reduction image; inputting the low-illumination image, the channel attention image and the de-noised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function. The invention can effectively realize noise reduction and enhancement at the same time, improve the definition of the pictures shot in a low-light environment and comprehensively improve the image quality.
Description
Technical Field
The invention relates to the technical field of image enhancement, in particular to a low-illumination image-oriented enhancement method and system.
Background
Images have long been used in various aspects of human daily life as a convenient and straightforward means of information acquisition and transmission. Under the environment of insufficient illumination, the captured image has the characteristics of low signal-to-noise ratio, contrast ratio and resolution ratio. To improve this, shooting is performed by means of a low-illumination camera in terms of hardware, and the low-illumination problem is improved directly from the acquisition side, but the hardware equipment is expensive, so that the method cannot be popularized. Various low-illumination enhancement algorithms are also developed at present, and certain effects are achieved. The algorithm is a single-stage enhancement strategy, i.e. generating an enhanced light image directly from a low-illumination image, but the enhancement range of such methods is loose.
Disclosure of Invention
The invention aims to provide a low-illumination image-oriented enhancement method and system, which can effectively realize noise reduction and enhancement at the same time, make up for the defects of imaging equipment on hardware, improve the definition of pictures shot in a low-light environment and comprehensively improve the image quality.
In order to achieve the purpose, the invention provides the following scheme:
a low-illumination image-oriented enhancement method, comprising:
acquiring a low-illumination image;
inputting the low-illumination image into a visual attention network to perform brightness information extraction processing so as to generate a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high brightness region and a low brightness region;
inputting the channel attention image into a noise reduction network to perform noise reduction processing so as to generate a noise reduction image;
inputting the low-light image, the channel attention image and the de-noised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function.
Optionally, the inputting the low-illuminance image into a visual attention network to perform luminance information extraction processing, so as to generate a channel attention image specifically includes:
converting the low-illumination image into an HSI feature space to obtain a multi-channel feature map;
inputting the multi-channel feature map into an average pooling layer to obtain an average pixel value of each channel;
inputting the multichannel feature map into a maximum pooling layer to obtain a maximum pixel value of each channel;
inputting the average pixel value and the maximum pixel value of each channel into a multilayer perceptron to generate a high-brightness region feature vector and a low-brightness region feature vector;
respectively carrying out summation operation on the high-brightness region feature vector and the low-brightness region feature vector elements to obtain a first feature vector and a second feature vector;
and processing the first characteristic vector and the second characteristic vector by adopting a sigmoid function to obtain a channel attention image.
Optionally, the inputting the channel attention image into a noise reduction network for performing noise reduction processing to generate a noise-reduced image specifically includes:
inputting the channel attention image into a noise reduction network for noise reduction processing by adopting a jump link mode, and further generating a noise reduction image;
the noise reduction network is a 5-layer network, the first 4-layer network consists of Conv + BN + Relu, the number of filters used by the first 4-layer network is 128, the 5-layer network consists of Conv + BN, the number of filters used by the 5-layer network is 1, the filters are convolution kernel filters, and the size of each filter is 7 x 7.
Optionally, the inputting the low-illuminance image, the channel attention image, and the denoised image into an enhancement network to generate an enhanced image specifically includes:
inputting the low-illumination image, the channel attention image and the denoising image into a feature extraction sub-network to extract a bright information feature, a structural information feature, a perceptual information feature and a regional information feature; the enhancement network comprises a feature extraction sub-network, an enhancement sub-network and a fusion sub-network which are connected in sequence;
inputting the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhancement feature information;
inputting all the enhanced feature information into the fused sub-network to obtain an enhanced image.
Optionally, the inputting the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhancement feature information specifically includes:
inputting the bright information feature into a first enhancement sub-network to obtain first enhancement feature information; the enhancer network comprises a first enhancement subnetwork, a second enhancement subnetwork, a third enhancement subnetwork, and a fourth enhancement subnetwork; the loss function of the first enhancement sub-network is a luminance loss sub-function;
inputting the structural information feature into the second enhancement sub-network to obtain second enhancement feature information; the loss function of the second enhancement sub-network is a structural loss sub-function;
inputting the perceptual information feature into the third enhancement sub-network to obtain third enhancement feature information; the penalty function of the third enhancement sub-network is a perceptual penalty sub-function;
inputting the regional information feature into the fourth enhancement sub-network to obtain fourth enhancement feature information; the loss function of the fourth enhancement sub-network is a regional loss sub-function.
A low-light image-oriented enhancement system, comprising:
the image acquisition module is used for acquiring a low-illumination image;
the channel attention image generation module is used for inputting the low-illumination image into a visual attention network to extract brightness information so as to generate a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high brightness region and a low brightness region;
the de-noising image generation module is used for inputting the channel attention image into a de-noising network to perform de-noising processing so as to generate a de-noising image;
an enhanced image generation module, configured to input the low-illumination image, the channel attention image, and the denoised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function.
Optionally, the channel attention image generating module specifically includes:
the conversion unit is used for converting the low-illumination image into an HSI (hue, saturation and intensity) feature space so as to acquire a multi-channel feature map;
the average pixel value calculation unit is used for inputting the multi-channel feature map into an average pooling layer so as to obtain an average pixel value of each channel;
the maximum pixel value calculation unit is used for inputting the multi-channel feature map into a maximum pooling layer so as to obtain the maximum pixel value of each channel;
a feature vector determination unit for inputting the average pixel value and the maximum pixel value of each channel into a multilayer perceptron to generate a high-luminance region feature vector and a low-luminance region feature vector;
the fusion unit is used for respectively carrying out summation operation on the high-brightness region feature vector and the low-brightness region feature vector elements to obtain a first feature vector and a second feature vector;
and the channel attention image generation unit is used for processing the first feature vector and the second feature vector by adopting a sigmoid function to obtain a channel attention image.
Optionally, the denoised image generating module specifically includes:
the de-noising image generating unit is used for inputting the channel attention image into a de-noising network in a jump link mode to perform de-noising processing so as to generate a de-noising image;
the noise reduction network is a 5-layer network, the first 4-layer network consists of Conv + BN + Relu, the number of filters used by the first 4-layer network is 128, the 5-layer network consists of Conv + BN, the number of filters used by the 5-layer network is 1, the filters are convolution kernel filters, and the size of each filter is 7 x 7.
Optionally, the enhanced image generating module specifically includes:
a feature extraction unit, configured to input the low-illuminance image, the channel attention image, and the denoised image into a feature extraction sub-network to extract a bright information feature, a structural information feature, a perceptual information feature, and a regional information feature; the enhancement network comprises a feature extraction sub-network, an enhancement sub-network and a fusion sub-network which are connected in sequence;
an enhanced feature information determination unit configured to input the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhanced feature information;
and the enhanced image generating unit is used for inputting all the enhanced characteristic information into the fusion sub-network to obtain an enhanced image.
Optionally, the enhanced feature information determining unit specifically includes:
a first enhancement feature information determination subunit for inputting the bright information feature into a first enhancement sub-network to obtain first enhancement feature information; the enhancer network comprises a first enhancement subnetwork, a second enhancement subnetwork, a third enhancement subnetwork, and a fourth enhancement subnetwork; the loss function of the first enhancement sub-network is a luminance loss sub-function;
a second enhanced feature information determining subunit, configured to input the structural information feature into the second enhanced sub-network to obtain second enhanced feature information; the loss function of the second enhancement sub-network is a structural loss sub-function;
a third enhancement feature information determining subunit, configured to input the perceptual information feature into the third enhancement sub-network to obtain third enhancement feature information; the penalty function of the third enhancement sub-network is a perceptual penalty sub-function;
a fourth enhanced feature information determining subunit, configured to input the regional information feature into the fourth enhanced sub-network to obtain fourth enhanced feature information; the loss function of the fourth enhancement sub-network is a regional loss sub-function.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
in order to improve the definition of a low-illumination image and avoid color distortion, the invention provides a low-illumination image-oriented enhancement method and system, namely a dense low-illumination enhancement network method and system based on an attention mechanism. Firstly, a visual attention mechanism capable of extracting spatial information and local object features is introduced, then the network is utilized to learn feature mapping from a low-illumination image to a normal-illumination image, and finally low-illumination image enhancement is realized through a fusion network.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a flow chart of an enhancement method for low-illumination images according to the present invention;
FIG. 2 is a block diagram of an enhancement system for low-light images according to the present invention;
FIG. 3 is a schematic diagram of a network structure of an enhancement method for low-illumination images according to the present invention;
FIG. 4 is a flow chart of a visual attention network according to the present invention;
FIG. 5 is a schematic flow chart of a noise reduction network according to the present invention;
FIG. 6 is a schematic flow chart of a feature extraction subnetwork of the present invention;
FIG. 7 is a flow chart of a converged sub-network of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a low-illumination image-oriented enhancement method and system, which can effectively realize noise reduction and enhancement at the same time, make up for the defects of imaging equipment on hardware, improve the definition of pictures shot in a low-light environment and comprehensively improve the image quality.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Example one
As shown in fig. 1, the present embodiment provides a method for enhancing a low-illuminance image, which includes the following steps.
Step 101: a low illumination image is acquired.
Step 102: inputting the low-illumination image into a visual attention network to perform brightness information extraction processing so as to generate a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high luminance region and a low luminance region.
Step 103: and inputting the channel attention image into a noise reduction network to perform noise reduction processing, thereby generating a noise reduction image.
Step 104: inputting the low-light image, the channel attention image and the de-noised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function.
Wherein, step 102 specifically comprises:
converting the low-illumination image into an HSI feature space to obtain a multi-channel feature map; inputting the multi-channel feature map into an average pooling layer to obtain an average pixel value of each channel; inputting the multichannel feature map into a maximum pooling layer to obtain a maximum pixel value of each channel; inputting the average pixel value and the maximum pixel value of each channel into a multilayer perceptron to generate a high-brightness region feature vector and a low-brightness region feature vector; respectively carrying out summation operation on the high-brightness region feature vector and the low-brightness region feature vector elements to obtain a first feature vector and a second feature vector; and processing the first characteristic vector and the second characteristic vector by adopting a sigmoid function to obtain a channel attention image.
Step 103 specifically comprises:
inputting the channel attention image into a noise reduction network for noise reduction processing by adopting a jump link mode, and further generating a noise reduction image; the noise reduction network is a 5-layer network, the first 4-layer network consists of Conv + BN + Relu, the number of filters used by the first 4-layer network is 128, the 5-layer network consists of Conv + BN, the number of filters used by the 5-layer network is 1, the filters are convolution kernel filters, and the size of each filter is 7 x 7.
Step 104 specifically includes:
inputting the low-illumination image, the channel attention image and the denoising image into a feature extraction sub-network to extract a bright information feature, a structural information feature, a perceptual information feature and a regional information feature; the enhancement network comprises a feature extraction sub-network, an enhancement sub-network and a fusion sub-network which are connected in sequence; inputting the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhancement feature information; inputting all the enhanced feature information into the fused sub-network to obtain an enhanced image.
Wherein the inputting the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhancement feature information specifically includes:
inputting the bright information feature into a first enhancement sub-network to obtain first enhancement feature information; the enhancer network comprises a first enhancement subnetwork, a second enhancement subnetwork, a third enhancement subnetwork, and a fourth enhancement subnetwork; the loss function of the first enhancement sub-network is a luminance loss sub-function; inputting the structural information feature into the second enhancement sub-network to obtain second enhancement feature information; the loss function of the second enhancement sub-network is a structural loss sub-function; inputting the perceptual information feature into the third enhancement sub-network to obtain third enhancement feature information; the penalty function of the third enhancement sub-network is a perceptual penalty sub-function; inputting the regional information feature into the fourth enhancement sub-network to obtain fourth enhancement feature information; the loss function of the fourth enhancement sub-network is a regional loss sub-function.
Example two
As shown in fig. 2, the enhancement system for low-illumination images provided in this embodiment includes:
an image obtaining module 201, configured to obtain a low-illumination image.
A channel attention image generation module 202, configured to input the low-illuminance image into a visual attention network to perform luminance information extraction processing, so as to generate a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high luminance region and a low luminance region.
And the denoised image generating module 203 is configured to input the channel attention image into a denoising network to perform denoising processing, so as to generate a denoised image.
An enhanced image generation module 204, configured to input the low-illuminance image, the channel attention image, and the denoised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function.
The channel attention image generation module 202 specifically includes:
the conversion unit is used for converting the low-illumination image into an HSI (hue, saturation and intensity) feature space so as to acquire a multi-channel feature map; the average pixel value calculation unit is used for inputting the multi-channel feature map into an average pooling layer so as to obtain an average pixel value of each channel; the maximum pixel value calculation unit is used for inputting the multi-channel feature map into a maximum pooling layer so as to obtain the maximum pixel value of each channel; a feature vector determination unit for inputting the average pixel value and the maximum pixel value of each channel into a multilayer perceptron to generate a high-luminance region feature vector and a low-luminance region feature vector; the fusion unit is used for respectively carrying out summation operation on the high-brightness region feature vector and the low-brightness region feature vector elements to obtain a first feature vector and a second feature vector; and the channel attention image generation unit is used for processing the first feature vector and the second feature vector by adopting a sigmoid function to obtain a channel attention image.
The denoised image generating module 203 specifically includes:
the de-noising image generating unit is used for inputting the channel attention image into a de-noising network in a jump link mode to perform de-noising processing so as to generate a de-noising image; the noise reduction network is a 5-layer network, the first 4-layer network consists of Conv + BN + Relu, the number of filters used by the first 4-layer network is 128, the 5-layer network consists of Conv + BN, the number of filters used by the 5-layer network is 1, the filters are convolution kernel filters, and the size of each filter is 7 x 7.
The enhanced image generation module 204 specifically includes:
a feature extraction unit, configured to input the low-illuminance image, the channel attention image, and the denoised image into a feature extraction sub-network to extract a bright information feature, a structural information feature, a perceptual information feature, and a regional information feature; the enhancement network comprises a feature extraction sub-network, an enhancement sub-network and a fusion sub-network which are connected in sequence; an enhanced feature information determination unit configured to input the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhanced feature information; and the enhanced image generating unit is used for inputting all the enhanced characteristic information into the fusion sub-network to obtain an enhanced image.
The enhanced feature information determining unit specifically includes:
a first enhancement feature information determination subunit for inputting the bright information feature into a first enhancement sub-network to obtain first enhancement feature information; the enhancer network comprises a first enhancement subnetwork, a second enhancement subnetwork, a third enhancement subnetwork, and a fourth enhancement subnetwork; the loss function of the first enhancement sub-network is a luminance loss sub-function; a second enhanced feature information determining subunit, configured to input the structural information feature into the second enhanced sub-network to obtain second enhanced feature information; the loss function of the second enhancement sub-network is a structural loss sub-function; a third enhancement feature information determining subunit, configured to input the perceptual information feature into the third enhancement sub-network to obtain third enhancement feature information; the penalty function of the third enhancement sub-network is a perceptual penalty sub-function; a fourth enhanced feature information determining subunit, configured to input the regional information feature into the fourth enhanced sub-network to obtain fourth enhanced feature information; the loss function of the fourth enhancement sub-network is a regional loss sub-function.
EXAMPLE III
The enhancement method for the low-illumination image provided by the embodiment comprises three modules, namely a visual attention network, a noise reduction network and an enhancement network. The network structure is shown in fig. 3.
In order to improve the image quality and the network stability, and further consider the structural information, performance and regional difference of the image, the embodiment provides a new loss function.
L＝ωaLa+wnLn+weLe；
Wherein L isa,Ln,LeRespectively representing the loss functions of the visual Attention-net (Attention-net), the noise reduction-net (noise-net), and the Enhancement-net (Enhancement-net), omegaa,ωn,ωeRespectively, are the corresponding coefficients.
The first step is as follows: an input image (i.e., a low-illuminance image) is first input into the visual attention network to perform luminance information extraction processing, thereby generating a channel attention image.
The visual attention mechanism is a special mechanism formed in the evolution process of human eye vision, and can consciously eliminate irrelevant information aiming at a global image input by human eyes and put more attention into an interested target area. Images captured in an environment with insufficient lighting are generally poor in visibility, low in contrast, and contain a large amount of noise. In order to solve the uneven distribution of the brightness in the low-illumination image, the conventional image enhancement method generally enhances the entire image, but ignores the brightness inconsistency of each area in the low-illumination image, which easily causes overexposure of the high-brightness area and underexposure of the low-brightness area. The embodiment adopts a visual attention network (a visual attention mechanism is arranged in the network) which is mainly used for estimating the distribution of the weak light areas in the low-illumination image, and the higher weight value allocated to the corresponding area is used for enhancing the expression of the weak light areas, so that the enhancement network is promoted to give more attention to the weak light areas. The specific flow of the visual attention network is shown in fig. 4.
The visual attention network provided by the embodiment generates a given low-illumination imageForm a multi-channel characteristic diagram F ═ RC×H×W(where C denotes the number of channels, H denotes the height, and W denotes the width), the information expressed by the different channels is different, and the purpose of the visual attention network is to use the information of each channel in the feature image to learn a one-dimensional weight WC∈RC×1×1And then multiplied by the corresponding channels, respectively. The average pooling layer has the function of intensively expressing all information in the image, so that the bias of image pixels is not generated, and the loss of details is prevented. The role of the max-pooling layer is to extract information of the feature regions in the image. Thus, the area with low brightness in the current image can be more focused, and the information of the area with low illumination intensity in the image can be more effectively represented.
The present embodiment first inputs the low-illuminance images into the maximum pooling layer and the average pooling layer, respectively, to aggregate the spatial dimension information, as shown in fig. 4. After the low-illumination image is converted into the HSI space, an average pooling layer (AVP) operation is performed, and the calculation formula is as follows:
wherein i represents the current channel of the feature map, k represents the number of pixels of the current channel, and fikRepresenting a particular pixel value, H, of the current channel in the low-light imageiIndicating the low-illumination image length, W, of the current channeliRepresenting the width of the low-illumination image of the current channel, and C representing the total number of channels of the feature map; the average pixel value of all channels of a current low-illumination image is obtained by an average pooling layer (AVP) operation, the output of which is a 1 × 1 × C one-dimensional vector. Meanwhile, after the low-illumination image is converted into the HSI space, the maximum pooling layer operation is performed, and the maximum pooling layer (MAX) calculation formula is as follows:
MAX＝max(sort(abs(fik)))(i＝1…C)
wherein f isikRepresenting a particular pixel value representing the current channel in the low-illumination image, abs representing the value f for the pixel in the current low-illumination imageikTaking the absolute value, sort, which represents the absolute value of the pixel takenThe values are sorted and max denotes taking the maximum value for the pixel value of the channel. After the maximum pooling layer and the average pooling layer are calculated, two feature descriptors are generated for the channel. And then inputting the feature descriptor into a shared multilayer perceptron with the number of layers of C/4 to generate a representative feature vector, combining and outputting the feature vector after element summation operation, and finally obtaining a channel attention image through a sigmoid function, wherein the MLP represents the multilayer perceptron.
WC(F)＝sigmoid(MLP(AVP(F))+MLP(MAX(F)))
To correctly obtain the attention characteristics to guide the enhancement network, the present embodiment employs an L2 loss function to predict the error.
La＝||F(I)-F||2
Where I is the input image (i.e., low-illumination image), F (I) (W)C(F) And F are luminance images of the predicted channel attention image and the original image, respectively.
The second step is that: and inputting the channel attention image into a noise reduction network to perform noise reduction processing, thereby generating a noise reduction image.
The low-illumination image processed in the first step is easy to amplify the noise of the original image, the image noise is easy to be confused with the original texture of the image, and an unnecessary blurring effect can be caused after a simple noise reduction method is applied. Because the noise distribution is related to the exposure distribution, the channel attention image is adopted to guide the generation of the noise image, and the enhancement network can effectively perform denoising under the guide.
The noise reduction network adopts a DnCNN-like network, a BN algorithm is added after convolution, the BN algorithm firstly calculates the mean value and the variance of the preprocessed image, and the formula is as follows:
where k represents the current channel pixel number, xiRepresenting the convolved pixel values. And then carrying out normalization processing on the data, wherein the formula is as follows.
Where λ is a bias constant, the prevention denominator is 0,
is the normalized and convolved pixel value. After the normalization operation on the image data, the data will follow the feature distribution with the standard deviation of 1.
As shown in fig. 5, the noise reduction network provided in this embodiment is designed to have 5 layers, where the first 4 layers of the network are composed of Conv + BN + Relu, the size of the convolution kernel filter is 7 × 7, the number of filters is 128, the 5 th layer is Conv + BN, the size of the convolution kernel filter is 7 × 7, and the number of filters is 1. After training, inputting a channel attention image in a jump link mode, carrying out information subtraction on the channel attention image and a noise image output by the noise reduction network to finally obtain a noise reduction image, and inputting the noise reduction image to the enhancement network.
The noise reduction network prediction error is measured using the L1 loss function.
L1＝||F′-I||
Where F' and I are the predicted noise image and the input image, respectively.
And thirdly, merging the channel attention image generated by the visual attention network and the de-noised image generated by the de-noising network with the original low-illumination image and inputting the merged image into the enhancement network.
The enhancement network is a core part of the present embodiment, and the purpose of the enhancement network is to decompose the enhancement problem into several sub-problems (such as noise elimination, texture preservation, color correction, etc.) in different aspects, and the enhancement network provided by the present embodiment is composed of three types of modules: feature extraction subnetworks (FEN), Enhancer Networks (EN), and fusion subnetworks (FN).
In which the feature extraction subnetwork, as shown in fig. 6, is a single-stream network with multiple convolutional layers, each convolutional layer using 3 × 3 kernels, with a step size of 1, and using the ReLU activation function, the output of each layer is the input of the next layer and the input size and output size are the same. The enhancement subnetwork is composed of 4 submodules, wherein the EM-1 network adopts a stack with a large convolution kernel and a deconvolution layer, and the U-net network can extract multi-level information from different depth layers and can reserve rich texture information. The EM-2 network adopts a jump U-net structure and synthesizes a high-quality graph by using multi-scale context information; the EM-3 network adopts a similar Res-net structure, only uses a plurality of Res blocks for enhancement, and reduces model parameters; the EM-4 network uses an expansion convolutional layer, whose input size is always the same as the output size.
The blending sub-network, as shown in FIG. 7, receives the outputs of all the enhancement sub-networks and produces the final enhanced image. The fusion sub-network uses a weighted summation of scientific weights, the principle being to concatenate all outputs from the EN network in the color channel dimension and combine them using a 1 x 1 convolution kernel.
Since the use of a common loss function in the enhancement network may cause blurring of the image due to the low brightness of the image, a new loss function is designed, which includes four parts, to improve the visual quality.
Le＝ωebLeb+ωesLes+ωepLep+werLer
Wherein L iseb,Les,Lep,LerRepresenting the loss of brightness, structural loss, perceptual loss and regional loss, respectively. Omegaeb,ωes,ωep,ωerRespectively, are the corresponding coefficients.
The purpose of the luminance loss is to ensure that the enhancement results have sufficient luminance. The loss function is as follows:
Leb＝||F′-I)||
where F' is the prediction enhanced picture and I is the input picture.
The structural loss is introduced to preserve the structural features and avoid image blurring, and the present embodiment adopts SSIM as a loss function of the structural features.
Wherein, mux,μyIs the average value of the pixel or pixels,
is the variance, σxyIs the covariance, C1,C2Is a bias constant to prevent the denominator from being zero. Where x, y represent the input image and the output image, respectively.
Perceptual loss is the use of more channel information to improve visual quality, with a loss function as follows:
where F', F are the predicted enhanced image and the original image, respectively, wij,hij,cijThe size and the channel of the image are described separately,
a feature map obtained for the jth convolutional layer in the ith block is shown.
In addition to enhancing the image as a whole, attention should be paid to underexposed low-light areas, and the regional loss function proposed by the method is used to balance the degree of low-light enhancement of different areas. The loss function is as follows:
Ler＝||F′·F′-F·F||+1-ssim(F′·F′,F·F)
wherein, the ssim () algorithm represents the image structure similarity, F', F represent the enhanced image and the original image respectively, and the image quality is measured from three aspects of brightness, contrast and structure respectively. F' is the predicted attention map.
Compared with the prior art, the invention has the following advantages:
the method has the advantages that: the effect of the image on color and detail can be enhanced.
The method has the advantages that: the details of dark areas of the image can be enhanced, and the later use efficiency of the image is increased.
The method has the advantages that: compared with other enhancement methods, the method has better robustness.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.
Claims (10)
1. A method of enhancing a low-illuminance image, comprising:
acquiring a low-illumination image;
inputting the low-illumination image into a visual attention network to perform brightness information extraction processing so as to generate a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high brightness region and a low brightness region;
inputting the channel attention image into a noise reduction network to perform noise reduction processing so as to generate a noise reduction image;
inputting the low-light image, the channel attention image and the de-noised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function.
2. The method according to claim 1, wherein the inputting the low-illuminance image into a visual attention network for luminance information extraction processing to generate a channel attention image specifically comprises:
converting the low-illumination image into an HSI feature space to obtain a multi-channel feature map;
inputting the multi-channel feature map into an average pooling layer to obtain an average pixel value of each channel;
inputting the multichannel feature map into a maximum pooling layer to obtain a maximum pixel value of each channel;
inputting the average pixel value and the maximum pixel value of each channel into a multilayer perceptron to generate a high-brightness region feature vector and a low-brightness region feature vector;
respectively carrying out summation operation on the high-brightness region feature vector and the low-brightness region feature vector elements to obtain a first feature vector and a second feature vector;
and processing the first characteristic vector and the second characteristic vector by adopting a sigmoid function to obtain a channel attention image.
3. The method of claim 1, wherein the inputting the channel attention image into a noise reduction network for performing noise reduction processing to generate a denoised image comprises:
inputting the channel attention image into a noise reduction network for noise reduction processing by adopting a jump link mode, and further generating a noise reduction image;
the noise reduction network is a 5-layer network, the first 4-layer network consists of Conv + BN + Relu, the number of filters used by the first 4-layer network is 128, the 5-layer network consists of Conv + BN, the number of filters used by the 5-layer network is 1, the filters are convolution kernel filters, and the size of each filter is 7 x 7.
4. The method for enhancing a low-illuminance image according to claim 1, wherein the inputting the low-illuminance image, the channel attention image and the denoised image into an enhancement network to generate an enhanced image comprises:
inputting the low-illumination image, the channel attention image and the denoising image into a feature extraction sub-network to extract a bright information feature, a structural information feature, a perceptual information feature and a regional information feature; the enhancement network comprises a feature extraction sub-network, an enhancement sub-network and a fusion sub-network which are connected in sequence;
inputting the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhancement feature information;
inputting all the enhanced feature information into the fused sub-network to obtain an enhanced image.
5. The method according to claim 4, wherein the inputting the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhancement feature information includes:
inputting the bright information feature into a first enhancement sub-network to obtain first enhancement feature information; the enhancer network comprises a first enhancement subnetwork, a second enhancement subnetwork, a third enhancement subnetwork, and a fourth enhancement subnetwork; the loss function of the first enhancement sub-network is a luminance loss sub-function;
inputting the structural information feature into the second enhancement sub-network to obtain second enhancement feature information; the loss function of the second enhancement sub-network is a structural loss sub-function;
inputting the perceptual information feature into the third enhancement sub-network to obtain third enhancement feature information; the penalty function of the third enhancement sub-network is a perceptual penalty sub-function;
inputting the regional information feature into the fourth enhancement sub-network to obtain fourth enhancement feature information; the loss function of the fourth enhancement sub-network is a regional loss sub-function.
6. A low-light image-oriented enhancement system, comprising:
the image acquisition module is used for acquiring a low-illumination image;
the channel attention image generation module is used for inputting the low-illumination image into a visual attention network to extract brightness information so as to generate a channel attention image; a visual attention mechanism is arranged in the visual attention network; the channel attention map image includes a high brightness region and a low brightness region;
the de-noising image generation module is used for inputting the channel attention image into a de-noising network to perform de-noising processing so as to generate a de-noising image;
an enhanced image generation module, configured to input the low-illumination image, the channel attention image, and the denoised image into an enhancement network to generate an enhanced image; the loss functions of the enhancement network include a luminance loss sub-function, a structural loss sub-function, a perceptual loss sub-function, and a regional loss sub-function.
7. The system according to claim 6, wherein the channel attention image generation module specifically comprises:
the conversion unit is used for converting the low-illumination image into an HSI (hue, saturation and intensity) feature space so as to acquire a multi-channel feature map;
the average pixel value calculation unit is used for inputting the multi-channel feature map into an average pooling layer so as to obtain an average pixel value of each channel;
the maximum pixel value calculation unit is used for inputting the multi-channel feature map into a maximum pooling layer so as to obtain the maximum pixel value of each channel;
a feature vector determination unit for inputting the average pixel value and the maximum pixel value of each channel into a multilayer perceptron to generate a high-luminance region feature vector and a low-luminance region feature vector;
the fusion unit is used for respectively carrying out summation operation on the high-brightness region feature vector and the low-brightness region feature vector elements to obtain a first feature vector and a second feature vector;
and the channel attention image generation unit is used for processing the first feature vector and the second feature vector by adopting a sigmoid function to obtain a channel attention image.
8. The system for enhancing a low-illuminance image according to claim 6, wherein the denoised image generating module specifically comprises:
the de-noising image generating unit is used for inputting the channel attention image into a de-noising network in a jump link mode to perform de-noising processing so as to generate a de-noising image;
the noise reduction network is a 5-layer network, the first 4-layer network consists of Conv + BN + Relu, the number of filters used by the first 4-layer network is 128, the 5-layer network consists of Conv + BN, the number of filters used by the 5-layer network is 1, the filters are convolution kernel filters, and the size of each filter is 7 x 7.
9. The system for enhancing a low-illuminance image according to claim 6, wherein the enhanced image generation module specifically comprises:
a feature extraction unit, configured to input the low-illuminance image, the channel attention image, and the denoised image into a feature extraction sub-network to extract a bright information feature, a structural information feature, a perceptual information feature, and a regional information feature; the enhancement network comprises a feature extraction sub-network, an enhancement sub-network and a fusion sub-network which are connected in sequence;
an enhanced feature information determination unit configured to input the bright information feature, the structural information feature, the perceptual information feature, and the regional information feature into the enhancement sub-network to obtain a plurality of enhanced feature information;
and the enhanced image generating unit is used for inputting all the enhanced characteristic information into the fusion sub-network to obtain an enhanced image.
10. The system for enhancing a low-illuminance image according to claim 9, wherein the enhancement feature information determining unit specifically includes:
a first enhancement feature information determination subunit for inputting the bright information feature into a first enhancement sub-network to obtain first enhancement feature information; the enhancer network comprises a first enhancement subnetwork, a second enhancement subnetwork, a third enhancement subnetwork, and a fourth enhancement subnetwork; the loss function of the first enhancement sub-network is a luminance loss sub-function;
a second enhanced feature information determining subunit, configured to input the structural information feature into the second enhanced sub-network to obtain second enhanced feature information; the loss function of the second enhancement sub-network is a structural loss sub-function;
a third enhancement feature information determining subunit, configured to input the perceptual information feature into the third enhancement sub-network to obtain third enhancement feature information; the penalty function of the third enhancement sub-network is a perceptual penalty sub-function;
a fourth enhanced feature information determining subunit, configured to input the regional information feature into the fourth enhanced sub-network to obtain fourth enhanced feature information; the loss function of the fourth enhancement sub-network is a regional loss sub-function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110287338.XA CN112927162A (en) | 2021-03-17 | 2021-03-17 | Low-illumination image oriented enhancement method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110287338.XA CN112927162A (en) | 2021-03-17 | 2021-03-17 | Low-illumination image oriented enhancement method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112927162A true CN112927162A (en) | 2021-06-08 |
Family
ID=76174886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110287338.XA Withdrawn CN112927162A (en) | 2021-03-17 | 2021-03-17 | Low-illumination image oriented enhancement method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112927162A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113450366A (en) * | 2021-07-16 | 2021-09-28 | 桂林电子科技大学 | AdaptGAN-based low-illumination semantic segmentation method |
| CN117670753A (en) * | 2024-01-30 | 2024-03-08 | 浙江大学金华研究院 | Infrared image enhancement method based on depth multi-brightness mapping non-supervision fusion network |
-
2021
- 2021-03-17 CN CN202110287338.XA patent/CN112927162A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113450366A (en) * | 2021-07-16 | 2021-09-28 | 桂林电子科技大学 | AdaptGAN-based low-illumination semantic segmentation method |
| CN113450366B (en) * | 2021-07-16 | 2022-08-30 | 桂林电子科技大学 | AdaptGAN-based low-illumination semantic segmentation method |
| CN117670753A (en) * | 2024-01-30 | 2024-03-08 | 浙江大学金华研究院 | Infrared image enhancement method based on depth multi-brightness mapping non-supervision fusion network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111741211B (en) | Image display method and apparatus | |
| CN109636754B (en) | Extremely-low-illumination image enhancement method based on generation countermeasure network | |
| WO2022042049A1 (en) | Image fusion method, and training method and apparatus for image fusion model | |
| EP2987134B1 (en) | Generation of ghost-free high dynamic range images | |
| KR101633893B1 (en) | Apparatus and Method for Image Fusion | |
| KR101257942B1 (en) | Pre-processing method and apparatus in Wide Dynamic Range image processing | |
| US10699395B2 (en) | Image processing device, image processing method, and image capturing device | |
| CN103854259B (en) | Image processing equipment and the method for handling image | |
| US10055872B2 (en) | System and method of fast adaptive blending for high dynamic range imaging | |
| Monakhova et al. | Dancing under the stars: video denoising in starlight | |
| US20090180704A1 (en) | Method and system for processing low-illuminance image | |
| CN110619593A (en) | Double-exposure video imaging system based on dynamic scene | |
| Liu et al. | Survey of natural image enhancement techniques: Classification, evaluation, challenges, and perspectives | |
| JP7253634B2 (en) | High dynamic range image generation by pre-combination denoising | |
| US8456541B2 (en) | Image processing apparatus and image processing program | |
| CN117408890B (en) | Video image transmission quality enhancement method and system | |
| CN112927162A (en) | Low-illumination image oriented enhancement method and system | |
| Priyanka et al. | Low-light image enhancement by principal component analysis | |
| CN114820405A (en) | Image fusion method, device, equipment and computer readable storage medium | |
| CN111242860A (en) | Super night scene image generation method and device, electronic equipment and storage medium | |
| CN113643214A (en) | Image exposure correction method and system based on artificial intelligence | |
| CN110246086B (en) | Image processing method for color image enhancement and image processing system thereof | |
| US11640654B2 (en) | Image processing method and apparatus | |
| KR102395305B1 (en) | Method for improving low illumination image | |
| Srinivas et al. | Channel prior based Retinex model for underwater image enhancement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210608 |
|
| WW01 | Invention patent application withdrawn after publication |