CN111161178A - Single low-light image enhancement method based on generation type countermeasure network - Google Patents

Single low-light image enhancement method based on generation type countermeasure network Download PDF

Info

Publication number
CN111161178A
CN111161178A CN201911361967.1A CN201911361967A CN111161178A CN 111161178 A CN111161178 A CN 111161178A CN 201911361967 A CN201911361967 A CN 201911361967A CN 111161178 A CN111161178 A CN 111161178A
Authority
CN
China
Prior art keywords
light image
low
image
gray
loss function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911361967.1A
Other languages
Chinese (zh)
Inventor
朱宁波
程秋锋
蒲斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan University
Original Assignee
Hunan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan University filed Critical Hunan University
Priority to CN201911361967.1A priority Critical patent/CN111161178A/en
Publication of CN111161178A publication Critical patent/CN111161178A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a single low-light image enhancement method based on a generating type countermeasure network, which comprises the following steps: collecting a training data set: acquiring a low-light image and a normal-light image from the same scene by changing the exposure time and the sensitivity of a camera, wherein the low-light image and the normal-light image of the same scene form an image group; processing of the data set: removing the misaligned image pair caused by uncontrollable factors such as camera shake, object movement and the like; constructing a low-light image enhancement model and a loss function based on a generative countermeasure network; training a low-light image enhancement model; and inputting the low-light image to be enhanced into the trained low-light image enhancement model to obtain an enhanced normal-light image. Compared with the related technology, the single low-light image enhancement method based on the generation type countermeasure network can obtain vivid and clear high-quality images, and the image generation speed is high.

Description

Single low-light image enhancement method based on generation type countermeasure network
Technical Field
The invention relates to the field of image enhancement, in particular to a single low-light image enhancement method based on a generative countermeasure network.
Background
High quality images, of course, play a crucial role in computer vision tasks such as object detection and scene understanding. However, images obtained in reality tend to be degraded in some cases, for example, images taken under low light conditions, always using very low contrast and brightness, which increases the difficulty of subsequent high-level tasks. In image capture, insufficient lighting can significantly reduce the visibility of the image, and lost detail and low contrast can not only cause an unpleasant subjective experience, but also impair the performance of many computer vision systems designed for normal-light instant messaging.
In the image acquisition process, due to various reasons such as a scanning system, a photoelectric conversion system or a field environment, the problems of insufficient illumination, too low light environment, limited performance of photographic equipment, improper equipment configuration and the like exist for a long time. In the past decades, many researchers have been working on the problem of low-light image enhancement, and in order to improve the subjective and objective quality of low-light images, many techniques have been developed, such as histogram equalization, retinal theory-based methods. In recent years, with the development of a deep neural network, more and more image enhancement technologies based on the deep neural network are proposed, a generation countermeasure network is also a network model which is proposed in recent years and is very suitable for solving image conversion, style migration and image generation, a plurality of excellent classical generation countermeasure networks are proposed so far, and the problem of low-light image enhancement by using the generation countermeasure network is very suitable.
However, existing generative countermeasure networks still suffer from some non-negligible drawbacks in addressing low-light image enhancement: firstly, the domain knowledge is excessively depended on, and the model complexity is high; secondly, the enhanced image has low quality, the detail texture is not clear enough, the color is not vivid enough, and the like.
Therefore, it is necessary to provide a new single low-light image enhancement method based on a generative countermeasure network to solve the above problems.
Disclosure of Invention
In response to the above-identified deficiencies in the art or needs for improvement, the present invention provides a single low-light image enhancement method based on a generative confrontation network.
A single low-light image enhancement method based on a generative countermeasure network comprises the following steps:
step S1, collecting a training data set: acquiring a low-light image and a normal-light image from the same scene by changing the exposure time and the sensitivity of a camera, wherein the low-light image and the normal-light image of the same scene form an image group;
step S2, processing of data set: removing the image groups which are not aligned due to uncontrollable factors such as camera shake, object movement and the like;
step S3, constructing a low-light image enhancement model and a loss function based on the generation type countermeasure network, wherein the low-light image enhancement model comprises a generator network G and a local discriminator network DlocalAnd global arbiter network Dglobal；
Step S4, training a low-light image enhancement model: randomly inputting the image pairs in the processed data set into the low-light image enhancement model for training, and repeatedly iterating through a minimum generator network G or a discriminator network DlocalAnd DglobalTraining the low-light image enhancement model until the low-light image enhancement model reaches a Nash equilibrium state, and finishing the training;
and step S5, inputting the low-light image to be enhanced into the trained low-light image enhancement model, and obtaining the enhanced normal light image.
Preferably, the step S1 specifically includes: in the same scene, two normal light images are firstly shot, which are recorded as N1 and N2, then a plurality of low light images are shot by reducing the exposure time and the sensitivity of the camera, and then the exposure time and the sensitivity of the camera are reset to shoot the two normal light images, which are recorded as N3 and N4, and N1, N2, N3, N4 and the plurality of low light images of the same scene form an image group.
Preferably, the step S2 includes the following steps:
step S21, by formula
Figure BDA0002335325480000031
Calculating a reference normal light image R of each image pair;
step S22, by formula
Figure BDA0002335325480000032
Calculating the mean square error M between N1-N4 and R:
step S23, determining whether the mean square error M of each image group exceeds a preset threshold, if so, removing the image group, and if not, retaining the image group.
Preferably, the step S4 includes the following steps:
step S41, randomly selecting an image group, randomly selecting a low light image and a real normal light image from the image group to form an image pair, and extracting the illumination intensity characteristics of the low light image and the low light image in the image pair;
step S42, inputting the illumination intensity characteristic and the low light image into a generator network G together to generate an enhanced normal light image;
step S43, using the generated normal light image and the real normal light image and the corresponding low light image and the illumination intensity feature as the discriminator network DlocalAnd DglobalThe input of (1);
step S44, according to the discriminator network DlocalAnd DglobalCalculating a loss function and optimizing parameters of the low-light image enhancement model according to the calculation result of the loss function;
and step S45, repeating the steps S41-S44 until the low-light image enhancement model reaches a Nash equilibrium state.
Preferably, the generator network G is a Unet + + network with a codec structure, and includes up-sampling, short-connection, long-connection, and down-sampling, the generator network G performs down-sampling for multiple times, and then performs down-sampling for multiple times, the same convolution kernel, step length, and padding are used for each down-sampling and up-sampling, the LeakyReLU function is used for activation after each down-sampling convolution, the ReLU function is used for each up-sampling convolution, the Tanh function is used for activation for the last layer, except that the first layer of down-sampling and the last layer of up-sampling all use instance regularization accelerated training, and the down-sampling and the up-sampling share features through short-connection and long-connection:
Figure BDA0002335325480000041
wherein x(i，j)Represents the jth characteristic diagram of the ith layer, and Conv represents convolution operation.
Preferably, the loss function includes a conditional generative confrontation loss function, a least squares GAN loss function, a L1 loss function, a content loss function, a color loss function, and a total loss function, wherein:
the conditional generative penalty function is:
Figure BDA0002335325480000042
wherein
Figure BDA0002335325480000043
Representing a mathematical expectation, x is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, D (x, x)grayY) is the discrimination output of the discriminator network D on the real normal light image, G (x, x)gray) For the normal light image generated, D (x, x)gray,G(x,xgray) Is the discrimination output of the discriminator network D for the generated normal light image;
the least squares GAN loss function is:
Figure BDA0002335325480000044
wherein
Figure BDA0002335325480000045
Representing a mathematical expectation, x is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) For the normal light image generated, D (x, x)grayY) is the discrimination output of the discriminator network D on the real normal light image, D (x, x)gray,G(x,xgray) For the discriminatory output of the discriminator network D on the generated normal light image,
Figure BDA0002335325480000046
representing a loss function that minimizes the arbiter network D,
Figure BDA0002335325480000047
a loss function representing a minimum generator network G;
the L1 loss function is:
Figure BDA0002335325480000048
wherein
Figure BDA0002335325480000051
Representing the mathematical expectation, | | | | purple1Expressing L1 loss function for specification of y and G (x, x)gray) X is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) Is the generated normal light image;
the content loss function is:
Figure BDA0002335325480000052
wherein
Figure BDA0002335325480000053
Representing the mathematical expectation, | | | | purple2Expressing the L2 loss function for the specification of Φ (y) and Φ (G (x, x)gray) X is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) For the generated normal light image, Φ (y) represents a feature map of a real normal light image extracted using the VGG19 network, Φ (G (x, x)gray) A feature map representing a generated normal light image extracted using a VGG19 network;
the color loss function is:
Figure BDA0002335325480000054
wherein
Figure BDA0002335325480000055
Representing the mathematical expectation, | | | | purple2Representing L2 loss function for specification G (x, x)gray)blurAnd yblurX is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray)blurIndicating that the generated normal light image is taken with Gaussian blur, yblurRepresenting that the real normal light image is subjected to Gaussian blur;
the overall loss function is:
Figure BDA0002335325480000056
in the formula ofl1、λcontent、λcolorWeighting parameters of L1 loss, content loss and color loss respectively, and taking lambda asl1＝100、λcontent＝10、λcolor＝100，
Figure BDA0002335325480000057
A loss function representing a least squares GAN is shown,
Figure BDA0002335325480000058
the L1 loss function is represented,
Figure BDA0002335325480000059
a function representing the loss of content is represented,
Figure BDA0002335325480000061
representing a color loss function.
Preferably, in step S41, the image is preprocessed before being input into the low-light image enhancement model for training, where the preprocessing includes normalizing the low-light image and the real normal-light image, and extracting the illumination intensity feature of the low-light image for training, specifically: will lower the light patternThe pixel matrix is divided by 255 so that the pixel values are normalized to [0, 1%]Is marked as xscaleThen using the formula
Figure BDA0002335325480000062
Each element is distributed to [ -1,1 [ ]]Where channel represents the three channels, x, of the imagenormExpressing normalized low light image, mean 0.5, std 0.5, true normal light image also making normalization operation, finally using formula
Figure BDA0002335325480000063
Obtaining the illumination intensity characteristic, wherein r is xnorm R+1，g＝xnorm G+1，b＝xnorm B+1，xnorm R、xnorm GAnd xnorm BThe light is the value, x, of the red channel image, the green channel image and the blue channel image of the low-light image after normalization operationgrayRepresenting the extracted illumination intensity characteristics.
Compared with the related art, the single low-light image enhancement method based on the generation type countermeasure network has the following beneficial effects:
1. the improved Unet + + network is used as a generator network, so that not only can image characteristics of each level be learned, but also the model parameters are fewer, and the model training speed and the image generation speed are greatly improved;
2. problems in the generation process are comprehensively considered, and various loss functions are applied to enable the generation network to generate vivid and clear high-quality images;
3. an attention mechanism is introduced, so that the network can sense the illumination intensity of each area of the low-light image, the illumination intensity of each area of the generated image is adaptively adjusted, and a vivid normal-light image is generated.
Drawings
FIG. 1 is a flow chart of a single low-light image enhancement method based on a generative countermeasure network according to the present invention;
FIG. 2 is a schematic diagram of a generator network in the low-light image enhancement model according to the present invention;
FIG. 3 is a schematic diagram of a structure of a discriminator network in the low-light image enhancement model according to the present invention;
fig. 4 is a comparison of a low light image and an image enhanced using the method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Referring to fig. 1, the present invention provides a single low-light image enhancement method based on a generative countermeasure network, comprising the following steps:
step S1, collecting a training data set: acquiring a low-light image and a normal-light image from the same scene by changing the exposure time and the sensitivity of a camera, wherein the low-light image and the normal-light image of the same scene form an image group;
specifically, in the same scene, two normal light images are firstly shot, which are recorded as N1 and N2, then a plurality of low light images are shot by reducing the exposure time and the sensitivity of the camera, and then the two normal light images are shot by resetting the exposure time and the sensitivity of the camera, which are recorded as N3 and N4, and N1, N2, N3, N4 and the plurality of low light images of the same scene form an image group. The more image groups in the data set, the better, different scenes can be changed, and a plurality of image groups are collected.
Step S2, processing of data set: removing the image groups which are not aligned due to uncontrollable factors such as camera shake, object movement and the like;
specifically, the step S2 includes the following steps:
step S21, by formula
Figure BDA0002335325480000081
Calculating a reference normal light image R of each image pair;
step S22, by formula
Figure BDA0002335325480000082
Calculating the mean square error M between N1-N4 and R:
step S23, determining whether the mean square error M of each image group exceeds a preset threshold, if so, removing the image group, and if not, retaining the image group. Preferably, the threshold is typically set to 0.1.
The processed data set was randomly divided into 2 parts, 90% of which were used as training set (Train set) and 10% of which were used as verification set (Validation set).
Step S3, constructing a low-light image enhancement model and a loss function based on the generation type countermeasure network, wherein the low-light image enhancement model comprises a generator network G and a discriminator network D, and the discriminator network D comprises a local discriminator network DlocalAnd global arbiter network Dglobal；
In particular, the model design is based on a generative countermeasure network, comprising a generator network G consisting of a modified Unet + + network, a local arbiter network DlocalAnd a global arbiter network Dglobal。
Please refer to fig. 2, which is a schematic structural diagram of a generator network in the low-light image enhancement model according to the present invention. The generator network G adopts a Unet + + network with a codec structure, including down-sampling, short connection, long connection, and up-sampling. The down sampling can increase robustness to small disturbances of an input image, reduce the risk of overfitting, reduce the amount of calculation, increase the size of a receptive field, the up sampling can restore and decode abstract features to the size of an original image, finally, a segmentation result is obtained, and the importance of the features of different depths of the image can be automatically learned and utilized through short connection and long connection. The invention improves the structure of the Unet + + network, so that the Unet + + network can be used as a generation network to complete the function of generating pictures.
Preferably, the generator network G performs downsampling for a plurality of times, and then performs downsampling for a plurality of times, each downsampling and upsampling uses the same convolution kernel, step size and padding, for example, each downsampling uses a convolution kernel of 4x4, the step size is 2, the padding is 1, each downsampling is performed, the image size becomes half of the original image size, the number of channels becomes twice of the original image size (the number of channels is set to 64 in the first downsampling), then performs upsampling for four times, the convolution kernel of 4x4 is also used, the step size is 2, the padding is 1, the image size becomes twice of the original image size in each upsampling, and the number of channels becomes half of the original image size (the number of channels in the last downsampling is 3. The method comprises the following steps that after convolution every time of downsampling, an LeakyReLU function is used for activation, after convolution every time of upsampling, a ReLU function is used, the last layer is activated by using a Tanh function, except that the first layer of downsampling and the last layer of upsampling are both trained in an acceleration mode by using an instance normalization, and features are shared between downsampling and upsampling through short connection and long connection:
Figure BDA0002335325480000091
wherein x(i，j)Represents the jth characteristic diagram of the ith layer, and Conv represents convolution operation.
Please refer to fig. 3, which is a schematic diagram of a structure of a discriminator network in a low-light image enhancement model according to the present invention, wherein a dual-scale discriminator network is adopted, one of which is a local discriminator network DlocalAnother global arbiter network Dglobal. Local arbiter network DlocalAnd global arbiter network DglobalThe invention takes the generated normal light image and the real normal light image as well as the corresponding low light image and the illumination intensity characteristic as input to be sent into a local discriminator network DlocalTraining, simultaneously performing 2-time down-sampling on the generated normal light image, the real normal light image, the corresponding low light image and the illumination intensity characteristic, and sending the images into a global discriminator network DglobalTraining, although the structures of the arbiter networks are all the same, the global arbiter network DglobalHas the maximum receptive fieldThis means that there is more global perspective information of the image, which can lead the generator network to generate globally consistent images, and on the other hand, the local discriminator network DlocalCapturing high frequency information such as local textures and patterns encourages the generator network to be able to generate finer details. In the embodiment of the invention, the arbiter network structure uses the concept of PatchGan, and uses a five-layer convolution structure, the first three layers use convolution kernels of 4x4, the step size is 2, the padding is 1, the second two layers set the step size to 1, each layer uses LeakyReLU as an activation function, except the first layer and the last layer use instance regularization accelerated training.
The loss function is designed as follows:
the resistance loss: the generation network G receives a low-light image, inputs and outputs a generation image, hopefully cheats the discriminator network as much as possible, and the discriminator network D distinguishes the real target image and the output of the generation network as much as possible, so that a game process of the two networks is formed, and the condition generation counteracts the loss function of the networks as follows:
Figure BDA0002335325480000101
wherein
Figure BDA0002335325480000102
Representing a mathematical expectation, x is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, D (x, x)grayY) is the discrimination output of the discriminator network D on the real normal light image, G (x, x)gray) For the normal light image generated, D (x, x)gray,G(x,xgray) Is output for the discriminator network D to discriminate the generated normal light image.
The generator network G wants the images generated by itself to fool the discriminator network D as much as possible, so it is desirable for G to be minimized
Figure BDA0002335325480000103
The discriminator network D is intended to distinguish between the normal light image and the image generated by the generator network as much as possibleSo it is desirable to maximize for D
Figure BDA0002335325480000104
In order to stabilize the training of the network, the invention adopts least square GAN loss as the countermeasure loss:
Figure BDA0002335325480000105
for arbiter network D, minimization
Figure BDA0002335325480000106
And the loss function of the upper half part enables the output value of the real target image after passing through the network D to be as close to 1 as possible, and the output value of the generated image after passing through the network D to be as close to 0 as possible. For the generator network G, minimize
Figure BDA0002335325480000107
The lower half loss function is such that the output value of the generated image after passing through the network D is as close to 1 as possible, where
Figure BDA0002335325480000108
Representing a mathematical expectation, x is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) For the normal light image generated, D (x, x)grayY) is the discrimination output of the discriminator network D on the real normal light image, D (x, x)gray,G(x,xgray) For the discriminatory output of the discriminator network D on the generated normal light image,
Figure BDA0002335325480000111
representing a loss function that minimizes the arbiter network D,
Figure BDA0002335325480000112
representing a loss function that minimizes the generator network G.
Loss of L1: in order to make the generated picture as close to the real target image as possible, the L1 loss of the generated image and the real target image is taken, and the formula is as follows:
Figure BDA0002335325480000113
wherein
Figure BDA0002335325480000114
Representing the mathematical expectation, | | | | purple1Expressing L1 loss function for specification of y and G (x, x)gray) X is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) Is the normal light image generated.
Calculating content loss by respectively extracting feature maps of a generated image and a target real image by using a pre-trained VGG19 network, wherein the deeper the feature maps, the more abstract the image features are, the feature maps of the 7 th convolutional layer are taken to calculate the content loss, and a function of the extracted features is defined as phi (x), so that the content loss function is as follows:
Figure BDA0002335325480000115
wherein
Figure BDA0002335325480000116
Representing the mathematical expectation, | | | | purple2Expressing the L2 loss function for the specification of Φ (y) and Φ (G (x, x)gray) X is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) For the generated normal light image, Φ (y) represents a feature map of a real normal light image extracted using the VGG19 network, Φ (G (x, x)gray) Characteristic map representing the generated normal light image extracted using the VGG19 network.
Color loss: in order to make the color of the generated image as close to the real target image as possible, the method firstly performs Gaussian blur on the generated image and the real target image, removes texture details and only retains color information, and then takes the L2 loss between the two as color loss:
Figure BDA0002335325480000117
wherein
Figure BDA0002335325480000118
Representing the mathematical expectation, | | | | purple2Representing L2 loss function for specification G (x, x)gray)blurAnd yblurX is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray)blurIndicating that the generated normal light image is taken with Gaussian blur, yblurIndicating that the true normal light image is gaussian blurred.
The overall loss function is:
Figure BDA0002335325480000121
in the formula ofl1、λcontent、λcolorWeighting parameters of L1 loss, content loss and color loss respectively, and taking lambda asl1＝100、λcontent＝10、λcolor＝100，
Figure BDA0002335325480000122
A loss function representing a least squares GAN is shown,
Figure BDA0002335325480000123
the L1 loss function is represented,
Figure BDA0002335325480000124
a function representing the loss of content is represented,
Figure BDA0002335325480000125
representing a color loss function.
Step S4, training a low-light image enhancement model: randomly inputting the image pairs in the processed data set into the low-light image enhancement model for training, and repeatedly iterating to generate the image pairs through minimizationDevice network G or discriminator network DlocalAnd DglobalTraining the low-light image enhancement model until the low-light image enhancement model reaches a Nash equilibrium state, and finishing the training;
specifically, the step S4 includes the following steps:
step S41, randomly selecting an image group, randomly selecting a low light image and a real normal light image from the image group to form an image pair, and extracting the illumination intensity characteristics of the low light image and the low light image in the image pair;
step S42, inputting the illumination intensity characteristic and the low light image into a generator network G together to generate an enhanced normal light image;
step S43, using the generated normal light image and the real normal light image as the discriminator network DlocalAnd DglobalThe input of (1);
step S44, according to the discriminator network DlocalAnd DglobalCalculating a loss function, and adjusting parameters of the low-light image enhancement model according to the calculation result of the loss function;
and step S45, repeating the steps S41-S44 until the low-light image enhancement model reaches a Nash equilibrium state.
The image is preprocessed before being input into the low-light image enhancement model for training, wherein the preprocessing comprises normalizing the low-light image and a real normal-light image, and simultaneously extracting the illumination intensity characteristic of the low-light image for training.
The method specifically comprises the following steps: dividing the low-light image matrix by 255 normalizes the pixel values to [0,1]Is marked as xscaleThen using the formula
Figure BDA0002335325480000131
Each element is distributed to [ -1,1 [ ]]Where channel represents the three channels, x, of the imagenormRepresenting the normalized low-light image, mean is 0.5, std is 0.5, the true normal-light image is also subjected to the normalization operationDo, use the formula finally
Figure BDA0002335325480000132
Obtaining the illumination intensity characteristic, wherein r is xnorm R+1，g＝xnorm G+1，b＝xnorm B+1，xnorm R、xnorm GAnd xnorm BThe light is the value, x, of the red channel image, the green channel image and the blue channel image of the low-light image after normalization operationgrayRepresenting the extracted illumination intensity characteristics.
And step S5, inputting the low-light image to be enhanced into the trained low-light image enhancement model to obtain the enhanced normal-light image.
And randomly selecting the low-light images in the verification set, and inputting the low-light images into a trained low-light image enhancement model for verification.
FIG. 4 is a graph showing a comparison of a low light image and an image enhanced using the method of the present invention. It is obvious from fig. 4 that the generated image enhanced by the method of the present invention is close to the normal light image, the generated image has high image quality, clear detail texture and vivid color, which shows that the method of the present invention can obtain high quality image.
Compared with the related art, the single low-light image enhancement method based on the generation type countermeasure network has the following beneficial effects:
1. the improved Unet + + network is used as a generator network, so that not only can image characteristics of each level be learned, but also the model parameters are fewer, and the model training speed and the image generation speed are greatly improved;
2. problems in the generation process are comprehensively considered, and various loss functions are applied to enable the generation network to generate vivid and clear high-quality images;
3. an attention mechanism is introduced, so that the network can sense the illumination intensity of each area of the low-light image, the illumination intensity of each area of the generated image is adaptively adjusted, and a vivid normal-light image is generated.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (7)

1. A single low-light image enhancement method based on a generative countermeasure network is characterized by comprising the following steps:
step S1, collecting a training data set: acquiring a low-light image and a normal-light image from the same scene by changing the exposure time and the sensitivity of a camera, wherein the low-light image and the normal-light image of the same scene form an image group;
step S2, processing of data set: removing the image groups which are not aligned due to uncontrollable factors such as camera shake, object movement and the like;
step S3, constructing a low-light image enhancement model and a loss function based on the generation type countermeasure network, wherein the low-light image enhancement model comprises a generator network G and a discriminator network D, and the discriminator network D comprises a local discriminator network DlocalAnd global arbiter network Dglobal；
Step S4, training a low-light image enhancement model: randomly inputting the image group in the processed data set into the low-light image enhancement model for training, and repeatedly iterating to train the low-light image enhancement model through the loss function value of the minimum generator network G or the discriminator network D until the low-light image enhancement model reaches a Nash equilibrium state, and finishing training;
and step S5, inputting the low-light image to be enhanced into the trained low-light image enhancement model, and obtaining the enhanced normal light image.
2. The single low-light image enhancement method according to claim 1, wherein the step S1 is specifically as follows: in the same scene, two normal light images are firstly shot, which are recorded as N1 and N2, then a plurality of low light images are shot by reducing the exposure time and the sensitivity of the camera, and then the exposure time and the sensitivity of the camera are reset to shoot the two normal light images, which are recorded as N3 and N4, and N1, N2, N3, N4 and the plurality of low light images of the same scene form an image group.
3. The single low-light image enhancement method according to claim 2, wherein the step S2 includes the steps of:
step S21, by formula
Figure FDA0002335325470000021
Calculating a reference normal light image R of each image pair;
step S22, by formula
Figure FDA0002335325470000022
Calculating the mean square error M between N1-N4 and R:
step S23, determining whether the mean square error M of each image group exceeds a preset threshold, if so, removing the image group, and if not, retaining the image group.
4. The single low-light image enhancement method according to claim 3, wherein the step S4 comprises the steps of:
step S41, randomly selecting an image group, randomly selecting a low light image and a real normal light image from the image group to form an image pair, and extracting the illumination intensity characteristics of the low light image and the low light image in the image pair;
step S42, inputting the illumination intensity characteristic and the low light image into a generator network G together to generate an enhanced normal light image;
step S43, using the generated normal light image, the real normal light image, the corresponding low light image and the illumination intensity characteristic as the discriminator network DlocalAnd DglobalThe input of (1);
step S44, according to the discriminator network DlocalAnd DglobalCalculating a loss function and optimizing parameters of the low-light image enhancement model according to the calculation result of the loss function;
and step S45, repeating the steps S41-S44 until the low-light image enhancement model reaches a Nash equilibrium state.
5. The method for enhancing a single low-light image according to claim 1, wherein the generator network G is a net + + network with a codec structure and includes up-sampling, short-connection, long-connection and down-sampling, the generator network G performs down-sampling a plurality of times and then down-sampling a plurality of times, the same convolution kernel, step size and padding are used for each down-sampling and up-sampling, the down-sampling is activated by using a LeakyReLU function after each convolution, the up-sampling is activated by using a ReLU function after each convolution, the last layer is activated by using a Tanh function, except that the first layer of the down-sampling and the last layer of the up-sampling are both accelerated training by using instance regularization, and the features are shared between the down-sampling and the up-sampling through the short-connection and the long-connection:
Figure FDA0002335325470000031
wherein x(i，j)Represents the jth characteristic diagram of the ith layer, and Conv represents convolution operation.
6. The single low-light image enhancement method according to claim 4, wherein the loss function includes a conditional-generation countermeasure loss function, a least-squares GAN loss function, an L1 loss function, a content loss function, a color loss function, and a total loss function, wherein:
the conditional generative penalty function is:
Figure FDA0002335325470000032
wherein
Figure FDA0002335325470000033
Representing a mathematical expectation, x is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, D (x, x)grayY) is the discrimination output of the discriminator network D on the real normal light image, G (x, x)gray) For the normal light image generated, D (x, x)gray,G(x,xgray) Is the discrimination output of the discriminator network D for the generated normal light image;
the least squares GAN loss function is:
Figure FDA0002335325470000034
wherein
Figure FDA0002335325470000035
Representing a mathematical expectation, x is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) For the normal light image generated, D (x, x)grayY) is the discrimination output of the discriminator network D on the real normal light image, D (x, x)gray,G(x,xgray) For the discriminatory output of the discriminator network D on the generated normal light image,
Figure FDA0002335325470000036
representing a loss function that minimizes the arbiter network D,
Figure FDA0002335325470000037
a loss function representing a minimum generator network G;
the L1 loss function is:
Figure FDA0002335325470000041
wherein
Figure FDA0002335325470000042
Representing the mathematical expectation, | | | | purple1Expressing L1 loss function for the specification y andG(x,xgray) X is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) Is the generated normal light image;
the content loss function is:
Figure FDA0002335325470000043
wherein
Figure FDA0002335325470000044
Representing the mathematical expectation, | | | | purple2Expressing the L2 loss function for the specification of Φ (y) and Φ (G (x, x)gray) X is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray) For the generated normal light image, Φ (y) represents a feature map of a real normal light image extracted using the VGG19 network, Φ (G (x, x)gray) A feature map representing a generated normal light image extracted using a VGG19 network;
the color loss function is:
Figure FDA0002335325470000045
wherein
Figure FDA0002335325470000046
Representing the mathematical expectation, | | | | purple2Representing L2 loss function for specification G (x, x)gray)blurAnd yblurX is a low light image, xgrayFor extracted illumination intensity features, y is the true normal light image, G (x, x)gray)blurIndicating that the generated normal light image is taken with Gaussian blur, yblurRepresenting that the real normal light image is subjected to Gaussian blur;
the overall loss function is:
Figure FDA0002335325470000047
in the formula ofl1、λcontent、λcolorWeighting parameters of L1 loss, content loss and color loss respectively, and taking lambda asl1＝100、λcontent＝10、λcolor＝100，
Figure FDA0002335325470000051
A loss function representing a least squares GAN is shown,
Figure FDA0002335325470000052
the L1 loss function is represented,
Figure FDA0002335325470000053
a function representing the loss of content is represented,
Figure FDA0002335325470000054
representing a color loss function.
7. The method of claim 4, wherein in step S41, the image is preprocessed before being input into the low-light image enhancement model for training, the preprocessing includes normalizing the low-light image and the real normal-light image, and extracting the illumination intensity feature of the low-light image for training, specifically: dividing the low-light image matrix by 255 normalizes the pixel values to [0,1]Is marked as xscaleThen using the formula
Figure FDA0002335325470000055
Each element is distributed to [ -1,1 [ ]]Where channel represents the three channels, x, of the imagenormExpressing normalized low light image, mean 0.5, std 0.5, true normal light image also making normalization operation, finally using formula
Figure FDA0002335325470000056
Obtaining the illumination intensity characteristic, wherein r is xnorm R+1，g＝xnorm G+1，b＝xnorm B+1，xnorm R、xnorm GAnd xnorm BThe normalized values, x, of the red channel image, the green channel image and the blue channel image of the low-light image respectivelygrayRepresenting the extracted illumination intensity characteristics.
CN201911361967.1A 2019-12-25 2019-12-25 Single low-light image enhancement method based on generation type countermeasure network Pending CN111161178A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911361967.1A CN111161178A (en) 2019-12-25 2019-12-25 Single low-light image enhancement method based on generation type countermeasure network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911361967.1A CN111161178A (en) 2019-12-25 2019-12-25 Single low-light image enhancement method based on generation type countermeasure network

Publications (1)

Publication Number Publication Date
CN111161178A true CN111161178A (en) 2020-05-15

Family

ID=70556629

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911361967.1A Pending CN111161178A (en) 2019-12-25 2019-12-25 Single low-light image enhancement method based on generation type countermeasure network

Country Status (1)

Country Link
CN (1) CN111161178A (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111640075A (en) * 2020-05-23 2020-09-08 西北工业大学 Underwater image occlusion removing method based on generation countermeasure network
CN111784596A (en) * 2020-06-12 2020-10-16 北京理工大学 General endoscope image enhancement method and device based on generation of antagonistic neural network
CN111798400A (en) * 2020-07-20 2020-10-20 福州大学 Non-reference low-illumination image enhancement method and system based on generation countermeasure network
CN111833268A (en) * 2020-07-10 2020-10-27 中国海洋大学 Underwater image enhancement method for generating countermeasure network based on conditions
CN111899193A (en) * 2020-07-30 2020-11-06 湖北工业大学 Criminal investigation photography system and method based on low-illumination image enhancement algorithm
CN111915525A (en) * 2020-08-05 2020-11-10 湖北工业大学 Low-illumination image enhancement method based on improved depth separable generation countermeasure network
CN112102186A (en) * 2020-09-07 2020-12-18 河海大学 Real-time enhancement method for underwater video image
CN112203064A (en) * 2020-09-30 2021-01-08 普联技术有限公司 Method and device for constructing color mapping relationship of different illumination intensities
CN112669242A (en) * 2021-03-16 2021-04-16 四川大学 Night scene restoration method based on improved image enhancement algorithm and generation countermeasure network
CN113592752A (en) * 2021-07-12 2021-11-02 四川大学 Road traffic optical stain image enhancement method and device based on countermeasure network
CN114065838A (en) * 2021-10-22 2022-02-18 中国科学院深圳先进技术研究院 Low-illumination obstacle detection method, system, terminal and storage medium
CN116091341A (en) * 2022-12-15 2023-05-09 南京信息工程大学 Exposure difference enhancement method and device for low-light image

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107704966A (en) * 2017-10-17 2018-02-16 华南理工大学 A kind of Energy Load forecasting system and method based on weather big data
CN109636754A (en) * 2018-12-11 2019-04-16 山西大学 Based on the pole enhancement method of low-illumination image for generating confrontation network
CN110223259A (en) * 2019-06-14 2019-09-10 华北电力大学（保定） A kind of road traffic fuzzy image enhancement method based on production confrontation network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107704966A (en) * 2017-10-17 2018-02-16 华南理工大学 A kind of Energy Load forecasting system and method based on weather big data
CN109636754A (en) * 2018-12-11 2019-04-16 山西大学 Based on the pole enhancement method of low-illumination image for generating confrontation network
CN110223259A (en) * 2019-06-14 2019-09-10 华北电力大学（保定） A kind of road traffic fuzzy image enhancement method based on production confrontation network

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
GUISIK KIM ET AL.: "Low-Lightgan: Low-Light Enhancement Via Advanced Generative Adversarial Network With Task-Driven Training", pages 2811 - 2815 *
KARIM ARMANIOUS ET AL.: "MedGAN: Medical Image Translation using GANs", pages 1 - 16 *
NINGBO ZHU: "Joint features classifier with genetic set for undersampled face recognition", pages 2987 *
TALHA IQBAL ET AL.: "Generative Adversarial Network for Medical Images (MI-GAN)", pages 1 - 11 *
YIFAN JIANG ET AL.: "EnlightenGAN: Deep Light Enhancement without Paired Supervision", pages 1 - 11 *
华淼 等: "基于二阶段生成对抗网络的人眼编辑", 《中国体视学与图像分析》, vol. 24, no. 4, pages 306 - 313 *

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111640075A (en) * 2020-05-23 2020-09-08 西北工业大学 Underwater image occlusion removing method based on generation countermeasure network
CN111784596A (en) * 2020-06-12 2020-10-16 北京理工大学 General endoscope image enhancement method and device based on generation of antagonistic neural network
CN111833268B (en) * 2020-07-10 2023-09-26 中国海洋大学 Underwater image enhancement method based on condition generation countermeasure network
CN111833268A (en) * 2020-07-10 2020-10-27 中国海洋大学 Underwater image enhancement method for generating countermeasure network based on conditions
CN111798400A (en) * 2020-07-20 2020-10-20 福州大学 Non-reference low-illumination image enhancement method and system based on generation countermeasure network
CN111798400B (en) * 2020-07-20 2022-10-11 福州大学 Non-reference low-illumination image enhancement method and system based on generation countermeasure network
CN111899193A (en) * 2020-07-30 2020-11-06 湖北工业大学 Criminal investigation photography system and method based on low-illumination image enhancement algorithm
CN111915525A (en) * 2020-08-05 2020-11-10 湖北工业大学 Low-illumination image enhancement method based on improved depth separable generation countermeasure network
CN111915525B (en) * 2020-08-05 2024-03-01 湖北工业大学 Low-illumination image enhancement method capable of generating countermeasure network based on improved depth separation
CN112102186A (en) * 2020-09-07 2020-12-18 河海大学 Real-time enhancement method for underwater video image
CN112102186B (en) * 2020-09-07 2024-04-05 河海大学 Real-time enhancement method for underwater video image
CN112203064A (en) * 2020-09-30 2021-01-08 普联技术有限公司 Method and device for constructing color mapping relationship of different illumination intensities
CN112669242A (en) * 2021-03-16 2021-04-16 四川大学 Night scene restoration method based on improved image enhancement algorithm and generation countermeasure network
CN113592752A (en) * 2021-07-12 2021-11-02 四川大学 Road traffic optical stain image enhancement method and device based on countermeasure network
CN114065838B (en) * 2021-10-22 2023-07-14 中国科学院深圳先进技术研究院 Low-light obstacle detection method, system, terminal and storage medium
CN114065838A (en) * 2021-10-22 2022-02-18 中国科学院深圳先进技术研究院 Low-illumination obstacle detection method, system, terminal and storage medium
CN116091341A (en) * 2022-12-15 2023-05-09 南京信息工程大学 Exposure difference enhancement method and device for low-light image
CN116091341B (en) * 2022-12-15 2024-04-02 南京信息工程大学 Exposure difference enhancement method and device for low-light image

Similar Documents

Publication Publication Date Title
CN111161178A (en) Single low-light image enhancement method based on generation type countermeasure network
Tian et al. Deep learning on image denoising: An overview
CN112233038B (en) True image denoising method based on multi-scale fusion and edge enhancement
Remez et al. Class-aware fully convolutional Gaussian and Poisson denoising
EP3948764B1 (en) Method and apparatus for training neural network model for enhancing image detail
CN111209952B (en) Underwater target detection method based on improved SSD and migration learning
CN110675328B (en) Low-illumination image enhancement method and device based on condition generation countermeasure network
CN110148088B (en) Image processing method, image rain removing method, device, terminal and medium
CN111915525B (en) Low-illumination image enhancement method capable of generating countermeasure network based on improved depth separation
CN112541877B (en) Defuzzification method, system, equipment and medium for generating countermeasure network based on condition
CN111127336A (en) Image signal processing method based on self-adaptive selection module
CN112507617B (en) Training method of SRFlow super-resolution model and face recognition method
CN109166102A (en) It is a kind of based on critical region candidate fight network image turn image interpretation method
CN111861894A (en) Image motion blur removing method based on generating type countermeasure network
CN113392711B (en) Smoke semantic segmentation method and system based on high-level semantics and noise suppression
CN111696049A (en) Deep learning-based underwater distorted image reconstruction method
CN111047543A (en) Image enhancement method, device and storage medium
CN114862698B (en) Channel-guided real overexposure image correction method and device
CN113284061B (en) Underwater image enhancement method based on gradient network
CN115641391A (en) Infrared image colorizing method based on dense residual error and double-flow attention
Saleh et al. Adaptive uncertainty distribution in deep learning for unsupervised underwater image enhancement
Zhao et al. A simple and robust deep convolutional approach to blind image denoising
CN116309178A (en) Visible light image denoising method based on self-adaptive attention mechanism network
Henz et al. Synthesizing camera noise using generative adversarial networks
Zhang et al. Underwater image enhancement using improved generative adversarial network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200515