Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a method and a system for enhancing a single low-light image based on Retinex and a convolutional neural network, which can effectively enhance the single low-light image, do not generate color distortion, can keep the texture details of an original image, and have good generalization capability on different data sets.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a single low-light image enhancement method based on Retinex and a convolutional neural network.
A single low-light image enhancement method based on Retinex and a convolutional neural network comprises the following steps:
acquiring an image, preprocessing the image, separating three channels and acquiring a hue component, a saturation component and a brightness component;
obtaining an illumination component according to the lightness component by adopting the trained deep convolution neural network model;
calculating a reflection component of the illumination component by utilizing a Retinex theory;
and recombining the reflection component with the hue component and the saturation component to obtain a three-channel image in the HSV color space.
And converting the three-channel image in the HSV color space into an RGB color space to obtain an enhanced low-illumination image.
Further, the process of obtaining the enhanced low-light image comprises: and converting the three-channel image in the HSV color space into an RGB color space, and adjusting the pixel range of the three-channel image from [0,1] to [0,255] to obtain the enhanced low-illumination image.
Further, the process of obtaining the hue component, the saturation component and the brightness component includes: normalizing the pixels of the image to [0,1], converting the normalized image from a color space RGB to an HSV space, and separating three channels to obtain hue components, saturation and brightness components.
Further, the process of training the deep convolutional neural network model includes:
constructing a deep convolutional neural network model, and establishing a target loss function of the neural network based on Retinex theory and prior hypothesis;
obtaining a logarithm brightness component after the brightness component is subjected to logarithm and normalization, and obtaining a bright channel prior component through the brightness component;
and (3) bringing the illumination component, the lightness component and the bright channel prior component into a target loss function, calculating an error, and realizing gradient updating on the weight and the parameters of the neural network by using an Adam optimization algorithm through the target loss function until the error is smaller than a set threshold or iteration reaches a preset number of times, and ending the model training.
The target loss function is:
E=Lossis+λ1Lossr+λ2Lossrs+λ3Losslc
therein, LossisRepresenting the Loss function of illumination smoothness, LossrRepresenting the reflection Loss function, LossrsRepresenting Loss of reflection smoothness function, LosslcRepresenting the prior loss function of the bright channel, λ1,λ2,λ3Respectively, reflection loss, reflection component smoothness loss, and bright channel prior loss.
As an embodiment, three weight values may be taken: lambda [ alpha ]1=0.05,λ2=0.1,λ30.5. It should be noted that the numerical values of the three weight values are only one embodiment of the present invention, and should not be construed as limiting the present invention.
The second aspect of the invention provides a single low-light image enhancement system based on Retinex and a convolutional neural network.
A single low-light image enhancement system based on Retinex and a convolutional neural network comprises:
an acquisition and pre-processing module configured to: acquiring an image, preprocessing the image, separating three channels and acquiring a hue component, a saturation component and a brightness component;
an illumination component obtaining module configured to: obtaining an illumination component according to the lightness component by adopting the trained deep convolution neural network model;
a reflected component obtaining module configured to: calculating a reflection component of the illumination component by utilizing a Retinex theory;
a reassembly module configured to: and recombining the reflection component with the hue component and the saturation component to obtain a three-channel image in the HSV color space.
An output module configured to: and converting the three-channel image in the HSV color space into an RGB color space to obtain an enhanced low-illumination image.
A third aspect of the invention provides a computer-readable storage medium.
A computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the Retinex and convolutional neural network-based single-low-illumination image enhancement method as described in the first aspect above.
A fourth aspect of the invention provides a computer apparatus.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the Retinex and convolutional neural network-based single low-light image enhancement method according to the first aspect when executing the program.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the invention, the low-illumination image is converted from the color space RGB to the color space HSV, the channels are separated, only the brightness component is enhanced, the color information is kept in the hue (H) and the saturation (S), and the color distortion in the image enhancing process is avoided.
2. The invention establishes the objective function of the deep learning neural network through Retinex theory and a series of prior, realizes the estimation of illumination component from the brightness component of a single image, further estimates the reflection component, thereby obtaining the enhanced image, completes the unsupervised image enhancement process, and has strong generalization capability.
3. The invention uses fractional order differentiation in the objective function of the neural network, and improves the retention capacity of the enhanced image details.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Detailed Description
The invention is further described with reference to the following figures and examples.
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example one
As shown in fig. 1, the embodiment provides a single low-light image enhancement method based on Retinex and convolutional neural network, and the embodiment is illustrated by applying the method to a server, it is understood that the method may also be applied to a terminal, and may also be applied to a system including a terminal and a server, and is implemented by interaction between the terminal and the server. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network server, cloud communication, middleware service, a domain name service, a security service CDN, a big data and artificial intelligence platform, and the like. The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein. In this embodiment, the method includes the steps of:
step 1: acquiring an image, preprocessing the image, separating three channels and acquiring a hue component, a saturation component and a brightness component;
the method comprises the following steps of preprocessing the acquired low-illumination image:
step 1.1, in the stage of reading in the low-illumination image, the pixel value range of the read image is changed into [0,1] by using a special read-in function; or create a normalization function to normalize pixels of the low-light image from [0,255] to [0,1 ].
Step 1.2: converting the image obtained in the step 1.1 from a color space RGB to a color space HSV, and extracting three channel components: hue (H), saturation (S), lightness (V). The specific implementation method comprises the following steps:
step 1.2.1: calculating the maximum channel (c) of the imagemax) Minimum channel (c)min) And a contrast (Δ), the calculation formula being: c. Cmax=max(R,G,B),cmin=min(R,G,B),Δ=cmax-cmin(wherein R, G, B are the three channels of the color space RGB, with values at [0,1]]In (d) of (a);
step 1.2.2: the hue (H), saturation (S), lightness (V) are calculated according to the following formula:
V=cmax。
step 1.3: the luminance component is logarithmized using a function y ═ log (x +1), and the result is normalized to obtain a logarithmized luminance component (v).
Step 1.4: calculating to obtain a prior component (v) of a bright channel by a formula
light) The formula is as follows:
where Ω is a k × k region centered at (i, j),
and v
p,qRepresenting the pixel at the specified location.
Step 2: obtaining an illumination component according to the lightness component by adopting the trained deep convolution neural network model;
the method comprises the following specific steps of constructing a deep convolutional neural network model:
step 2.1: and constructing a deep convolutional network. The network structure of the invention is shown in fig. 2, the input is the lightness component (v) obtained in step 1.3, the output is the illumination component (l), the first 4 layers of the network model are composed of a convolution operation and a ReLU function, the latter layer only has 1 convolution operation, and finally, a sigmoid layer is connected. The specific information of each layer of the network model is as follows:
TABLE 1 deep convolutional network
Step 2.2: and establishing an objective function of the neural network based on Retinex theory and prior hypothesis. Fig. 3 shows the imaging process of an object, which the Retinex theory shows as:
where S represents the acquired (or observed) image, R is the reflected component of the object, L is the illumination component of the environment,
representing multiplication by element. Under the assumption that the spatial variation of the illumination is smooth, a loss of illumination smoothness is proposed:
wherein l is the illumination component output by the network,
is v is
1(being positive) derivative of order, v
2Is a positive number and N is the total number of pixels of the image. By applying constraint to the fractional order gradient of the illumination component, the illumination component obtained by the neural network is smooth when the space changes.
In order to avoid too large difference between the obtained illumination image and the original image, the reflection loss is designed:
where l is the illumination component of the network output, v is the lightness component of the input, v is the luminance component of the network output2Is a positive number.
In order to make the spatial variation of the reflection component as smooth as possible, to guarantee its clarity and visual effect, a loss of reflection smoothness is proposed:
in order to have the value of the reflection component (R) between [0,1], it is necessary to guarantee that the illumination component (L) is greater than (V), thus proposing a bright channel prior penalty:
wherein v islightIs the bright channel prior component calculated from the luma component v,
in the above loss, the present invention uses fractional order gradient and fractional order differential to improve the retention of image texture details by the model.
And synthesizing the loss functions of all parts, wherein the target loss function of the model is as follows:
E=Lossis+λ1Lossr+λ2Lossrs+λ3Losslc
wherein λ is1,λ2,λ3The weights are respectively reflection loss, reflection component smoothness loss and bright channel prior loss, and the values of the three weights are respectively: lambda [ alpha ]1=0.05,λ2=0.1,λ30.5. The resulting illumination components can be spatially smoothed with 4 lossy constraints, preserving texture details.
Step 2.3: randomly initializing a network weight, and determining a weight optimization algorithm as an Adam optimization algorithm.
Wherein a neural network is trained. Inputting the lightness component (v) obtained in the step 1.3 into a convolution neural network, and obtaining an illumination component (l), a lightness component (v) and a bright channel prior component (v) by the neural networklight) And (3) carrying the error into the objective function in the step 2.2, calculating the error, and realizing gradient updating on the weight and the parameters of the neural network by using an Adam optimization algorithm through the objective function, wherein the iteration is stopped when the error meets the requirement or the iteration reaches a preset number.
And step 3: calculating a reflection component of the illumination component by utilizing a Retinex theory;
step 3.1: and (3) inputting the brightness component (v) obtained in the step (3.1) into the deep convolutional neural network trained in the step (3) to obtain an output illumination component (l).
Step 3.2: and (3) gamma correction is carried out on the illumination component obtained in the step (3.1), and then the lightness component (V) and the illumination component (L) obtained in the step (1.3) are subjected to indexing processing to obtain a lightness component (V) and an illumination component (L) after indexing. Obtaining a calculation formula of the reflection component according to Retinex theory: and R is V/L. The reflection component (R) is calculated by a calculation formula.
And 4, step 4: recombining the reflection component with the hue component and the saturation component to obtain a three-channel image in an HSV color space; and converting the three-channel image in the HSV color space into an RGB color space to obtain an enhanced low-illumination image.
Specifically, the reflection component (R) is used as lightness component (V), and combined with hue (H) and saturation (S) obtained in step 1.2 to restore an image enhanced in color space HSV, and then the image is converted from color space HSV to color space RGB, and then the image value range is changed from [0,1] to [0,255] by a special read-out function or a created mapping function.
Through the steps, the enhanced image can be obtained.
We performed tests on DICM low-light image datasets and evaluated network performance using peak Signal-to-Noise ratio psnr (peak Signal to Noise ratio) and structural similarity ssim (structural similarity index). Meanwhile, visualization and quantitative comparison are carried out with the current advanced algorithm, including a natural color retention enhancement algorithm NPE, a DONG based on defogging, an MF based on fusion, a LIME based on illumination component estimation, a BIMEF based on illumination component estimation and multi-exposure fusion, an SRIE based on reflection component and illumination component estimation, and an ULE, Retinex-Net and KinD based on deep learning.
The environment of the experiment is CPU processor Intel (R) Xeon (R) CPU E5-2620 v3@2.40GHz, memory 128G, video card Nvidia GeForce GTX TITAN X, and video memory 12G. The software system is Windows10, the deep learning framework of the pytorech, python3.8, CUDA version 10.0, cuDNN version 7.4. The development software used was Pycharm2020 and Matlab2020 a.
The learning rate of the parameter setting of this experiment is 0.001, and the total iteration number is 1000.
Fig. 4(a) - (k) illustrate the visual comparison results of different algorithms on a DICM dataset. Fig. 4(a) is an input low-light image, fig. 4(b) -fig. 4(j) are results of other methods, and fig. 4(k) is a method proposed by the present invention. Fig. 4 (b): BIMEF, fig. 4 (d): LIME, fig. 4 (e): MF, fig. 4 (f): NPE, fig. 4 (h): ULE and fig. 4 (i): the Retinex-Net method over-enhances the image and the enhanced image suffers from color distortion. Fig. 4 (c): DONG and fig. 4 (g): the consequences of SIRE are unnatural and in some places, excessive enhancement occurs. As can be seen from the image fig. 4(k), the method of the present embodiment retains the image details, makes the image more natural, and does not generate color distortion and excessive enhancement. By contrast, the method of the invention has better overall effect representation, and fine color and structure can be better recovered.
The quantitative test results on the DICM dataset are as follows:
TABLE 2 quantitative comparison on DICM datasets
Example two
The embodiment provides a single low-light image enhancement system based on Retinex and a convolutional neural network.
A single low-light image enhancement system based on Retinex and a convolutional neural network comprises:
an acquisition and pre-processing module configured to: acquiring an image, preprocessing the image, separating three channels and acquiring a hue component, a saturation component and a brightness component;
an illumination component obtaining module configured to: obtaining an illumination component according to the lightness component by adopting the trained deep convolution neural network model;
a reflected component obtaining module configured to: calculating a reflection component of the illumination component by utilizing a Retinex theory;
a reassembly module configured to: and recombining the reflection component with the hue component and the saturation component to obtain a three-channel image in the HSV color space.
An output module configured to: and converting the three-channel image in the HSV color space into an RGB color space to obtain an enhanced low-illumination image.
EXAMPLE III
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the single-low-illumination image enhancement method based on Retinex and convolutional neural network as described in the first embodiment above.
Example four
The embodiment provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the steps in the single-low-light image enhancement method based on Retinex and convolutional neural network as described in the first embodiment.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.