CN114049264A - Dim light image enhancement method and device, electronic equipment and storage medium - Google Patents

Dim light image enhancement method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN114049264A
CN114049264A CN202111138589.8A CN202111138589A CN114049264A CN 114049264 A CN114049264 A CN 114049264A CN 202111138589 A CN202111138589 A CN 202111138589A CN 114049264 A CN114049264 A CN 114049264A
Authority
CN
China
Prior art keywords
image
target image
target
neural network
convolutional neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111138589.8A
Other languages
Chinese (zh)
Inventor
袁梦轲
庞有鑫
常玉春
严冬明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dalian University of Technology
Institute of Automation of Chinese Academy of Science
Original Assignee
Dalian University of Technology
Institute of Automation of Chinese Academy of Science
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dalian University of Technology, Institute of Automation of Chinese Academy of Science filed Critical Dalian University of Technology
Priority to CN202111138589.8A priority Critical patent/CN114049264A/en
Publication of CN114049264A publication Critical patent/CN114049264A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a dim light image enhancement method, a dim light image enhancement device, electronic equipment and a storage medium, wherein the method comprises the following steps: determining an illumination guide map of the target image based on RGB three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; inputting the target image and the illumination guide image into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold. The method and the device improve the enhancement effect of the target image by utilizing the complementary information between the detail perception characteristic and the structure perception characteristic, thereby avoiding the problems of overexposure and color shift.

Description

Dim light image enhancement method and device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for enhancing a dark image, an electronic device, and a storage medium.
Background
When a user shoots an image by using common hardware equipment such as a mobile phone, the user often encounters adverse conditions such as dim light and the like, so that the brightness of the image captured by the mobile phone is too low, objects and scenes are not clear, details are lost, blurring and the like are caused, the perception aesthetics is lacked, and subsequent visual analysis is seriously hindered, such as tasks of text recognition, object detection and the like. Therefore, how to process the dark light image to have enough brightness and satisfactory natural characteristics has been one of the important research points in the fields of computational photography, computer graphics, computer vision, and the like.
The existing deep learning method comprises a supervised learning mode and an unsupervised learning mode. The existing supervised learning mode utilizes a pair of images (a dim light image I and an aligned normal light image N) to learn the mapping relation between I and N; however, the model trained by using the pairing data in the supervised learning mode can only produce a good effect on the test images of the same type as the training data set, so that the test images cannot be generalized to wider scenes, and the limitation is large. The existing unsupervised learning mode adopts a dim light image and an unpaired normal light image as training data, restricts an enhanced result to be close to the normal light image through a Generative Adaptive Network (GAN), and restricts the enhanced result to keep the texture and content of the dim light image through perception loss; however, the conventional unsupervised learning method has a problem that overexposure and color shift occur in many originally bright places.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a dim light image enhancement method, a dim light image enhancement device, electronic equipment and a storage medium, which can improve the enhancement effect of a target image by utilizing complementary information between the detail perception feature and the structure perception feature of the target image and avoid the problems of overexposure and color shift.
The invention provides a dark light image enhancement method, which comprises the following steps:
under the condition that the illumination intensity of a target image is lower than a target threshold value, determining an illumination guide map of the target image based on three channels of red, green and blue (RGB) of the target image; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model;
the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
According to the dim image enhancement method provided by the invention, the determining of the illumination guide map of the target image based on the RGB three channels of the target image comprises the following steps:
performing image enhancement on the target image to obtain a first image;
normalizing the first image to obtain a second image; wherein the range of pixel values of the second image comprises [0, 1 ];
determining a grayscale map of the second image based on the RGB three channels of the second image;
and determining the illumination guide map of the second image based on the gray scale map of the second image, and determining the illumination guide map of the second image as the illumination guide map of the target image.
According to the dim light image enhancement method provided by the invention, the convolution neural network image enhancement model comprises the following steps: the system comprises a structure perception generator SAG, a detail perception generator DAG and a feature attention fusion module FAM;
the inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, including:
inputting the target image and the illumination guide map into the SAG to obtain the structural perception characteristic of the target image generated by the SAG;
inputting the target image and the illumination guide map into the DAG to obtain detail perception features of the target image generated by the DAG;
inputting the structure perception feature and the detail perception feature to the FAM to obtain a fusion feature of the structure perception feature and the detail perception feature;
and inputting the fusion features into the DAG to obtain an enhanced image output by the DAG.
According to the dim light image enhancement method provided by the invention, the convolutional neural network image enhancement model further comprises: the pixel intensity adjusting module IAM is used for adjusting the pixel intensity of the enhanced image output by the DAG; the IAM is located after the last convolutional layer of the DAG;
after the target image and the illumination guide map are input into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, the method comprises the following steps:
and inputting the enhanced image into the IAM to obtain the enhanced image which is output by the IAM and is subjected to pixel intensity adjustment.
According to the dim light image enhancement method provided by the invention, the inputting the target image and the illumination guide map into the SAG to obtain the structural perception feature of the target image generated by the SAG comprises the following steps:
normalizing the RGB three channels of the target image to be between [ -1,1 ];
connecting the normalized RGB three channels of the target image and the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the SAG to obtain the structural perception characteristics of the target image generated by the SAG;
wherein in the SAG jump connection, the illumination guide diagram is used for adaptively learning the structural consistency of the feature diagram; the SAG employs a supervised loss function comprising at least one of: a first global perceptual loss function, a first local perceptual loss function, a first global countermeasure loss function, a first local countermeasure loss function; and the first local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the first image blocks after the target image and the output image of the SAG are divided into the first image blocks with fixed sizes.
According to a dim light image enhancement method provided by the present invention, the inputting the target image and the illumination guide map into the DAG to obtain detail perception features of the target image generated by the DAG includes:
normalizing the RGB three channels of the target image to be between [ -1,1 ];
connecting the normalized RGB three channels of the target image with the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the DAG to obtain the detail perception characteristics of the target image generated by the DAG;
wherein the DAG does not change image resolution; the supervised loss function employed by the DAG includes at least one of: a second global perceptual loss function, a second local perceptual loss function, a second global countermeasure loss function, a second local countermeasure loss function; and the second local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the second image blocks after the target image and the output image of the DAG are divided into the second image blocks with fixed sizes.
According to the dim light image enhancement method provided by the invention, the step of inputting the structure perception feature and the detail perception feature into the FAM to obtain a fusion feature of the structure perception feature and the detail perception feature comprises the following steps:
and under the condition that the structure perception feature comprises at least two layers of reciprocal convolution results of the SAG, and the detail perception feature comprises at least two layers of positive number convolution results of the DAG, connecting the at least two layers of reciprocal convolution results of the SAG and the at least two layers of positive number convolution results of the DAG in a channel dimension based on the FAM, guiding the connected images by using the illumination guide graph with a single channel and the same resolution, and sequentially inputting the guided images into at least one layer of convolution kernel and a hyperbolic tangent activation function for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature.
The present invention also provides a dim light image enhancing apparatus, comprising:
the determination module is used for determining an illumination guide map of a target image based on three channels of red, green and blue (RGB) of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
the enhancement module is used for inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model;
the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
The present invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the dim image enhancement method as described in any one of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the dim image enhancement method as described in any one of the above.
The dim light image enhancement method, the dim light image enhancement device, the electronic equipment and the storage medium provided by the invention are based on the illumination guide image, and the convolution neural network image enhancement model is used for carrying out image enhancement on the target image with lower illumination intensity.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a dim light image enhancement method according to the present invention;
FIG. 2 is a second flowchart of a dim image enhancement method according to the present invention;
FIG. 3 is a third schematic flowchart of a dim light image enhancement method according to the present invention;
FIG. 4 is a schematic structural diagram of a dim light image enhancement device provided by the present invention;
fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides a dark light image enhancement method, which comprises the following steps: determining an illumination guide map of the target image based on red, green and blue (RGB) three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein, the illumination guide map is used for representing the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide image into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold. The method is based on the illumination guide image, the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image with lower illumination intensity, the convolutional neural network image enhancement model is obtained by taking an unpaired dim light image and a normal illumination image as training samples and training based on the detail perception characteristics and the structure perception characteristics of the training samples, the detail perception characteristics and the structure perception characteristics of the target image can be subjected to feature fusion, the enhancement effect of the target image is improved by utilizing complementary information between the detail perception characteristics and the structure perception characteristics, and therefore the problems of overexposure and color shift are avoided.
The dim light image enhancement method provided by the invention can be applied to usage scenes such as mobile phone photographing, industrial detection, background correction, automatic driving and the like. The dim image enhancement method of the present invention is described below in conjunction with fig. 1-3.
Fig. 1 is a schematic flow chart of a dim light image enhancement method provided by the present invention. As shown in fig. 1, the method includes:
step 101, determining an illumination guide map of a target image based on RGB three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
102, inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
Optionally, the illumination intensity of the dim light image is below a target threshold; the illumination intensity of the normal illumination image is greater than or equal to a target threshold.
The dim light image enhancement method provided by the invention is based on the illumination guide image, and uses the convolution neural network image enhancement model to carry out image enhancement on the target image with lower illumination intensity, and the convolution neural network image enhancement model takes the dim light image and the normal illumination image which are not paired as training samples and is obtained after training based on the detail perception characteristic and the structure perception characteristic of the training samples, so that the detail perception characteristic and the structure perception characteristic of the target image can be subjected to feature fusion, and the enhancement effect of the target image is improved by utilizing the complementary information between the detail perception characteristic and the structure perception characteristic, thereby avoiding the problems of overexposure and color shift.
Optionally, fig. 2 is a second schematic flow chart of the dim-light image enhancement method provided by the present invention. As shown in fig. 2, the method includes:
step 201, under the condition that the illumination intensity of a target image is lower than a target threshold value, performing image enhancement on the target image to obtain a first image;
step 202, normalizing the first image to obtain a second image; wherein the range of pixel values of the second image comprises [0, 1 ];
step 203, determining a gray scale map of the second image based on RGB three channels of the second image;
step 204, determining an illumination guide map of a second image based on the gray scale map of the second image, and determining the illumination guide map of the second image as the illumination guide map of the target image; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
step 205, inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
Optionally, the image enhancement operation on the target image comprises at least one of: luminance enhancement, chrominance enhancement, contrast enhancement, and sharpness enhancement.
Optionally, based on the RGB three channels of the second image, a Gray scale graph Gray of the second image is calculated by using formula (1):
Gray＝(0.299*R+0.587*G+0.114*B) (1)
wherein R, G, B represents three channels of the target image.
Optionally, based on the grayscale map of the second image, the lighting guide map GM of the second image is calculated by using formula (2), and then the lighting guide map of the second image is determined as the lighting guide map GM of the target image:
GM＝1-Gray (2)
it should be noted that, for the description and explanation of the step 205, reference may be made to the step 102, and the same technical effect can be achieved, and in order to avoid repetition, the description is not repeated here.
According to the dim light image enhancement method provided by the invention, before the illumination guide image of the target image is calculated, the image enhancement is firstly carried out on the target image, then the calculation of the gray scale image is carried out based on RGB three channels, and then the illumination guide image is calculated, so that the illumination guide image has higher distinguishability to the degree of enhancement required at different positions, and the probability of overexposure phenomenon at the original brighter position due to the integral enhancement of the image can be avoided.
Optionally, fig. 3 is a third schematic flow chart of the dim-light image enhancement method provided by the present invention. The convolutional neural network image enhancement model mentioned in the present invention may include: a Structure Aware Generator (SAG), a Detail Aware Generator (DAG), a feature attention fusion module (FAM); the FAM module can eliminate the difference between SAG and DAG advanced features, thereby better fusing structure and detail-aware features together. As shown in fig. 3, the dim light image enhancement method based on the convolutional neural network image enhancement model in the present invention includes:
step 301, determining an illumination guide map of a target image based on RGB three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
step 302, inputting the target image and the illumination guide map into an SAG (SAG), and obtaining the structure perception characteristic of the target image generated by the SAG;
step 303, inputting the target image and the illumination guide map into a DAG to obtain detail perception features of the target image generated by the DAG;
step 304, inputting the structure perception feature and the detail perception feature into an FAM to obtain a fusion feature of the structure perception feature and the detail perception feature;
and 305, inputting the fusion features into the DAG to obtain an enhanced image output by the DAG.
Optionally, the step 302 of inputting the target image and the illumination guide map into the SAG to obtain an implementation of a structure-aware feature of the target image generated by the SAG, includes: normalizing the RGB three channels of the target image to be between [ -1,1 ]; connecting the normalized RGB three channels of the target image and the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the SAG to obtain the structural perception characteristics of the target image generated by the SAG; wherein the SAG does not change image resolution; in the jump level connection of the SAG, utilizing the illumination guide diagram to adaptively learn the structural consistency of the feature diagram; the SAG employs a supervised loss function comprising at least one of: a first global perceptual loss function, a first local perceptual loss function, a first global countermeasure loss function, a first local countermeasure loss function; and the first local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the first image blocks after the target image and the output image of the SAG are divided into the first image blocks with fixed sizes.
Optionally, a first global perceptual loss function
Figure BDA0003283153440000101
Expressed as equation (3), the first local perceptual loss function
Figure BDA0003283153440000102
Expressed as equation (4), a first global penalty function
Figure BDA0003283153440000103
Expressed as equation (5), a first local penalty function DRa(x,z)、DRa(z, x) is represented by formula (6):
Figure BDA0003283153440000104
Figure BDA0003283153440000105
Figure BDA0003283153440000106
Figure BDA0003283153440000111
wherein the content of the first and second substances,
Figure BDA0003283153440000112
i, O are the results of the dark light image and SAG output respectively, philRepresenting a feature map extracted from a pre-trained VGG-16 model, wherein W and H are the resolution of the feature map, and l is the l-th layer of the VGG-16; x and z in equations (5) and (6) represent sampling from true and false distributions, respectively; σ (-) is a sigmoid function; c (-) is a non-transformed discrimination function.
Optionally, the inputting a target image and the illumination guide map into the DAG in step 303, and an implementation manner of obtaining the detail perception feature of the target image generated by the DAG may include: normalizing the RGB three channels of the target image to be between [ -1,1 ]; connecting the normalized RGB three channels of the target image with the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the DAG to obtain the detail perception characteristics of the target image generated by the DAG; wherein the DAG does not change image resolution; the supervised loss function employed by the DAG includes at least one of: a second global perceptual loss function, a second local perceptual loss function, a second global countermeasure loss function, a second local countermeasure loss function; and the second local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the second image blocks after the target image and the output image of the DAG are divided into the second image blocks with fixed sizes.
Optionally, the DAG employs global and local loss functions for supervisory purposes, including perceptual and opportune loss. Since the output forms of DAG and SAG are identical, the usage of the loss functions in DAG and SAG are identical.
Optionally, the step 304 of inputting the structure-aware feature and the detail-aware feature into the FAM, and an implementation manner of obtaining a fusion feature of the structure-aware feature and the detail-aware feature may include: and under the condition that the structure perception feature comprises at least two layers of reciprocal convolution results of the SAG, and the detail perception feature comprises at least two layers of positive number convolution results of the DAG, connecting the at least two layers of reciprocal convolution results of the SAG and the at least two layers of positive number convolution results of the DAG in a channel dimension based on the FAM, guiding the connected images by using the illumination guide graph with a single channel and the same resolution, sequentially inputting the guided images into at least one layer of convolution kernel and a hyperbolic tangent activation function for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature, and outputting the fusion feature of 2 times of the input channel number.
The dim light image enhancement method provided by the invention is based on the illumination guide graph, and uses the convolution neural network image enhancement model to perform image enhancement on a target image with lower illumination intensity, because the convolution neural network image enhancement model takes an unpaired dim light image and a normal illumination image as training samples, uses SAG to generate the structure perception characteristic of the target image, uses DAG to generate the detail perception characteristic of the target image, the detail perception characteristic and the structure perception characteristic of the target image are subjected to feature fusion, and the complementary information between the detail perception characteristic and the structure perception characteristic is utilized to jointly improve the enhancement effect of the target image, thereby avoiding the problems of overexposure and color shift.
Optionally, the convolutional neural network image enhancement model further comprises: the pixel intensity adjusting module IAM is used for adjusting the pixel intensity of the enhanced image output by the DAG; the IAM is located after the last convolutional layer of the DAG. And after inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, inputting the enhanced image into the IAM to obtain an enhanced image output by the IAM after adjusting the pixel intensity. The input of the IAM module is a preliminary enhancement result of 3 channels, and the output is a final enhancement result of 3 channels; the IAM is divided into an upper branch and a lower branch: one branch is based on convolution and Tanh and aims to continuously change the pixel value range into the range of [ -1,1] which is the same as the input range; and the other branch is based on convolution and ReLU, and aims to extract features so as to assist in adjusting the pixel intensity, and then final pixel intensity adjustment is carried out on the enhancement result based on IAM so as to obtain an enhanced image after pixel intensity adjustment.
The dim light image enhancement method provided by the invention can process the dim light image with lower illumination intensity in an unsupervised learning mode, and obtains a real and natural normal illumination image through image enhancement. The invention obtains better dark light image enhancement effect by adjusting the light guide diagram, combining the perception characteristics of two levels of detail and structure and fusing different types of perception characteristics.
The following describes the dim-light image enhancement device provided by the present invention, and the dim-light image enhancement device described below and the dim-light image enhancement method described above can be referred to correspondingly.
Fig. 4 is a schematic structural diagram of the dark image enhancement apparatus provided by the present invention, and as shown in fig. 4, the dark image enhancement apparatus 400 includes:
a determining module 401, configured to determine, based on three RGB channels of a target image, an illumination guide map of the target image when an illumination intensity of the target image is lower than a target threshold; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
an enhancement module 402, configured to input the target image and the illumination guide map into a convolutional neural network image enhancement model, so as to obtain an enhanced image output by the convolutional neural network image enhancement model;
the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
The dim light image enhancement device provided by the invention uses the convolutional neural network image enhancement model to perform image enhancement on a target image with lower illumination intensity based on the illumination guide diagram, and the convolutional neural network image enhancement model is obtained by taking an unpaired dim light image and a normal illumination image as training samples and training based on the detail perception characteristics and the structure perception characteristics of the training samples, so that the detail perception characteristics and the structure perception characteristics of the target image can be subjected to feature fusion, and the enhancement effect of the target image is improved by utilizing complementary information between the detail perception characteristics and the structure perception characteristics, thereby avoiding the problems of overexposure and color shift.
Optionally, the determining module 401 is specifically configured to:
performing image enhancement on the target image to obtain a first image;
normalizing the first image to obtain a second image; wherein the range of pixel values of the second image comprises [0, 1 ];
determining a grayscale map of the second image based on the RGB three channels of the second image;
and determining the illumination guide map of the second image based on the gray scale map of the second image, and determining the illumination guide map of the second image as the illumination guide map of the target image.
Optionally, the convolutional neural network image enhancement model includes: the system comprises a structure perception generator SAG, a detail perception generator DAG and a feature attention fusion module FAM;
the enhancement module 402 is specifically configured to:
inputting the target image and the illumination guide map into the SAG to obtain the structural perception characteristic of the target image generated by the SAG;
inputting the target image and the illumination guide map into the DAG to obtain detail perception features of the target image generated by the DAG;
inputting the structure perception feature and the detail perception feature to the FAM to obtain a fusion feature of the structure perception feature and the detail perception feature;
and inputting the fusion features into the DAG to obtain an enhanced image output by the DAG.
Optionally, the convolutional neural network image enhancement model further includes: the pixel intensity adjusting module IAM is used for adjusting the pixel intensity of the enhanced image output by the DAG; the IAM is located after the last convolutional layer of the DAG;
the enhancement module 402 is specifically configured to:
and inputting the enhanced image into the IAM to obtain the enhanced image which is output by the IAM and is subjected to pixel intensity adjustment.
Optionally, the enhancing module 402 is specifically configured to:
normalizing the RGB three channels of the target image to be between [ -1,1 ];
connecting the normalized RGB three channels of the target image and the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the SAG to obtain the structural perception characteristics of the target image generated by the SAG;
wherein in the SAG jump connection, the illumination guide diagram is used for adaptively learning the structural consistency of the feature diagram; the SAG employs a supervised loss function comprising at least one of: a first global perceptual loss function, a first local perceptual loss function, a first global countermeasure loss function, a first local countermeasure loss function; and the first local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the first image blocks after the target image and the output image of the SAG are divided into the first image blocks with fixed sizes.
Optionally, the enhancing module 402 is specifically configured to:
normalizing the RGB three channels of the target image to be between [ -1,1 ];
connecting the normalized RGB three channels of the target image with the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the DAG to obtain the detail perception characteristics of the target image generated by the DAG;
wherein the DAG does not change image resolution; the supervised loss function employed by the DAG includes at least one of: a second global perceptual loss function, a second local perceptual loss function, a second global countermeasure loss function, a second local countermeasure loss function; and the second local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the second image blocks after the target image and the output image of the DAG are divided into the second image blocks with fixed sizes.
Optionally, the enhancing module 402 is specifically configured to:
and under the condition that the structure perception feature comprises at least two layers of reciprocal convolution results of the SAG, and the detail perception feature comprises at least two layers of positive number convolution results of the DAG, connecting the at least two layers of reciprocal convolution results of the SAG and the at least two layers of positive number convolution results of the DAG in a channel dimension based on the FAM, guiding the connected images by using the illumination guide graph with a single channel and the same resolution, and sequentially inputting the guided images into at least one layer of convolution kernel and a hyperbolic tangent activation function for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature.
The dim-light image enhancement device in the present invention may be a device, a device with an operating system, or an electronic device, such as a mobile phone, or a component, an integrated circuit, or a chip in an electronic device.
The dim light image enhancement device provided by the invention can realize the processes realized by the method embodiments of fig. 1 to fig. 3, and achieve the same technical effect, and is not repeated here for avoiding repetition.
Fig. 5 is a schematic physical structure diagram of an electronic device provided in the present invention, and as shown in fig. 5, the electronic device 500 may include: a processor (processor)510, a communication Interface (Communications Interface)520, a memory (memory)530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may invoke logic instructions in memory 530 to perform a dim light image enhancement method, the method comprising: determining an illumination guide map of a target image based on RGB three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
Furthermore, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the dim image enhancement method provided by the above methods, the method comprising: determining an illumination guide map of a target image based on RGB three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the dim image enhancement method provided above, the method comprising: determining an illumination guide map of a target image based on RGB three channels of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of enhancing a scotopic image, comprising:
under the condition that the illumination intensity of a target image is lower than a target threshold value, determining an illumination guide map of the target image based on three channels of red, green and blue (RGB) of the target image; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model;
the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
2. The dim-light image enhancement method according to claim 1, wherein the determining the illumination guide map of the target image based on the RGB three channels of the target image comprises:
performing image enhancement on the target image to obtain a first image;
normalizing the first image to obtain a second image; wherein the range of pixel values of the second image comprises [0, 1 ];
determining a grayscale map of the second image based on the RGB three channels of the second image;
and determining the illumination guide map of the second image based on the gray scale map of the second image, and determining the illumination guide map of the second image as the illumination guide map of the target image.
3. The dim-light image enhancement method according to claim 1, wherein the convolutional neural network image enhancement model comprises: the system comprises a structure perception generator SAG, a detail perception generator DAG and a feature attention fusion module FAM;
the inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, including:
inputting the target image and the illumination guide map into the SAG to obtain the structural perception characteristic of the target image generated by the SAG;
inputting the target image and the illumination guide map into the DAG to obtain detail perception features of the target image generated by the DAG;
inputting the structure perception feature and the detail perception feature to the FAM to obtain a fusion feature of the structure perception feature and the detail perception feature;
and inputting the fusion features into the DAG to obtain an enhanced image output by the DAG.
4. The dim-light image enhancement method according to claim 3, wherein the convolutional neural network image enhancement model further comprises: the pixel intensity adjusting module IAM is used for adjusting the pixel intensity of the enhanced image output by the DAG; the IAM is located after the last convolutional layer of the DAG;
after the target image and the illumination guide map are input into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, the method comprises the following steps:
and inputting the enhanced image into the IAM to obtain the enhanced image which is output by the IAM and is subjected to pixel intensity adjustment.
5. The dim-light image enhancement method according to claim 3, wherein said inputting said target image and said illumination guide map into said SAG resulting in a structure-aware feature of said target image generated by said SAG comprises:
normalizing the RGB three channels of the target image to be between [ -1,1 ];
connecting the normalized RGB three channels of the target image and the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the SAG to obtain the structural perception characteristics of the target image generated by the SAG;
wherein in the SAG jump connection, the illumination guide diagram is used for adaptively learning the structural consistency of the feature diagram; the SAG employs a supervised loss function comprising at least one of: a first global perceptual loss function, a first local perceptual loss function, a first global countermeasure loss function, a first local countermeasure loss function; and the first local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the first image blocks after the target image and the output image of the SAG are divided into the first image blocks with fixed sizes.
6. The dim-light image enhancement method according to claim 3, wherein the inputting the target image and the illumination guide map into the DAG to obtain detail perception features of the target image generated by the DAG comprises:
normalizing the RGB three channels of the target image to be between [ -1,1 ];
connecting the normalized RGB three channels of the target image with the single channel of the illumination guide graph in channel dimensions, and inputting the obtained four channels into the DAG to obtain the detail perception characteristics of the target image generated by the DAG;
wherein the DAG does not change image resolution; the supervised loss function employed by the DAG includes at least one of: a second global perceptual loss function, a second local perceptual loss function, a second global countermeasure loss function, a second local countermeasure loss function; and the second local perceptual loss function is used for representing a result obtained by summing the global perceptual loss functions of the second image blocks after the target image and the output image of the DAG are divided into the second image blocks with fixed sizes.
7. The dim-light image enhancement method according to claim 3, wherein the inputting the structure-aware feature and the detail-aware feature to the FAM resulting in a fused feature of the structure-aware feature and the detail-aware feature comprises:
and under the condition that the structure perception feature comprises at least two layers of reciprocal convolution results of the SAG, and the detail perception feature comprises at least two layers of positive number convolution results of the DAG, connecting the at least two layers of reciprocal convolution results of the SAG and the at least two layers of positive number convolution results of the DAG in a channel dimension based on the FAM, guiding the connected images by using the illumination guide graph with a single channel and the same resolution, and sequentially inputting the guided images into at least one layer of convolution kernel and a hyperbolic tangent activation function for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature.
8. A scotopic image enhancing device, comprising:
the determination module is used for determining an illumination guide map of a target image based on three channels of red, green and blue (RGB) of the target image under the condition that the illumination intensity of the target image is lower than a target threshold value; wherein the illumination guide map is used for representing the degree of enhancement required at different positions in the target image;
the enhancement module is used for inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model;
the convolutional neural network image enhancement model is obtained by taking unpaired dim light images and normal light images as training samples and training based on detail perception characteristics and structure perception characteristics of the training samples; the convolutional neural network image enhancement model is used for carrying out image enhancement on the target image; the illumination intensity of the enhanced image is higher than or equal to the target threshold.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the method of enhancing a scotopic image according to any one of claims 1 to 7 are implemented when the program is executed by the processor.
10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the steps of the dim-light image enhancement method according to any one of claims 1 to 7.
CN202111138589.8A 2021-09-27 2021-09-27 Dim light image enhancement method and device, electronic equipment and storage medium Pending CN114049264A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111138589.8A CN114049264A (en) 2021-09-27 2021-09-27 Dim light image enhancement method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111138589.8A CN114049264A (en) 2021-09-27 2021-09-27 Dim light image enhancement method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114049264A true CN114049264A (en) 2022-02-15

Family

ID=80204903

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111138589.8A Pending CN114049264A (en) 2021-09-27 2021-09-27 Dim light image enhancement method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114049264A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114331918A (en) * 2022-03-08 2022-04-12 荣耀终端有限公司 Training method of image enhancement model, image enhancement method and electronic equipment
CN115115540A (en) * 2022-06-08 2022-09-27 大连海事大学 Unsupervised low-light image enhancement method and unsupervised low-light image enhancement device based on illumination information guidance

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114331918A (en) * 2022-03-08 2022-04-12 荣耀终端有限公司 Training method of image enhancement model, image enhancement method and electronic equipment
CN115115540A (en) * 2022-06-08 2022-09-27 大连海事大学 Unsupervised low-light image enhancement method and unsupervised low-light image enhancement device based on illumination information guidance

Similar Documents

Publication Publication Date Title
CN112288658B (en) Underwater image enhancement method based on multi-residual joint learning
CN108364270B (en) Color reduction method and device for color cast image
CN108876742B (en) Image color enhancement method and device
CN110675328A (en) Low-illumination image enhancement method and device based on condition generation countermeasure network
CN114049264A (en) Dim light image enhancement method and device, electronic equipment and storage medium
US11398016B2 (en) Method, system, and computer-readable medium for improving quality of low-light images
JP2021531571A (en) Certificate image extraction method and terminal equipment
CN110674759A (en) Monocular face in-vivo detection method, device and equipment based on depth map
CN111415304A (en) Underwater vision enhancement method and device based on cascade deep network
CN107256539B (en) Image sharpening method based on local contrast
CN112465709A (en) Image enhancement method, device, storage medium and equipment
CN113052768A (en) Method for processing image, terminal and computer readable storage medium
CN114187515A (en) Image segmentation method and image segmentation device
CN110717864A (en) Image enhancement method and device, terminal equipment and computer readable medium
CN117218039A (en) Image processing method, device, computer equipment and storage medium
CN114648467B (en) Image defogging method and device, terminal equipment and computer readable storage medium
CN115457015A (en) Image no-reference quality evaluation method and device based on visual interactive perception double-flow network
CN114862707A (en) Multi-scale feature recovery image enhancement method and device and storage medium
CN114266803A (en) Image processing method, image processing device, electronic equipment and storage medium
CN112699760A (en) Face target area detection method, device and equipment
CN115456917B (en) Image enhancement method, device, equipment and medium beneficial to accurate target detection
KR102199572B1 (en) Reverse object detection method based on deep learning
US20240177316A1 (en) Method for segmenting roads in images, electronic device, and storage medium
CN117994161B (en) RAW format weak light image enhancement method and device
CN115484447B (en) Projection method, projection system and projector based on high color gamut adjustment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination