CN114862698A - Method and device for correcting real overexposure image based on channel guidance - Google Patents
Method and device for correcting real overexposure image based on channel guidance Download PDFInfo
- Publication number
- CN114862698A CN114862698A CN202210381937.2A CN202210381937A CN114862698A CN 114862698 A CN114862698 A CN 114862698A CN 202210381937 A CN202210381937 A CN 202210381937A CN 114862698 A CN114862698 A CN 114862698A
- Authority
- CN
- China
- Prior art keywords
- image
- overexposure
- real
- correction
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012937 correction Methods 0.000 claims abstract description 60
- 238000003702 image correction Methods 0.000 claims abstract description 19
- 230000008447 perception Effects 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 6
- 238000013528 artificial neural network Methods 0.000 claims abstract description 5
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 238000013507 mapping Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 15
- 230000000007 visual effect Effects 0.000 claims description 12
- 238000013527 convolutional neural network Methods 0.000 claims description 11
- 238000010606 normalization Methods 0.000 claims description 11
- 238000005286 illumination Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 7
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 6
- 230000008901 benefit Effects 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 230000007613 environmental effect Effects 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims description 2
- 230000004927 fusion Effects 0.000 claims description 2
- 238000009499 grossing Methods 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 claims description 2
- 230000006978 adaptation Effects 0.000 claims 1
- 230000002035 prolonged effect Effects 0.000 claims 1
- 230000014759 maintenance of location Effects 0.000 abstract description 3
- 238000004422 calculation algorithm Methods 0.000 description 8
- 229920006395 saturated elastomer Polymers 0.000 description 5
- 230000035945 sensitivity Effects 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 230000011514 reflex Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention relates to a method and a device for correcting a real overexposure image based on channel guidance, and belongs to the technical field of image processing. According to the method, through analyzing the data characteristics of the overexposed image based on the RAW format, the brightness and the perception intensity of each color channel of the overexposed RAW image R-G-G-B (red-green-blue) obtained by the visible light sensor are different, the convolution calculation of channel guide correction is designed according to the data characteristics of inconsistent texture characteristics and information content retention, a channel guide convolution neural network is further constructed, the overexposed image correction neural network is trained by using the acquired data set, the high-precision overexposed image correction of channel guide is realized, and the correction quality of the real overexposed image is improved. The invention can improve the quality of the real image overexposure correction and ensure the fidelity of the real overexposure corrected image.
Description
Technical Field
The invention relates to a method and a device for correcting a real overexposure image based on channel guidance, and belongs to the technical field of image processing.
Background
Overexposure correction of images is a technical problem that has a significant impact on image processing and social media related tasks and industries.
In the imaging process of the camera, the unreasonable exposure level can directly influence the imaging global brightness and texture information, so that the image quality is obviously reduced, the visual effect and the information content of the image are obviously influenced, and the effect of a downstream computer visual algorithm (such as target detection, segmentation and the like) on the image is greatly reduced.
Although the overexposure problem can be alleviated to some extent by adjusting the digital camera parameters such as aperture, exposure time or sensitivity, due to the limited physical dynamic range or wrong parameter setting of the digital camera and various image sensors, the sensor often generates saturated pixels in a relatively bright area in a scene with excessively high illuminance or severe brightness change, which results in information loss and generation of an overexposed image. In addition, all the visual images in formats such as 8-bit quantization JPEG captured by the existing Image capturing devices are obtained from RAW domain Image processing through a built-in ISP (Image Signal Processor), and only semantic information of an overexposed Image is lost in the process, so that pixels in an output Image are usually heavily saturated, and sometimes even scene information is permanently lost.
The overexposure correction technology is an image processing technology which converts an overexposure image into a normal illumination image by using a post-processing algorithm and recovers the lost texture detail information of the overexposure area so as to adjust saturated pixels, supplement missing content and correct global brightness. The technology can effectively improve the image visual quality, enhance the image details and improve the performance of a downstream visual task algorithm.
However, the prior art does not have much targeted research on the overexposure correction technique, and there are few ways to address this problem. The existing exposure-correction (exposure-correction) method generally focuses on restoration correction (i.e. dark light enhancement) of an under-exposed image, and an overexposed image and an under-exposed image before a restoration correction task thereof have essential differences due to texture information retention, pixel value characteristics, noise problems and the like, so that when a related algorithm for correcting the under-exposed image is directly used for the overexposure correction task, a good effect cannot be achieved.
Some conventional general image enhancement techniques mainly utilize a statistical principle and an artificial prior model to process an overexposed image, correct global brightness by means of an optimization strategy and rules, and generate content completion for a saturated region according to pixels of a region where texture information of the image is not lost. However, these methods depend to a large extent on the accuracy of their hand-made a priori assumptions, which are not sufficient to represent the various luminance saturation and texture loss cases of the real world. Some existing exposure correction algorithms usually focus on adjustment of image exposure and contrast, rather than repair of lost information, and therefore, the restored images cannot obtain the expected visual quality or texture information details.
In recent years, end-to-end exposure correction networks based on deep learning have emerged as some of the research results. Features are automatically extracted from the overexposed and underexposed data using a convolutional neural network. However, deep learning approaches rely heavily on training data sets. However, the conventional deep learning method generally relies on a training data set only containing sRGB images, which is synthesized by only using gamma transformation or linear change to increase the brightness of natural illumination images, or further using nonlinear function mapping simulation, etc. On one hand, the synthesized data and the real original data have data distribution difference, and due to the lack of the real overexposure correction data with reference, the application of the trained exposure correction method in the real world is limited; on the other hand, limited by the data, these algorithms focus on image exposure and color adjustment, rather than on the restoration of lost information, and especially, it is difficult to recover real-world images or regions with severe overexposure (almost completely saturated pixel values), and the processed images often still cannot meet aesthetic quality requirements or requirements of downstream high-level vision algorithms.
Disclosure of Invention
The invention aims to creatively provide a method and a device for correcting a real overexposed image based on channel guidance, aiming at the technical problems that no real paired overexposed image correction data set exists in the prior art, and the conventional learning algorithm-based method is difficult to effectively recover texture information of a serious overexposed image or region.
The innovation points of the invention are as follows: analyzing the data characteristics of an overexposed image based on a RAW (original) format, designing the convolution calculation of channel guide correction according to the data characteristics of different texture characteristics and inconsistent information content retention according to the different brightness and perception intensity of each color channel of the overexposed RAW image obtained by a visible light sensor, and further constructing a channel-guided convolution neural network. Meanwhile, a real paired overexposure correction data acquisition device is designed and used for acquiring a data set, the acquired data set is used for training an overexposure image correction neural network, channel-guided high-precision overexposure image correction is realized, and the correction quality of a real overexposure image is improved.
A real overexposure image correction method based on channel guidance comprises the following steps:
step 101: a real pair of overexposure correction data sets is collected (a digital single lens reflex camera can be used to construct the real data collection device, but is not limited in this way).
The data set contains pairs of overexposed and normally illuminated reference images, each image having both RAW and other data in a desired format (e.g., sRGB format, etc.).
Step 102: and analyzing the data characteristics of the RAW format real overexposure image, and designing a channel to guide convolution branches.
In general, human eyes have different visual sensitivities for different colors, and a visible light camera sensor is designed based on the visual sensitivities.
Therefore, according to the data characteristics that the brightness and the perceived intensity of different color channels in various color modes (such as an RGB mode, a CMYK mode, a Lab mode and the like) of the RAW image are different, the content of the retained texture information is inconsistent, and the channels with higher brightness and perceived intensity are easier to overexpose under the overexposure condition, the channels are designed to guide the convolution branches.
Step 103: and constructing a channel guide overexposure correction convolutional neural network by using the channel guide convolution branch and utilizing the characteristics of the RAW image data.
Preferably, according to the data characteristic that the channel with relatively high brightness and perception intensity of the RAW image analyzed in step 102 is easier to overexpose under the overexposure condition, the more abundant texture information retained by the channel with relatively low brightness and perception intensity in the input image can be utilized to effectively recover the fine texture details of the severely overexposed image or region.
Step 104: and establishing a training target function for overexposure enhancement of the real image, and training parameters of the convolutional neural network to obtain a mapping relation between the overexposure RAW image and the reference image.
Step 105: inputting an overexposed RAW image to be tested, and mapping relation between the overexposed RAW image obtained in step 104 and a reference image. And mapping the overexposed RAW image into an image with a format required by normal illumination through the mapping relation between the exposed RAW image and the reference image. Therefore, high-efficiency and high-precision image overexposure correction is realized, and the imaging quality in an overexposure scene is obviously improved.
In order to achieve the purpose of the invention, the invention further provides a real overexposure image correction device based on channel guidance, which comprises a real paired overexposure correction data acquisition submodule, a color channel guidance learning submodule and a real overexposure image correction network submodule.
And the real paired overexposure correction data acquisition submodule is used for acquiring a real paired overexposure correction data set.
And the color channel guide learning submodule is used for designing a channel guide convolution branch, so that the color channel guide convolution branch can be used for guiding a backbone network to recover lost pixel information in an overexposure area by utilizing the information advantages of a color channel with higher brightness and perception intensity based on the data characteristics of a real overexposure image in a RAW format.
And the real overexposure image correction network submodule is used for training acquired real paired overexposure correction data to obtain a network for correcting and enhancing the overexposure real image.
The connection relationship among the modules is as follows:
and the output end of the real paired overexposure correction data acquisition submodule is connected with the input end of the color channel guidance learning submodule.
And the output end of the color channel guide learning submodule is connected with the input end of the real overexposure image correction network submodule.
Advantageous effects
1. According to the invention, a real data acquisition device is built, so that real paired normal illumination reference image data sets are acquired, and simultaneously each image is ensured to have data of RAW and other required formats (such as sRGB format). The data acquisition device can efficiently acquire real paired overexposure correction data sets, and the acquired data sets can effectively improve the precision of the overexposure correction method on real images.
2. According to the method, channel-guided convolution calculation branches are designed according to the data characteristics that the brightness and the perception intensity of each color channel are different and the texture features and the information content are not consistent under different color modes of the RAW domain overexposure image, and a channel-guided overexposure correction network is constructed. The channel-guided overexposure correction network can fully utilize texture information reserved in a color channel with low overexposure RAW image brightness and perception intensity, gradually guide the channel with serious information loss to carry out information recovery, share similar edge information, and effectively improve the robustness recovery of the overexposure correction network on overexposure images with different information contents.
3. According to the invention, the convolutional neural network is utilized to learn the mapping relation between the overexposed RAW image and the normal illumination reference image, and the generalization and robustness of the convolutional neural network can be improved by combining the beneficial effects 1 and 2, the overexposure correction quality of the real image can be improved, and the fidelity of the real overexposure corrected image is ensured.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of a channel-guided overexposure correction network using RAW data according to the present invention;
fig. 3 is a schematic structural diagram of an apparatus according to an embodiment of the present invention.
Detailed Description
To better illustrate the objects and advantages of the present invention, the following further description is provided in conjunction with the accompanying drawings.
Examples
The method and apparatus of the present invention will be described below with reference to RGB images as an example.
As shown in fig. 1, a method for correcting a real overexposure image based on channel guidance includes the following steps:
step 101: and acquiring real paired overexposure correction data sets (which can be realized by constructing a real data acquisition device by using a digital single lens reflex). The data set contains pairs of overexposed and normally illuminated reference images, each image having data in both RAW and sRGB formats.
To acquire high quality RAW and sRGB images simultaneously, a digital single lens reflex camera (or other satisfactory image acquisition device) capable of storing RAW format data may be used.
Specifically, to capture each pair of overexposed RAW/sRGB images, a pre-capture scene is first selected, ensuring no environmental or human interference, and richness of the scene and content of the captured image data. Secondly, shooting hardware equipment is arranged, the camera is fixed on a tripod to form a real data acquisition device, and the shutter is controlled by remote software to shoot so as to avoid dislocation caused by movement of the camera. And then, setting camera shooting parameters, and in each scene, adjusting parameters such as an aperture, a focal length and exposure time to improve the visual quality of the reference image to the maximum extent (the detail information of the image can be clearly distinguished by naked eyes). During specific acquisition, a normal light clean reference image of a target scene is acquired at first, then the exposure time parameter of the image acquisition equipment is adjusted through remote control software, and the exposure time of the image acquisition equipment is increased by 3-10 times, so that image shooting under actual exposure conditions is realized. Preferably, to better achieve the dark light image acquisition, the ISO parameter may be fixed to 100 to reduce noise interference, and the exposure time multiple is selected from 3, 5, 8, 10, etc. to simulate the overexposed images with different overexposure degrees.
Through the process, the real paired overexposure correction data sets can be acquired.
Step 102: and analyzing the data characteristics of the RAW format real overexposed image, and designing a channel to guide convolution branches.
Because the human eyes have different visual sensitivities for different colors, the visible light camera sensor is also designed based on the visual sensitivities. According to the data characteristics that after the RAW image is decomposed into four-channel R-G-G-B (red-green-blue) images in an RGB color mode, the brightness and the perception intensity of different color channels are different, the content of the reserved texture information is inconsistent, the brightness and the perception intensity of a green channel are higher under an overexposure condition, and overexposure is easier, a channel is designed to guide convolution branches.
Based on the observation that the green channel of the overexposed RAW image obtained in step 101 loses more texture information in the overexposed bright area of the image, the information-rich channel is used to guide the backbone network to recover the lost pixel information in the overexposed area.
Furthermore, since the green channel shares similar edges with the remaining channels, initial estimation of the remaining channels can benefit the green channel.
Specifically, as shown in fig. 2, first, pixels belonging to the red and blue channels at corresponding positions are extracted in each 2 × 2 block of the RAW image in bayer pattern, the red and blue channels are obtained and fed together to the channel-directed convolution branches to generate initial estimates of the respective red and blue channels in the output sRGB image.
The channel-directed convolution branch consists of one directed enhancement module and four downsampling modules, where the directed enhancement module contains three 3 × 3 depth separable convolution layers, an example normalization operation, and a LeakyReLU activation function, allowing full exploration and extraction of local information. Each downsampling module comprises an average pooling layer capable of reducing the size of the feature map, a self-attention mechanism structure and two depth separable convolutions, so that the branch network is ensured to gradually enhance the input red and blue channels each time of downsampling, and can be spliced with the main network to help the main network to recover the overexposed RAW image in a multi-scale mode.
At this point, the design of channel guided convolution branches is completed, and the guided enhancement of texture detail recovery is completed.
Step 103: and constructing a channel guide overexposure correction convolutional neural network by using the channel guide convolution branch and utilizing the characteristics of the RAW image data.
According to the data characteristics that the brightness and the perception intensity of the green channel of the RAW image analyzed in the step 102 are higher and overexposure is easier under the overexposure condition, the fine texture details of the severely overexposed image or region can be effectively recovered by utilizing richer texture information reserved in the red and blue channels of the input image.
In particular, as shown in fig. 2, the channel leads the main branch of the exposure correction network, which is a U-net structure consisting of 4 encoder stages and 4 corresponding decoder stages. In which the main branches are first convolved by a 3 x 3 convolution to extract the original features from the four-channel RAW image. In the encoder part, a half-instance normalized block is used to replace the traditional residual block to expand the receptive field and improve the robustness of the features on each down-sampling scale. During the down-sampling operation, the number of channels in the extracted feature map is doubled. In the decoder part, local residual learning is introduced, and a residual block is used for extracting high-level features. For the jump connection structure, the cascade hollow residual block is used for replacing the conventional jump connection structure to extract high-level features, and the high-level features are fused with the features of each level of the encoder part, so that the loss of detail information caused by downsampling is better compensated. At the same time, the channel-guided convolution branches constructed in step 102 are introduced into the structure.
Preferably, the half-instance normalization block is composed of a standard 3 × 3 convolution, a LeakyReLU activation function, and an instance normalization structure, wherein the instance normalization structure is used only on half of the channels, that is, only half of the extracted feature information is subjected to the instance normalization process, and the other half of the channels retain context information. By adopting the design, the method is more friendly to the characteristics of the shallow layer of the network, can expand the receptive field of each scale in the encoder part, and also improves the robustness of the characteristics.
The cascade hole residual block comprises three residual connections consisting of hole convolution and LeakyReLU activation function, and a layer of 1 multiplied by 1 convolution layer. The structure can well utilize the characteristics of each stage of the encoder in a fine-grained mode and fully explore local texture information. Meanwhile, the introduction of the hole convolution can expand the receptive field of the hollow image, and the characteristic diagram for extracting multi-scale context information is provided.
Therefore, the construction of the channel guide overexposure correction network by using the RAW data is completed, and the fine texture details of the serious overexposure image or area can be effectively recovered according to richer texture information reserved in the red and blue channels of the input image.
Step 104: and establishing a training objective function for real image overexposure enhancement, and training a network parameter theta to obtain a mapping relation F between the input overexposed RAW image and the reference sRGB image.
In particular, an overall training objective function of an overexposure-corrected convolutional neural network
Comprises the following steps:
where θ denotes a network parameter, I OE Representing an input overexposed RAW image, I GT Representing the reference sRGB image, 0.5 and 0.2 are the loss function weights respectively,
representing the pixel-level root mean square error loss,
the loss of perception is indicated by the presence of,
which is indicative of the loss of the gradient,
indicating a color loss.
In formula 1
For optimizing adaptationThe high frequency details to generate a smoothed reconstruction result are represented as:
wherein W and H represent the width and height of the input image, respectively; f denotes a network mapping relationship, F (I) OE ) The reconstructed image after overexposure correction is obtained; x and y are the output pixel coordinates in the reference image space.
In formula 1
The information of image loss is recovered from the feature space by using the VGG network, and is expressed as:
wherein, W i，j And H i，j The dimension, φ, of each feature graph in a VGG network is described i，j For representing the feature map obtained by the jth convolution (after activation) before the ith max pooling layer in the VGG network.
In formula 1
The sobel operator is utilized to calculate the image gradient, further, the loss of the image texture information is evaluated, and the whole image is subjected to small-scale smoothing, so that the texture detail characteristics of the output result are effectively enhanced, and the expression is as follows:
wherein Grad is the reconstructed image F (I) OE ) And a reference picture I GT The fusion gradient of (a).
Wherein, based on F (I) OE ) Is expressed as:
Grad(F(I OE ，θ)) x，y ＝|f x ·F(I OE ，θ) | +|f y ·F(I OE ，θ)| (5)
in the formula 5, the first step is,
in formula 1
Evaluating and optimizing the color difference between the overexposed image and the reference image using gaussian blur, expressed as:
in formula 6, F (I) OE ，θ) b 、I GT b Are respectively F (I) OE ) And I GT The blurred image of (2). b represents a blurred image obtained by gaussian blur calculation of the original image.
Wherein, based on F (I) OE ，θ) b Expressed as:
F(I OE ，θ) b (i，j)＝∑ k，l F (I OE ，θ)(i+m，j+n)·G(m，n) (7)
in equation 7, the two-dimensional gaussian blur operation G (m, n) is defined as:
wherein K represents a blur level parameter, μ x 、μ y Respectively representing the mean parameter, σ x 、σ y Respectively, the variance parameters. Preferably, the parameter K is set to 0.053, and the parameter μ x ＝μ y 0, parameter σ x ＝σ y ＝3。
And obtaining the optimized network parameter theta by optimizing the trained target function formula 1.
Thus, training of the overexposure correction network is completed, and a mapping relation F between the overexposure RAW image and the reference sRGB image of different exposure degrees and scenes is obtained.
Step 105: the overexposed RAW image to be tested is input, and the mapping relationship F between the overexposed RAW image obtained in step 104 and the reference sRGB image is input. And mapping the overexposed RAW image into a high-quality normal illumination sRGB image through a mapping relation F between the exposed RAW image and the reference sRGB image, thereby realizing high-efficiency and high-precision image overexposure correction and obviously improving the imaging quality in an overexposed scene.
Wherein the overexposed RAW image is
Wherein Y is,
Overexposed RAW images and normal-illumination sRGB images, respectively.
Preferably, the GPU is used to complete the training process of the step 104 network and the image overexposure correction process of the step 105, and the cuDNN library is used to accelerate the operation speed of the convolutional neural network.
Meanwhile, in order to achieve the purpose of the present invention, the present invention further provides a real overexposure image correction device based on channel guidance, as shown in fig. 3, which includes a real paired overexposure correction data acquisition sub-module 10, a color channel guidance learning sub-module 20, and a real overexposure image correction network sub-module 30.
Wherein, the real paired overexposure correction data acquisition submodule 10 is used for acquiring real paired overexposure correction data sets.
The color channel guiding learning sub-module 20 is configured to design a channel guiding convolution branch, so that the channel guiding convolution branch can guide a backbone network to recover lost pixel information in an overexposed area based on data characteristics of a RAW format real overexposed image by using information advantages of red and blue channels.
The real overexposure image correction network submodule 30 trains using the acquired real paired overexposure correction data, and the training obtains a network for correction and enhancement of the overexposure real image.
The connection relationship among the modules is as follows:
the output of the real paired overexposure correction data acquisition submodule 10 is connected to the input of the color channel guided learning submodule 20.
The output of the color channel guidance learning sub-module 20 is connected to the input of the true overexposure image correction network sub-module 30.
The above detailed description is intended to illustrate the objects, aspects and advantages of the present invention, and it should be understood that the above detailed description is only exemplary of the present invention and is not intended to limit the scope of the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (9)
1. A real overexposure image correction method based on channel guidance is characterized by comprising the following steps:
step 101: acquiring a true pair of overexposure correction datasets containing pairs of overexposed and normal illumination reference images, each image having both RAW and other data of a desired format;
step 102: analyzing the data characteristics of the RAW format real overexposure image, and designing a channel to guide convolution branches;
step 103: using a channel guide convolution branch to construct a channel guide overexposure correction convolution neural network by using the characteristics of the RAW image data;
step 104: establishing a training target function for overexposure enhancement of a real image, and training parameters of a convolutional neural network to obtain a mapping relation between an overexposure RAW image and a reference image;
step 105: inputting an overexposed RAW image to be tested and the mapping relation between the overexposed RAW image obtained in the step 104 and a reference image; and mapping the overexposed RAW image into an image with a format required by normal illumination through the mapping relation between the exposed RAW image and the reference image.
2. The method as claimed in claim 1, wherein in step 101, a pre-shot scene is first selected to ensure no interference from environmental or human factors, and richness of the scene and content of the acquired image data;
secondly, setting shooting hardware equipment, fixing the camera on a tripod to form a real data acquisition device, and controlling a shutter by software to shoot so as to avoid dislocation caused by the motion of the camera;
then, camera shooting parameters are set, and in each scene, the visual quality of the reference image is improved to the maximum extent by adjusting parameters such as an aperture, a focal length and exposure time;
during specific acquisition, a normal light clean reference image of a target scene is acquired first each time, and then exposure time parameters of the image acquisition equipment are adjusted through software, so that the exposure time of the image acquisition equipment is prolonged.
3. The method of claim 1, wherein in step 102, the channel-guided convolution branch is composed of a guided enhancement module and four downsampling modules, wherein the guided enhancement module comprises three 3 × 3 depth separable convolution layers, an example normalization operation and a LeakyReLU activation function, allowing full exploration and extraction of local information; each downsampling module comprises an average pooling layer capable of reducing the size of the feature map, a self-attention mechanism structure and two depth separable convolutions, so that the branch network is ensured to gradually enhance the input red and blue channels each time of downsampling, and can be spliced with the main network to help the main network to recover the overexposed RAW image in a multi-scale mode.
4. The method as claimed in claim 1, wherein in step 103, the main branch of the channel-guided overexposure correction network is a U-net structure consisting of 4 encoder stages and 4 corresponding decoder stages; firstly, extracting original features from a four-channel RAW image through a 3 multiplied by 3 convolution by a main branch; in the encoder part, a half-instance normalization block is used for expanding a receptive field and improving the robustness of features on each down-sampling scale; doubling the number of channels in the extracted feature map during the down-sampling operation; in the decoder part, local residual error learning is introduced, and a residual error block is used for extracting high-level features; for the jump connection structure, a cascade hollow residual block is used for replacing a conventional jump connection structure so as to extract high-level features, and the high-level features are fused with the features of each level of the encoder part; at the same time, the channel-guided convolution branches constructed in step 102 are introduced into the structure.
5. The method according to claim 4, wherein the method comprises the following steps:
the semi-instance normalization block consists of a standard 3 × 3 convolution, a LeakyReLU activation function and an instance normalization structure, wherein the instance normalization structure is only used on half of channels, namely only half of the extracted feature information is subjected to instance normalization processing, and the other half of the channels retain context information;
the cascade hole residual block comprises three residual connections consisting of hole convolution and LeakyReLU activation function, and a layer of 1 multiplied by 1 convolution layer.
6. The method as claimed in claim 1, wherein the overexposure correction is performed on the total training objective function of the convolutional neural network in step 104
Comprises the following steps:
where θ denotes a network parameter, I OE Representing an input overexposed RAW image, I GT Representing a reference picture, 0.5 and 0.2 being losses, respectivelyThe weight of the loss-of-function,
representing the pixel-level root mean square error loss,
the loss of perception is indicated by the presence of,
which is indicative of the loss of the gradient,
indicating a color loss;
in formula 1
For optimizing the adaptation of high frequency details to generate a smoothed reconstruction result, expressed as:
wherein W and H represent the width and height of the input image, respectively; f denotes a network mapping relationship, F (I) OE ) The reconstructed image after overexposure correction is obtained; x and y are the output pixel coordinates in reference image space;
in formula 1
The information of image loss is recovered from the feature space by using the VGG network, and is expressed as:
wherein, W i,j And H i,j The dimension, φ, of each feature graph in a VGG network is described i,j For representing the ith in a VGG networkObtaining a characteristic diagram by a jth convolution before the large pooling layer;
in formula 1
Calculating the image gradient by using a sobel operator to further evaluate the loss of image texture information and performing small-scale smoothing on the whole image, wherein the expression is as follows:
wherein Grad is the reconstructed image F (I) OE ) And a reference picture I GT A fusion gradient of (a);
wherein, based on F (I) OE ) Is expressed as:
Grad(F(I OE ,θ)) x,y ＝|f x ·F(I OE ,θ)|+|f y ·F(I OE ,θ)| (5)
in the formula 5, the first step is,
in formula 1
Evaluating and optimizing the color difference between the overexposed image and the reference image using gaussian blur, expressed as:
in formula 6, F (I) OE ,θ) b 、I GT b Are respectively F (I) OE ) And I GT The blurred image of (1); b represents a fuzzy image obtained by Gaussian fuzzy calculation of the original image;
wherein, based on F (I) OE ,θ) b Expressed as:
F(I OE ,θ) b (i,j)＝∑ k,l F(I OE ,θ)(i+m,j+n)·G(m,n) (7)
in equation 7, the two-dimensional gaussian blur operation G (m, n) is defined as:
wherein K represents a blur level parameter, μ x 、μ y Respectively representing the mean parameter, σ x 、σ y Respectively representing variance parameters;
and obtaining the optimized network parameter theta by optimizing the trained target function formula 1.
7. The method as claimed in claim 6, wherein in step 104, the parameter K is set to 0.053 and the parameter μ is set to x ＝μ y 0, parameter σ x ＝σ y ＝3。
8. The method as claimed in claim 1, wherein the training process of step 104 network and the image overexposure correction process of step 105 are completed by using GPU, and the running speed of convolutional neural network is increased by using cuDNN library.
9. A real overexposure image correction device based on channel guidance comprises a real paired overexposure correction data acquisition submodule, a color channel guidance learning submodule and a real overexposure image correction network submodule;
the real paired overexposure correction data acquisition submodule is used for acquiring a real paired overexposure correction data set;
the color channel guiding learning submodule is used for designing a channel guiding convolution branch so that the channel guiding convolution branch can be used for guiding a backbone network to recover lost pixel information in an overexposure area based on the data characteristic of a real overexposure image in a RAW format by utilizing the information advantage of a color channel with higher brightness and perception intensity;
the real overexposure image correction network submodule is used for training acquired real paired overexposure correction data to obtain a network for correction and enhancement of the overexposure real image;
the connection relationship among the modules is as follows:
the output end of the real paired overexposure correction data acquisition submodule is connected with the input end of the color channel guidance learning submodule;
the output end of the color channel guide learning sub-module is connected with the input end of the real overexposure image correction network sub-module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210381937.2A CN114862698B (en) | 2022-04-12 | 2022-04-12 | Channel-guided real overexposure image correction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210381937.2A CN114862698B (en) | 2022-04-12 | 2022-04-12 | Channel-guided real overexposure image correction method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114862698A true CN114862698A (en) | 2022-08-05 |
| CN114862698B CN114862698B (en) | 2024-06-07 |
Family
ID=82632523
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210381937.2A Active CN114862698B (en) | 2022-04-12 | 2022-04-12 | Channel-guided real overexposure image correction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114862698B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116193264A (en) * | 2023-04-21 | 2023-05-30 | 中国传媒大学 | Camera adjusting method and system based on exposure parameters |
| CN117726541A (en) * | 2024-02-08 | 2024-03-19 | 北京理工大学 | Dim light video enhancement method and device based on binarization neural network |
| CN117793538A (en) * | 2024-02-23 | 2024-03-29 | 北京理工大学 | Automatic image exposure correction and enhancement method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108805836A (en) * | 2018-05-31 | 2018-11-13 | 大连理工大学 | Method for correcting image based on the reciprocating HDR transformation of depth |
| US20200234414A1 (en) * | 2019-01-23 | 2020-07-23 | Inception Institute of Artificial Intelligence, Ltd. | Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures |
| CN113191995A (en) * | 2021-04-30 | 2021-07-30 | 东北大学 | Video image automatic exposure correction method based on deep learning |
| CN113643214A (en) * | 2021-10-12 | 2021-11-12 | 江苏维沛通信科技发展有限公司 | Image exposure correction method and system based on artificial intelligence |
-
2022
- 2022-04-12 CN CN202210381937.2A patent/CN114862698B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108805836A (en) * | 2018-05-31 | 2018-11-13 | 大连理工大学 | Method for correcting image based on the reciprocating HDR transformation of depth |
| US20200234414A1 (en) * | 2019-01-23 | 2020-07-23 | Inception Institute of Artificial Intelligence, Ltd. | Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures |
| CN113191995A (en) * | 2021-04-30 | 2021-07-30 | 东北大学 | Video image automatic exposure correction method based on deep learning |
| CN113643214A (en) * | 2021-10-12 | 2021-11-12 | 江苏维沛通信科技发展有限公司 | Image exposure correction method and system based on artificial intelligence |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116193264A (en) * | 2023-04-21 | 2023-05-30 | 中国传媒大学 | Camera adjusting method and system based on exposure parameters |
| CN116193264B (en) * | 2023-04-21 | 2023-06-23 | 中国传媒大学 | Camera adjusting method and system based on exposure parameters |
| CN117726541A (en) * | 2024-02-08 | 2024-03-19 | 北京理工大学 | Dim light video enhancement method and device based on binarization neural network |
| CN117793538A (en) * | 2024-02-23 | 2024-03-29 | 北京理工大学 | Automatic image exposure correction and enhancement method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114862698B (en) | 2024-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11055827B2 (en) | Image processing apparatus and method | |
| US10666873B2 (en) | Exposure-related intensity transformation | |
| US11037278B2 (en) | Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures | |
| EP2987135B1 (en) | Reference image selection for motion ghost filtering | |
| CN114862698B (en) | Channel-guided real overexposure image correction method and device | |
| CN110930301B (en) | Image processing method, device, storage medium and electronic equipment | |
| CN115442515A (en) | Image processing method and apparatus | |
| WO2022000397A1 (en) | Low-illumination image enhancement method and apparatus, and computer device | |
| CN111064904A (en) | Dark light image enhancement method | |
| CN110619593A (en) | Double-exposure video imaging system based on dynamic scene | |
| CN113888437A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN111724317A (en) | Method for constructing Raw domain video denoising supervision data set | |
| CN111047543A (en) | Image enhancement method, device and storage medium | |
| Moriwaki et al. | Hybrid loss for learning single-image-based HDR reconstruction | |
| CN111986106A (en) | High dynamic image reconstruction method based on neural network | |
| CN113379609B (en) | Image processing method, storage medium and terminal equipment | |
| Lv et al. | Low-light image enhancement via deep Retinex decomposition and bilateral learning | |
| Zhang et al. | Deep motion blur removal using noisy/blurry image pairs | |
| CN116723413A (en) | RAW domain image denoising method and shooting device | |
| CN116563183A (en) | High dynamic range image reconstruction method and system based on single RAW image | |
| CN114299180A (en) | Image reconstruction method, device, equipment and storage medium | |
| CN114897916A (en) | Image processing method and device, nonvolatile readable storage medium and electronic equipment | |
| Khan et al. | A deep hybrid few shot divide and glow method for ill-light image enhancement | |
| CN112839167B (en) | Image processing method, device, electronic equipment and computer readable medium | |
| WO2023246392A1 (en) | Image acquisition method, apparatus and device, and non-transient computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |