CN113139909B - Image enhancement method based on deep learning - Google Patents
Image enhancement method based on deep learning Download PDFInfo
- Publication number
- CN113139909B CN113139909B CN202010056687.6A CN202010056687A CN113139909B CN 113139909 B CN113139909 B CN 113139909B CN 202010056687 A CN202010056687 A CN 202010056687A CN 113139909 B CN113139909 B CN 113139909B
- Authority
- CN
- China
- Prior art keywords
- image
- neural network
- loss function
- output
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000013135 deep learning Methods 0.000 title claims abstract description 15
- 238000013528 artificial neural network Methods 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims abstract description 23
- 238000005286 illumination Methods 0.000 claims abstract description 13
- 238000012360 testing method Methods 0.000 claims abstract description 8
- 238000003062 neural network model Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 11
- 238000009499 grossing Methods 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 230000003321 amplification Effects 0.000 claims 1
- 230000003190 augmentative effect Effects 0.000 claims 1
- 238000003199 nucleic acid amplification method Methods 0.000 claims 1
- 238000005457 optimization Methods 0.000 claims 1
- 239000003086 colorant Substances 0.000 abstract description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- RTAQQCXQSZGOHL-UHFFFAOYSA-N Titanium Chemical compound [Ti] RTAQQCXQSZGOHL-UHFFFAOYSA-N 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image enhancement method based on deep learning, which comprises the following steps: selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets T train And test set T test (ii) a A neural network S (-) adopting U-Net with global characteristics; the input of which is an original image S subjected to data enhancement Input And prior illumination estimation I, and output is enhanced R output And I output (ii) a Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-); training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model; compared with the prior method, the method has more natural colors, more attractiveness and better contrast, has smaller difference with the images repaired by professional diagraphers, has no artifact phenomenon, has little inference time, and can run on equipment such as a mobile phone and the like in ms level in real time.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an image enhancement method based on deep learning.
Background
The photography is an art of light and shadow, and a favorite photo is accurately grasped for brightness and color shading of the light and shadow. With the popularity of portable photography devices such as mini-tickets, mobile phones and the like, more and more people use photos to record their lives, start enjoying photography and enjoy sharing their photography works on social networks. In general, people often take photos of themselves due to unnatural brightness, insufficiently saturated colors, and the like. Therefore, people often spend a lot of time beautifying their images before sharing their photographic works. Despite the large number of interactive and semi-automatic image processing tools available on the market, these tools still have a large threshold for the user, and the beautified photos are also associated with the aesthetic level of the user.
Image enhancement has been a long-standing challenge in the field of computer vision and has been of interest to scholars. The traditional image enhancement algorithm mainly comprises histogram equalization, gray world assumption, wavelet transformation algorithm, automatic white balance and the like. These algorithms focus mainly on the contrast of the image and the correction of the color. However, these algorithms are only suitable for specific conditions, such as histogram equalization without selection of data processing, automatic white balance only suitable for uniform illumination, and the like. And the processed images still have a large gap from the expectations of people. In recent years, image processing algorithms based on deep learning have been greatly successful in the field of image enhancement, but these methods still have the disadvantages of low processing speed, undesirable effect and the like.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide an image enhancement method for deep learning, which meets the requirements of real-time performance, has more natural colors, more attractiveness and better contrast compared with the prior method, has smaller difference with the image repaired by a professional repairman, and has no artifact phenomenon.
1. An image enhancement method based on deep learning is characterized by comprising the following steps:
(A) selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets T train And test set T test ；
(B) A neural network S (-) adopting U-Net with global characteristics;
(C) the input of which is an original image S subjected to data enhancement Input And prior illumination estimation I, and output is enhanced R output And I output ；
(D) Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-);
(E) and training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model.
2. The deep learning-based image enhancement method according to claim 1, wherein in the step (B), the U-Net neural network S (-) with global features is composed of a connecting step and an expanding step. In the cont parsing step of each step, the convolution layer with 3 × 3 filter steps of 1, the batch normalization layer, and the maximum pooling layer with 2 × 2 filter steps of 2 are included. In the expanding step, each step firstly executes deconvolution, the result of the deconvolution is spliced with the feature map of the corresponding step, and then the convolution layer with 3 multiplied by 3 and step length of 1 and the batch processing normalization layer are processed by 2 filters.
3. The method for enhancing image based on deep learning as claimed in claim 2, wherein in the step (C), the data is enhanced by down-sampling the original image to a specified resolution size and performing random cropping, rotation and the like on the original image.
4. The method for enhancing image based on deep learning of claim 3, wherein in the step (E), the weight map is a formula of loss calculation, which is composed of six modules including a local loss function, a color loss function, and L 1 And an MS-SSIM loss function, a VGG loss function and an illumination smoothing loss function, which are expressed as:
(1) a local loss function. Randomly cropping image blocks from the input and label images and calculating L between the input and label image blocks 1 The loss, expressed as:
wherein, f (x) represents the image block generated by prediction, Y is the label image block, and n is the batch processing parameter of training.
(2) A color loss function. Expressed as:
wherein X b And Y b Representing X and Y gaussian blurred images, respectively.
(3)L 1 And MS-SSIM loss function. Expressed as:
L 1 ＝||X-Y|| 1 (4)
L MS-SSIM ＝1-MS-SSIM(X,Y) (5)
wherein X and Y represent the predicted image and the target image, respectively.
(4) VGG loss function. Expressed as:
(5) A structure-aware illumination smoothing loss function. Expressed as:
wherein
Is shown horizontally
To above and below
Gradient of (a) t The coefficients are the strength of the structural perceptual coefficients.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a sample diagram of a data set of an image enhancement method based on deep learning according to the present invention.
FIG. 2 is a network model diagram of the image enhancement method based on deep learning according to the present invention.
Detailed Description
In order to make the objects, embodiments and advantages of the present invention more apparent, further detailed description is given herein with reference to specific examples:
(A) selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets T train And test set T test ；
MIT-Adobe FiveK provides 5000 original images, as well as image data that was manually retouched by 5 professional reviewers (A, B, C, D, E). However, the following disadvantages of the vek data set still exist, and firstly, the data volume of the vek data set is still small, so that the training of the neural network cannot be met, overfitting is easily caused, and the requirements under a real scene cannot be met. Secondly, it has the problem of less data diversity, a significant portion of the original image data is low contrast and low brightness, and a small portion of the data is over-exposed and covers only limited lighting conditions.
We hired a photographer, using a different single lens reflex camera device, to take a total of 18000 photos, containing a very rich scene, lighting, theme, etc. We have a good professional reviewer who revises these captured original photographs to achieve satisfactory revised photograph data, as shown in fig. 1, for the purpose of presenting a partial data set sample.
(B) A neural network S (-) adopting U-Net with global characteristics;
to improve the execution efficiency of the model, we reduce the channel dimension of each convolutional layer as shown in fig. 2. The backbone network of our model is based on U-Net. It consists of a contracting step and an expanding step. In the concentrating step of each step, the convolution layer with 3 × 3 filters and 1 step size, the batch normalization layer and the maximum pooling layer with 2 × 2 filters and 2 step size are formed. In the expanding step, each step firstly executes deconvolution, the result of the deconvolution is spliced with the feature map of the corresponding step, and then the convolution layer with 3 multiplied by 3 and step length of 1 and the batch processing normalization layer are processed by 2 filters.
(C) The input of which is an original graph enhanced by dataImage S Input And prior illumination estimation I, and output is enhanced R output And I output ；
The Retinex theory is based on the color constancy theory. He considers that the image S of the object seen by a person is obtained by reflecting incident light I from the surface of the object, the reflectivity R is determined by the object itself and is not changed by the incident light I, and the process can be expressed by the formula:
S＝RοI (1)
although the method based on Retinex theory makes great progress in dim light enhancement and underwater image enhancement. However, in image beautification, the label objects learned by the neural network are from professional reviewers who make artistic modifications to the original image, such as modifying the color of the image equally. These operations break the assumption of Retinex color constancy.
Inspired by the above, we designed a decomposed neural network to transform the original image S Input Output as enhanced R via a neural network S (-) output And I output We only design one illumination smoothness loss as R output And I output And (c) a constraint therebetween. And reconstructing S according to the formula (1) output 。
(D) Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-);
we constructed our neural network on Tensorflow, in
The TITAN V GPU is trained, the batch processing size is 8, and 100 batches are trained. Adam optimizer was used and the learning rate was set to 5e in the first 10 batches -3 The remaining lot is set to 5e -4 . We scale the original resolution of their image pairs to 1048 on the smallest side length and randomly crop the data to 1024 when training the network.
(E) Training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model;
weight ofThe figure calculation loss formula consists of six modules including local loss function, color loss function and L 1 And an MS-SSIM loss function, a VGG loss function and an illumination smoothing loss function, which are expressed as:
(1) a local loss function. Randomly cropping image blocks from the input and label images and calculating L between the input and label image blocks 1 The loss, expressed as:
wherein, f (x) represents the image block generated by prediction, Y is the label image block, and n is the batch processing parameter of training.
(2) A color loss function. Expressed as:
wherein X b And Y b Representing X and Y gaussian blurred images, respectively.
(3)L 1 And MS-SSIM loss function. Expressed as:
L 1 ＝||X-Y|| 1 (5)
L MS-SSIM ＝1-MS-SSIM(X,Y) (6)
wherein X and Y represent the predicted image and the target image, respectively.
(4) VGG loss function. Expressed as:
(5) A structure-aware illumination smoothing loss function. Expressed as:
wherein
Is shown on the horizontal
To the vertical
Gradient of (a) t The coefficients are the strength of the structural perceptual coefficients.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.
Claims (3)
1. An image enhancement method based on deep learning is characterized by comprising the following steps:
(A) selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into a training set T train And test set T test ；
(B) A neural network S (-) adopting U-Net with global characteristics;
(C) s (-) input as the raw image S subjected to data amplification input And a priori illumination estimation map I input The output is the adjusted reflection chart R output And an illumination pattern I output And R is output And I output Element-wise multiplication to obtain an enhanced image S output ；
Wherein the training process of S (-) of the neural network of U-Net of global features is as follows: randomly initializing weight parameters, learning rate and batch size related parameters in a neural network S (-) to train the neural network model by adopting an error back propagation algorithm, training the network S (-) by using an adam optimization method, calculating loss based on a weight map, and stopping training and storing the network S (-) to obtain an image enhancement model when a loss function reaches an expectation;
the weight map is used for calculating loss and consists of six modules including a local loss function, a color loss function and L 1 And an MS-SSIM loss function, a VGG loss function and an illumination smoothing loss function, which are expressed as:
(1) a local loss function; randomly cropping image blocks from the predicted image and label images and calculating the L between image blocks 1 The loss, expressed as:
wherein, C x Representing cropped image blocks in a predicted image, C y The method comprises the steps of (1) obtaining a cut image block in a label image, wherein n is a training batch processing parameter;
(2) a color loss function; the predicted image X and the tag image Y are gaussian blurred and the euclidean distance between them is calculated, and is expressed as:
wherein X b And Y b Respectively represent X and YImages after Gaussian blur;
(3)L 1 and an MS-SSIM loss function; expressed as:
L 1 ＝||X-Y|| 1 (4)
L MS-SSIM ＝1-MS-SSIM(X,Y) (5)
wherein X and Y represent a predicted image and a tag image, respectively;
(4) VGG loss function, expressed as:
is a feature map obtained from the jth convolutional layer obtained from VGG-19, parameter C j ，H j ，W j Respectively representing the number of channels, height and width of the related j layers of convolution layers, wherein X and Y respectively represent a predicted image and a label image;
(5) a structure-aware illumination smoothing loss function; expressed as:
wherein
Is shown horizontally
To above and below
Gradient of (a) t The coefficients are the strength of the structural perceptual coefficients.
2. The deep learning-based image enhancement method according to claim 1, wherein in the step (B), the U-Net neural network with global features S (-) is specifically composed of a contracting step and an expanding step, each contracting step is composed of 2 convolution layers with 3 × 3 step size 1 filter, a batch normalization layer and a maximum pooling layer with 2 × 2 step size 2 filter, each expanding step is performed with deconvolution first, the result of deconvolution is spliced with the feature map of the corresponding step, and then the convolution layer and the batch normalization layer with 3 × 3 step size 1 filter are passed through 2 filters.
3. The method for enhancing image based on deep learning as claimed in claim 2, wherein in the step (C), the data is augmented by down-sampling the original image to a specified resolution size and performing random cropping and rotation operations on the original image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010056687.6A CN113139909B (en) | 2020-01-19 | 2020-01-19 | Image enhancement method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010056687.6A CN113139909B (en) | 2020-01-19 | 2020-01-19 | Image enhancement method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113139909A CN113139909A (en) | 2021-07-20 |
| CN113139909B true CN113139909B (en) | 2022-08-02 |
Family
ID=76808533
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010056687.6A Expired - Fee Related CN113139909B (en) | 2020-01-19 | 2020-01-19 | Image enhancement method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113139909B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114511462B (en) * | 2022-02-11 | 2023-04-18 | 电子科技大学 | Visual image enhancement method |
| CN115018729B (en) * | 2022-06-17 | 2024-04-02 | 重庆米弘科技有限公司 | Content-oriented white box image enhancement method |
| CN115294263B (en) * | 2022-10-08 | 2023-02-03 | 武汉大学 | Illumination estimation method and system |
| CN117671073B (en) * | 2024-01-31 | 2024-05-17 | 三亚学院 | Language prompt-based image style imaging system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018042388A1 (en) * | 2016-09-02 | 2018-03-08 | Artomatix Ltd. | Systems and methods for providing convolutional neural network based image synthesis using stable and controllable parametric models, a multiscale synthesis framework and novel network architectures |
| CN107464244A (en) * | 2017-03-09 | 2017-12-12 | 广东顺德中山大学卡内基梅隆大学国际联合研究院 | A kind of image irradiation method of estimation based on neutral net |
| CN109003231B (en) * | 2018-06-11 | 2021-01-29 | 广州视源电子科技股份有限公司 | Image enhancement method and device and display equipment |
| CN109410129A (en) * | 2018-09-28 | 2019-03-01 | 大连理工大学 | A kind of method of low light image scene understanding |
| CN110232661B (en) * | 2019-05-03 | 2023-01-06 | 天津大学 | Low-illumination color image enhancement method based on Retinex and convolutional neural network |
| CN110264423A (en) * | 2019-06-19 | 2019-09-20 | 重庆米弘科技有限公司 | A method of the image visual effect enhancing based on full convolutional network |
| CN110517203B (en) * | 2019-08-30 | 2023-06-23 | 山东工商学院 | Defogging method based on reference image reconstruction |
-
2020
- 2020-01-19 CN CN202010056687.6A patent/CN113139909B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN113139909A (en) | 2021-07-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113139909B (en) | Image enhancement method based on deep learning | |
| Zhang et al. | Dual illumination estimation for robust exposure correction | |
| Liang et al. | Cameranet: A two-stage framework for effective camera isp learning | |
| CN109064423B (en) | Intelligent image repairing method for generating antagonistic loss based on asymmetric circulation | |
| CN102082864A (en) | Camare360 mobile phone photographic platform and instant processing method | |
| Pan et al. | MIEGAN: Mobile image enhancement via a multi-module cascade neural network | |
| CN111612707B (en) | Neural network image denoising method based on wavelet transformation | |
| Bianco et al. | Personalized image enhancement using neural spline color transforms | |
| Asha et al. | Auto removal of bright spot from images captured against flashing light source | |
| CN113284061B (en) | Underwater image enhancement method based on gradient network | |
| CN110610526A (en) | Method for segmenting monocular portrait and rendering depth of field based on WNET | |
| CN111192226A (en) | Image fusion denoising method, device and system | |
| CN113222845A (en) | Portrait external shadow removing method based on convolution neural network | |
| US11138693B2 (en) | Attention-driven image manipulation | |
| Lv et al. | Low-light image enhancement via deep Retinex decomposition and bilateral learning | |
| WO2023081399A1 (en) | Integrated machine learning algorithms for image filters | |
| CN114862698A (en) | Method and device for correcting real overexposure image based on channel guidance | |
| CN115641391A (en) | Infrared image colorizing method based on dense residual error and double-flow attention | |
| CN117593235A (en) | Retinex variation underwater image enhancement method and device based on depth CNN denoising prior | |
| CN112070686B (en) | Backlight image cooperative enhancement method based on deep learning | |
| CN117391987A (en) | Dim light image processing method based on multi-stage joint enhancement mechanism | |
| CN112184586A (en) | Method and system for rapidly blurring monocular visual image background based on depth perception | |
| CN116152128A (en) | High dynamic range multi-exposure image fusion model and method based on attention mechanism | |
| CN114663300A (en) | DCE-based low-illumination image enhancement method, system and related equipment | |
| CN113810597B (en) | Rapid image and scene rendering method based on semi-predictive filtering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220802 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |