CN113139909A - Image enhancement method based on deep learning - Google Patents

Image enhancement method based on deep learning Download PDF

Info

Publication number
CN113139909A
CN113139909A CN202010056687.6A CN202010056687A CN113139909A CN 113139909 A CN113139909 A CN 113139909A CN 202010056687 A CN202010056687 A CN 202010056687A CN 113139909 A CN113139909 A CN 113139909A
Authority
CN
China
Prior art keywords
image
neural network
loss function
expressed
deep learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010056687.6A
Other languages
Chinese (zh)
Other versions
CN113139909B (en
Inventor
桑葛楠
李浬
袁峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Oying Network Technology Co ltd
Original Assignee
Hangzhou Oying Network Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Oying Network Technology Co ltd filed Critical Hangzhou Oying Network Technology Co ltd
Priority to CN202010056687.6A priority Critical patent/CN113139909B/en
Publication of CN113139909A publication Critical patent/CN113139909A/en
Application granted granted Critical
Publication of CN113139909B publication Critical patent/CN113139909B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G06T5/90—
    • G06T5/77—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses an image enhancement method based on deep learning, which comprises the following steps: selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets TtrainAnd test set Ttest(ii) a A neural network S (-) adopting U-Net with global characteristics; the input of which is an original image S subjected to data enhancementInputAnd prior illumination estimation I, and output is enhanced RoutputAnd Ioutput(ii) a Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-); training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model; compared with the prior method, the method has more natural colors, more attractiveness and better contrast, has smaller difference with the images repaired by professional diagraphers, has no artifact phenomenon, has little inference time, and can run on equipment such as a mobile phone and the like in ms level in real time.

Description

Image enhancement method based on deep learning
Technical Field
The invention relates to the technical field of image processing, in particular to an image enhancement method based on deep learning.
Background
The photography is an art of light and shadow, and a favorite photo is accurately grasped for brightness and color shading of the light and shadow. With the popularity of portable photography devices such as mini-tickets, mobile phones and the like, more and more people use photos to record their lives, start enjoying photography and enjoy sharing their photography works on social networks. In general, people often take photos of themselves due to unnatural brightness, insufficiently saturated colors, and the like. Therefore, people often spend a lot of time beautifying their images before sharing their own photographic works. Despite the large number of interactive and semi-automatic image processing tools available on the market, these tools still have a large threshold for the user, and the beautified photos are also associated with the aesthetic level of the user.
Image enhancement has been a long-standing challenge in the field of computer vision and has been of interest to scholars. The traditional image enhancement algorithm mainly comprises histogram equalization, gray world assumption, wavelet transformation algorithm, automatic white balance and the like. These algorithms focus mainly on the contrast of the image and the correction of the color. However, these algorithms are only suitable for specific conditions, such as histogram equalization without selection of data processing, automatic white balance only suitable for uniform illumination, and the like. And the processed images still have a large gap from the expectations of people. In recent years, image processing algorithms based on deep learning have been greatly successful in the field of image enhancement, but these methods still have the disadvantages of low processing speed, undesirable effect and the like.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide an image enhancement method for deep learning, which meets the requirements of real-time performance, has more natural colors, more attractiveness and better contrast compared with the prior method, has smaller difference with the image repaired by a professional repairman, and has no artifact phenomenon.
1. An image enhancement method based on deep learning is characterized by comprising the following steps:
(A) selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets TtrainAnd test set Ttest；
(B) A neural network S (-) adopting U-Net with global characteristics;
(C) the input of which is an original image S subjected to data enhancementInputIn combination with the a-priori illumination estimate I,the output is enhanced RoutputAnd Ioutput；
(D) Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-);
(E) and training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model.
2. The deep learning-based image enhancement method according to claim 1, wherein in the step (B), the U-Net neural network S (-) with global features is composed of a connecting step and an expanding step. In the cont parsing step of each step, the convolution layer with 3 × 3 filter steps of 1, the batch normalization layer, and the maximum pooling layer with 2 × 2 filter steps of 2 are included. In the expanding step, each step firstly executes deconvolution, the result of the deconvolution is spliced with the feature map of the corresponding step, and then the convolution layer with 3 multiplied by 3 and step length of 1 and the batch processing normalization layer are processed by 2 filters.
3. The method for enhancing image based on deep learning as claimed in claim 2, wherein in the step (C), the data is enhanced by down-sampling the original image to a specified resolution size and performing random cropping, rotation and the like on the original image.
4. The method for enhancing image based on deep learning of claim 3, wherein in the step (E), the weight map is a formula of loss calculation, which is composed of six modules including a local loss function, a color loss function, and L1And an MS-SSIM loss function, a VGG loss function and an illumination smoothing loss function, which are expressed as:
Figure BDA0002373138700000036
(1) a local loss function. Randomly cropping image blocks from the input and label images and calculating L between the input and label image blocks1The loss, expressed as:
Figure BDA0002373138700000031
wherein, f (x) represents the image block generated by prediction, Y is the label image block, and n is the batch processing parameter of training.
(2) A color loss function. Expressed as:
Figure BDA0002373138700000032
wherein XbAnd YbRepresenting X and Y gaussian blurred images, respectively.
(3)L1And MS-SSIM loss function. Expressed as:
L1＝||X-Y||1 (4)
LMS-SSIM＝1-MS-SSIM(X,Y) (5)
wherein X and Y represent the predicted image and the target image, respectively.
(4) VGG loss function. Expressed as:
Figure BDA0002373138700000033
Figure BDA0002373138700000034
is a feature map obtained from the jth convolutional layer obtained from VGG-19, parameter Cj，Hj，WjRespectively representing the number of channels, height and width of the convolution layer of the relevant j layers, and X and Y respectively representing the predicted image and the target image.
(5) A structure-aware illumination smoothing loss function. Expressed as:
Figure BDA0002373138700000035
wherein
Figure BDA0002373138700000041
Is shown horizontally
Figure BDA0002373138700000042
To above and below
Figure BDA0002373138700000043
Gradient of (a)tThe coefficients are the strength of the structural perceptual coefficients.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a sample diagram of a data set of an image enhancement method based on deep learning according to the present invention.
FIG. 2 is a network model diagram of the image enhancement method based on deep learning according to the present invention.
Detailed Description
In order to make the objects, embodiments and advantages of the present invention more apparent, further detailed description is given herein with reference to specific examples:
(A) selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets TtrainAnd test set Ttest；
MIT-Adobe FiveK provides 5000 original images, as well as image data that was manually retouched by 5 professional reviewers (A, B, C, D, E). However, the following disadvantages of the vek data set still exist, and firstly, the data volume of the vek data set is still small, so that the training of the neural network cannot be met, overfitting is easily caused, and the requirements under a real scene cannot be met. Secondly, it has the problem of less data diversity, a significant portion of the original image data is low contrast and low brightness, and a small portion of the data is over-exposed and covers only limited lighting conditions.
We hired a photographer, using a different single lens reflex camera device, to take a total of 18000 photos, containing a very rich scene, lighting, theme, etc. We have a good professional reviewer who revises these captured original photographs to achieve satisfactory revised photograph data, as shown in fig. 1, for the purpose of presenting a partial data set sample.
(B) A neural network S (-) adopting U-Net with global characteristics;
to improve the execution efficiency of the model, we reduce the channel dimension of each convolutional layer as shown in fig. 2. The backbone network of our model is based on U-Net. It consists of a contracting step and an expanding step. In the concentrating step of each step, the convolution layer with 3 × 3 filters and 1 step size, the batch normalization layer and the maximum pooling layer with 2 × 2 filters and 2 step size are formed. In the expanding step, each step firstly executes deconvolution, the result of the deconvolution is spliced with the feature map of the corresponding step, and then the convolution layer with 3 multiplied by 3 and step length of 1 and the batch processing normalization layer are processed by 2 filters.
(C) The input of which is an original image S subjected to data enhancementInputAnd prior illumination estimation I, and output is enhanced RoutputAnd Ioutput；
The Retinex theory is based on the color constancy theory. He considers that the image S of the object seen by a person is obtained by reflecting incident light I from the surface of the object, the reflectivity R is determined by the object itself and is not changed by the incident light I, and the process can be expressed by the formula:
S＝RοI (1)
although the method based on Retinex theory makes great progress in dim light enhancement and underwater image enhancement. However, in image beautification, the label object learned by the neural network is from a professional reviewer who performs artistic modification on the original image, such as modifying the color of the image to be equal. These operations break the assumption of Retinex color constancy.
Inspired by the above, we designed a decomposed neural network to transform the original image SInputOutput as enhanced R through neural network S (-)outputAnd IoutputWe only design oneSmooth loss of illumination as RoutputAnd IoutputAnd (c) a constraint therebetween. And reconstructing S according to the formula (1)output。
(D) Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-);
we constructed our neural network on Tensorflow, in
Figure BDA0002373138700000051
The TITAN V GPU is trained, the batch processing size is 8, and 100 batches are trained. Adam optimizer was used and the learning rate was set to 5e in the first 10 batches-3The remaining lot is set to 5e-4. We scale the original resolution of their image pairs to 1048 on the smallest side length and randomly crop the data to 1024 when training the network.
(E) Training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model;
the weight graph calculation loss formula consists of six modules including local loss function, color loss function and L1And an MS-SSIM loss function, a VGG loss function and an illumination smoothing loss function, which are expressed as:
Figure BDA0002373138700000065
(1) a local loss function. Randomly cropping image blocks from the input and label images and calculating L between the input and label image blocks1The loss, expressed as:
Figure BDA0002373138700000061
wherein, f (x) represents the image block generated by prediction, Y is the label image block, and n is the batch processing parameter of training.
(2) A color loss function. Expressed as:
Figure BDA0002373138700000062
wherein XbAnd YbRepresenting X and Y gaussian blurred images, respectively.
(3)L1And MS-SSIM loss function. Expressed as:
L1＝||X-Y||1 (5)
LMS-SSIM＝1-MS-SSIM(X,Y) (6)
wherein X and Y represent the predicted image and the target image, respectively.
(4) VGG loss function. Expressed as:
Figure BDA0002373138700000063
Figure BDA0002373138700000064
is a feature map obtained from the jth convolutional layer obtained from VGG-19, parameter Cj，Hj，WjRespectively representing the number of channels, height and width of the convolution layer of the relevant j layers, and X and Y respectively representing the predicted image and the target image.
(5) A structure-aware illumination smoothing loss function. Expressed as:
Figure BDA0002373138700000071
wherein
Figure BDA0002373138700000072
Is shown horizontally
Figure BDA0002373138700000073
To above and below
Figure BDA0002373138700000074
Gradient of (a)tIntensity of coefficient being structural perception coefficient。
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.

Claims (4)

1. An image enhancement method based on deep learning is characterized by comprising the following steps:
(A) selecting images shot by professional photographers, making the professional photographers perform image retouching, constructing a neural network training data set, and dividing the neural network training data set into training sets TtrainAnd test set Ttest；
(B) A neural network S (-) adopting U-Net with global characteristics;
(C) the input of which is an original image S subjected to data enhancementInputAnd prior illumination estimation I, and output is enhanced RoutputAnd Ioutput；
(D) Randomly initializing relevant parameters such as weight parameters, learning rate, batch size and the like in the neural network S (-);
(E) and training the image enhancement neural network model by adopting an error back propagation algorithm, and calculating loss based on the weight map so as to obtain a depth image enhancement model.
2. The deep learning-based image enhancement method according to claim 1, wherein in the step (B), the U-Net neural network S (-) with global features is composed of a connecting step and an expanding step. In the cont parsing step of each step, the convolution layer with 3 × 3 filter steps of 1, the batch normalization layer, and the maximum pooling layer with 2 × 2 filter steps of 2 are included. In the expanding step, each step firstly executes deconvolution, the result of the deconvolution is spliced with the feature map of the corresponding step, and then the convolution layer with 3 multiplied by 3 and step length of 1 and the batch processing normalization layer are processed by 2 filters.
3. The method for enhancing image based on deep learning as claimed in claim 2, wherein in the step (C), the data is enhanced by down-sampling the original image to a specified resolution size and performing random cropping, rotation and the like on the original image.
4. The method for enhancing image based on deep learning of claim 3, wherein in the step (E), the weight map is a formula of loss calculation, which is composed of six modules including a local loss function, a color loss function, and L1And an MS-SSIM loss function, a VGG loss function and an illumination smoothing loss function, which are expressed as:
Figure FDA0002373138690000021
(1) a local loss function. Randomly cropping image blocks from the input and label images and calculating L between the input and label image blocks1The loss, expressed as:
Figure FDA0002373138690000022
wherein, f (x) represents the image block generated by prediction, Y is the label image block, and n is the batch processing parameter of training.
(2) A color loss function. Expressed as:
Figure FDA0002373138690000023
wherein XbAnd YbRepresenting X and Y gaussian blurred images, respectively.
(3)L1And MS-SSIM loss function. Expressed as:
L1＝||X-Y||1 (4)
LMS-SSIM＝1-MS-SSIM(X,Y) (5)
wherein X and Y represent the predicted image and the target image, respectively.
(4) VGG loss function. Expressed as:
Figure FDA0002373138690000024
Figure FDA0002373138690000025
is a feature map obtained from the jth convolutional layer obtained from VGG-19, parameter Cj，Hj，WjRespectively representing the number of channels, height and width of the convolution layer of the relevant j layers, and X and Y respectively representing the predicted image and the target image.
(5) A structure-aware illumination smoothing loss function. Expressed as:
Figure FDA0002373138690000031
wherein
Figure FDA0002373138690000032
Is shown horizontally
Figure FDA0002373138690000033
To above and below
Figure FDA0002373138690000034
Gradient of (a)tThe coefficients are the strength of the structural perceptual coefficients.
CN202010056687.6A 2020-01-19 2020-01-19 Image enhancement method based on deep learning Expired - Fee Related CN113139909B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010056687.6A CN113139909B (en) 2020-01-19 2020-01-19 Image enhancement method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010056687.6A CN113139909B (en) 2020-01-19 2020-01-19 Image enhancement method based on deep learning

Publications (2)

Publication Number Publication Date
CN113139909A true CN113139909A (en) 2021-07-20
CN113139909B CN113139909B (en) 2022-08-02

Family

ID=76808533

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010056687.6A Expired - Fee Related CN113139909B (en) 2020-01-19 2020-01-19 Image enhancement method based on deep learning

Country Status (1)

Country Link
CN (1) CN113139909B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114511462A (en) * 2022-02-11 2022-05-17 电子科技大学 Visual image enhancement method
CN115018729A (en) * 2022-06-17 2022-09-06 重庆米弘科技有限公司 White box image enhancement method for content
CN115294263A (en) * 2022-10-08 2022-11-04 武汉大学 Illumination estimation model, network, method and system
CN117671073A (en) * 2024-01-31 2024-03-08 三亚学院 Language prompt-based image style imaging system
CN117671073B (en) * 2024-01-31 2024-05-17 三亚学院 Language prompt-based image style imaging system

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107464244A (en) * 2017-03-09 2017-12-12 广东顺德中山大学卡内基梅隆大学国际联合研究院 A kind of image irradiation method of estimation based on neutral net
US20180144509A1 (en) * 2016-09-02 2018-05-24 Artomatix Ltd. Systems and Methods for Providing Convolutional Neural Network Based Image Synthesis Using Stable and Controllable Parametric Models, a Multiscale Synthesis Framework and Novel Network Architectures
CN109003231A (en) * 2018-06-11 2018-12-14 广州视源电子科技股份有限公司 A kind of image enchancing method, device and display equipment
CN109410129A (en) * 2018-09-28 2019-03-01 大连理工大学 A kind of method of low light image scene understanding
CN110232661A (en) * 2019-05-03 2019-09-13 天津大学 Low illumination colour-image reinforcing method based on Retinex and convolutional neural networks
CN110264423A (en) * 2019-06-19 2019-09-20 重庆米弘科技有限公司 A method of the image visual effect enhancing based on full convolutional network
CN110517203A (en) * 2019-08-30 2019-11-29 山东工商学院 A kind of defogging method rebuild based on reference picture

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180144509A1 (en) * 2016-09-02 2018-05-24 Artomatix Ltd. Systems and Methods for Providing Convolutional Neural Network Based Image Synthesis Using Stable and Controllable Parametric Models, a Multiscale Synthesis Framework and Novel Network Architectures
CN107464244A (en) * 2017-03-09 2017-12-12 广东顺德中山大学卡内基梅隆大学国际联合研究院 A kind of image irradiation method of estimation based on neutral net
CN109003231A (en) * 2018-06-11 2018-12-14 广州视源电子科技股份有限公司 A kind of image enchancing method, device and display equipment
CN109410129A (en) * 2018-09-28 2019-03-01 大连理工大学 A kind of method of low light image scene understanding
CN110232661A (en) * 2019-05-03 2019-09-13 天津大学 Low illumination colour-image reinforcing method based on Retinex and convolutional neural networks
CN110264423A (en) * 2019-06-19 2019-09-20 重庆米弘科技有限公司 A method of the image visual effect enhancing based on full convolutional network
CN110517203A (en) * 2019-08-30 2019-11-29 山东工商学院 A kind of defogging method rebuild based on reference picture

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114511462A (en) * 2022-02-11 2022-05-17 电子科技大学 Visual image enhancement method
CN114511462B (en) * 2022-02-11 2023-04-18 电子科技大学 Visual image enhancement method
CN115018729A (en) * 2022-06-17 2022-09-06 重庆米弘科技有限公司 White box image enhancement method for content
CN115018729B (en) * 2022-06-17 2024-04-02 重庆米弘科技有限公司 Content-oriented white box image enhancement method
CN115294263A (en) * 2022-10-08 2022-11-04 武汉大学 Illumination estimation model, network, method and system
CN115294263B (en) * 2022-10-08 2023-02-03 武汉大学 Illumination estimation method and system
CN117671073A (en) * 2024-01-31 2024-03-08 三亚学院 Language prompt-based image style imaging system
CN117671073B (en) * 2024-01-31 2024-05-17 三亚学院 Language prompt-based image style imaging system

Also Published As

Publication number Publication date
CN113139909B (en) 2022-08-02

Similar Documents

Publication Publication Date Title
CN113139909B (en) Image enhancement method based on deep learning
Zhang et al. Dual illumination estimation for robust exposure correction
CN109064423B (en) Intelligent image repairing method for generating antagonistic loss based on asymmetric circulation
CN110148088B (en) Image processing method, image rain removing method, device, terminal and medium
CN102082864A (en) Camare360 mobile phone photographic platform and instant processing method
Bianco et al. Personalized image enhancement using neural spline color transforms
CN113284061B (en) Underwater image enhancement method based on gradient network
Asha et al. Auto removal of bright spot from images captured against flashing light source
CN111612707B (en) Neural network image denoising method based on wavelet transformation
CN113222845A (en) Portrait external shadow removing method based on convolution neural network
WO2023081399A1 (en) Integrated machine learning algorithms for image filters
CN114862698A (en) Method and device for correcting real overexposure image based on channel guidance
WO2005031646A1 (en) Method and system for differentially and regularly modifying a digital image by pixel
CN112070686B (en) Backlight image cooperative enhancement method based on deep learning
CN117391987A (en) Dim light image processing method based on multi-stage joint enhancement mechanism
CN114663300A (en) DCE-based low-illumination image enhancement method, system and related equipment
CN116152128A (en) High dynamic range multi-exposure image fusion model and method based on attention mechanism
Tatanov et al. LFIEM: Lightweight filter-based image enhancement model
Li et al. Deep multi-path low-light image enhancement
CN112184586A (en) Method and system for rapidly blurring monocular visual image background based on depth perception
EP3913572A1 (en) Loss function for image reconstruction
CN113810597A (en) Rapid image and scene rendering method based on semi-prediction filtering
CN112132923A (en) Two-stage digital image style transformation method and system based on style thumbnail high-definition
Ghimpeţeanu et al. Local denoising applied to raw images may outperform non-local patch-based methods applied to the camera output
CN111445437A (en) Method, system and equipment for processing image by skin processing model constructed based on convolutional neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20220802

CF01 Termination of patent right due to non-payment of annual fee